Repository: moby/moby (language: go)

Issue:
Docker daemon is leaking file descriptors (with reproduction)
**Description**

Not sure this is related, but I have in production after a few days / weeks that dockerd starts leaking tons of file descriptors of dead connections that it simply stops responding at all (not even to SIGUSR1).

While trying to understand how this can happen (since when it happens I can't do anything but kill it), I tried to figure out how you get a dead connection, and have found some conclusions on how to get dockerd to leak such file descriptors, although I'm not sure that what I'm about to say is the issue that is happening in production.

How do those leaked file descriptors look ? I run the command on linux:
ss | grep docker.sock
and I see tons of entries that most look like this:
u_str  ESTAB      0      0      /var/run/docker.sock 272245                * 0                    
u_str  ESTAB      0      0      /var/run/docker.sock 279780                * 0                    
u_str  ESTAB      0      0      /var/run/docker.sock 279118                * 0                    
u_str  ESTAB      0      0      /var/run/docker.sock 272201                * 0                    
u_str  ESTAB      0      0      /var/run/docker.sock 272217                * 0

You can see this are dead connections because the peer info (the * 0 in the end) points to nothing which means the other side closed the unix domain socket, but the docker daemon did not.

also this sockets take up file descriptors (as expected):
ls -l /proc/$(pidof dockerd)/fd
...
lrwx------ 1 root root 64 may 31 08:37 33 -> socket:[272217]
...

an alive connection looks like this (there is peer info):
u_str  ESTAB      0      0      /var/run/docker.sock 280489                * 279381               

I'm going now to show you a convoluted way to cause this resource leak, but remember that still dockerd shouldn't leak resources and that in the end there might be similar code that really needs to do something like this...

This also seems to happen on variety of docker versions from old to new.

**Steps to reproduce the issue:**
1. docker create -it --rm python:2.7 python -c "while True: print 1;"
\<container id>
2. docker start \<container id>
\<container id>
3. python2
```python
import socket
s = socket.socket(socket.AF_UNIX)
s.connect('/var/run/docker.sock')
s.send('POST /v1.24/containers/<container id>/attach?logs=1&stream=0&stdout=1 HTTP/1.1\r\nHost: localhost\r\n\r\n')
s.recv(1024)
s.close()
```
You can also try changing the logs=1&stream=0 with logs=0&stream=1
Also don't think the v1.24 is of particulate importance.
Also you can kill the python instead of s.close() (sometimes it acts differently)

4. ss | grep docker.sock
u_str  ESTAB      0      0      /var/run/docker.sock 280489                * 0

**Describe the results you received:**

There is a dead unclosed unix domain socket on the docker daemon side.

**Describe the results you expected:**

The docker daemon should figure the socket has been closed on the other side, and close it's side as well.

**Versions**

Tried on multiple versions, now both client and server: 17.12.1-ce and on ubuntu 16.04.4 with kernel 4.13.0-36-generic with default settings.



Discussion / hints:
Confirmed there are a few leaks per attach (depending on what's attached).
#37184 should fix this.
Isn't the problem more deeply rooted ? let's change my example code to:
```docker create -it --rm python:2.7 python -c "while True: pass"```
 Where no output it sent to neither stdout nor stderr. In this case wouldn't:
https://github.com/cpuguy83/docker/blob/0f5147701775a6c5d4980a7b7c0ed2e830688034/container/stream/attach.go#L111
Block because nothing is written to the streampipe, and thus that code is blocked while stream is already dead ?
The pipes are still closed and thus unblock the copy call.
Note I tested this just as you have here and at least there are no goroutines left behind.

You can see the goroutine count if you run `dockerd` with debug mode, and then check `docker info`.
I do not understand why is streamPipe (which is the stdout/stderr of the container) closed. I would think that this line would block (called from pools.Copy line) because the container stdout/stderr still exists (although nobody writes to it):
https://github.com/golang/go/blob/15ac56fd60a28879fd65f6d761cc30c4c57ec0f1/src/io/io.go#L402
Pretty sure this is what I saw on the current code base where if I listen to stderr, but nothing is written to it, this line never happens until I kill the container:
https://github.com/moby/moby/blob/71cd53e4a197b303c6ba086bd584ffd67a884281/container/stream/attach.go#L131
(and only after i kill the container, that line shows and the leak stops, and the only think blocking before this line is the pools.Copy).
I am talking about the case where I do not attach to stdin, just to stdout/err which are not written to from the container.
In this case, it's because the stdin stream from the client is closed.
However you are correct that when `stdin` is not attached there is a leak still.
So I guess before calling pools.Copy there should be some kind of select on both pipes to see if one has data and if the other has been closed and only then decide if to quit or actually Copy
I was thinking of handling this by passing down the context from the API and making sure the context is cancelled when the client disconnects.
Oh that doesn't work because of the hijacked connection.
I started messing around with a fix for this, but am taking some time off after today.

Idea is to use epoll on the client FD and cancel the attach context when we get a close event from epoll.
It's not really working yet as it's getting the close event immediately for some reason. I only have a little experience with epoll.... if someone wants to take a look in the meantime:

https://github.com/moby/moby/compare/master...cpuguy83:attach_leak_no_data?expand=1
Why was this issue closed ? Did you read the discussion ? That patch does not resolve the issue
It was automatically closed because the PR had a `fixes #37182`, I can reopen if it's not resolved 
Looks like the patch in https://github.com/moby/moby/pull/37184 was different than the one linked/discussed above https://github.com/cpuguy83/docker/compare/attach_leak...attach_leak_no_data
There's just the remaining case where there is no output from the container where we still leak goroutines.
I have a similar issue for the HijackResponse.
If I run a none output command, like "mkdir test", the client is going to be timeout, it's caused by the same issue of pool.copy that @tzickel mentioned .
For fixing this issue , I have to wrap all my command with echo `<Command>` 

hey look at this the bug I was hoping to find.

I have a reproduction in my CI system's runner if anyone would like to peek at that. The issue seems to be attach and http.Transport's CloseIdleConnections() not competing well. code is here: https://github.com/tinyci/ci-runners/blob/master/runners/overlay-runner/docker.go#L201-L247 I've highlighted the entrypoint of the relevant bits.

If you store and manage a *single* docker client object, that's the workaround for now. It leaves one fd open and the attach eventually times out. The re-establishment of the docker client seems to leave the attaches hanging for some reason, which manifests itself as a single goroutine holding a http client piped to a buffer. Looks a lot like a lot of these just without any parents:

```
goroutine 29 [IO wait, 29 minutes]:
internal/poll.runtime_pollWait(0x7f14ca7bed58, 0x72, 0xffffffffffffffff)
        /usr/local/go/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc0001b6218, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc0001b6200, 0xc0001a1000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b
net.(*netFD).Read(0xc0001b6200, 0xc0001a1000, 0x1000, 0x1000, 0x1, 0x42fb0c, 0xc000224b88)
        /usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc000326060, 0xc0001a1000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/net.go:177 +0x69
net/http.(*persistConn).Read(0xc000510240, 0xc0001a1000, 0x1000, 0x1000, 0xc000224c88, 0x406c15, 0xc0002d2540)
        /usr/local/go/src/net/http/transport.go:1524 +0x7b
bufio.(*Reader).fill(0xc000094d20)
        /usr/local/go/src/bufio/bufio.go:100 +0x10f
bufio.(*Reader).Peek(0xc000094d20, 0x1, 0x0, 0x0, 0x1, 0xc0002ef300, 0x0)
        /usr/local/go/src/bufio/bufio.go:138 +0x4f
net/http.(*persistConn).readLoop(0xc000510240)
        /usr/local/go/src/net/http/transport.go:1677 +0x1a3
created by net/http.(*Transport).dialConn
        /usr/local/go/src/net/http/transport.go:1357 +0xae8
```

Strong suggestion would be to wrap all the attaches in a http.Request.WithContext() (if possible) so you can cancel it once the attach closes, if your hijacks will work with this.
I have no good, simple reproduction, and while I tried, I failed to do so and I have only a little time to spend on this. Sorry.
I did try this on a multitude of client versions, fwiw. Now I'll stop with the spam hose. everything at from around 1.10.1 to at least up to 18.09.7 is affected with an 18.09 daemon.
I don't think a request context will help since in the case of hijacking the http conn, all the http stuff is dropped, you just have raw net.Conns.

I believe the issue is likely due to what I've described in https://medium.com/@cpuguy83/non-blocking-i-o-in-go-bc4651e3ac8d around how io.Copy works.
Ah, you're probably right.

On Tue, Jul 9, 2019 at 8:36 AM Brian Goff <notifications@github.com> wrote:

> I don't think a request context will help since in the case of hijacking
> the http conn, all the http stuff is dropped, you just have raw net.Conns.
>
> I believe the issue is likely due to what I've described in
> https://medium.com/@cpuguy83/non-blocking-i-o-in-go-bc4651e3ac8d around
> how io.Copy works.
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/moby/moby/issues/37182?email_source=notifications&email_token=AAAET22LZLHNRGW5G4RI43TP6SV7ZA5CNFSM4FCR7NT2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZQUW7I#issuecomment-509692797>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAAET22FSOJHMJC4QXTGUJLP6SV7ZANCNFSM4FCR7NTQ>
> .
>

Came across this after spending few hours trying to understand why several Kubernetes worker nodes became "Not Ready", we see the exact same issue.

Running Docker 18.09.7, a lot of "dead" socket descriptors (>30K) and kubelet is unable to communicate with Docker... only thing that helps is killing docker (or rebooting the server).

this happened on 6 different servers (Debian stretch, kernel 4.19.28), it seems that a deployment process was the trigger (causing a lot of pods to be terminated and new created).

We previously used Docker 18.06.1 and never hit such an issue.

Is anyone actively looking into this? is there anything we can provide to help investigate?
The solution I came up with was to use a persistent docker client and not
open/close clients indiscriminately. It's in code and may not be of
assistance, but I can offer a link if needed to the code in question.

On Tue, Nov 26, 2019 at 2:13 PM Danny Kulchinsky <notifications@github.com>
wrote:

> Came across this after spending few hours trying to understand why several
> Kubernetes worker nodes became "Not Ready", we see the exact same issue.
>
> Running Docker 18.09.7, a lot of "dead" socket descriptors (>30K) and
> kubelet is unable to communicate with Docker... only thing that helps is
> killing docker (or rebooting the server).
>
> this happened on 6 different servers (Debian stretch, kernel 4.19.28), it
> seems that a deployment process was the trigger (causing a lot of pods to
> be terminated and new created).
>
> We previously used Docker 18.06.1 and never hit such an issue.
>
> Is anyone actively looking into this? is there anything we can provide to
> help investigate?
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/moby/moby/issues/37182?email_source=notifications&email_token=AAAET25VGUBRJ2UBKVI6DNLQVWNPJA5CNFSM4FCR7NT2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFHTSJA#issuecomment-558840100>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAAET22VAOUW2GOY26JBX2DQVWNPJANCNFSM4FCR7NTQ>
> .
>

thanks @erikh, unfortunately since this is being provoked by queries from kubelet to docker, I don't think we could use that workaround :disappointed: 

please do share the link, I might cross reference this in a separate kubernetes issue, to consider to use a persistent client.

Given our more positive experience with 18.06.1, we may try to downgrade and see if the issue disappears, from what I can see there are a bunch of reports out there for similar issue so would be great if the root cause could be resolved.
@dannyk81 it looks like https://github.com/moby/moby/issues/37182#issuecomment-509681877 covers most of what I was about to say, hope it helps and please feel free to follow up with any questions. that repo's "overlay-runner" does not suffer from this defect on any version of docker.
Docker version 19.03.1
Having trouble with leaked sockets to `/var/run/docker.sock`. Looks like issue due to `ContainerExecAttach`.
I tried to use single docker client instead of creating it each time. It did not helped.
Also tried to attach only Stdout which always have output in my case. Still the same.

Here is an example of goroutine that is started every hour to collect db size of postgres running in container. During execution it opens  new socket with `type=stream` and leaves it open forever. Basically until soft limit is reached.

```golang
func collectDBSizePoints(conf *config, cli *dockerclient.Client) []point {
	var points []point
	ctx := context.Background()
	containers, err := cli.ContainerList(ctx, types.ContainerListOptions{
		Filters: filters.NewArgs(filters.Arg("name", conf.PgHost),
			filters.Arg("status", "running")),
	})
	if err != nil {
		log.Println(err)
		return points
	}
	if len(containers) != 1 {
		return points
	}
	cmd := []string{"psql", "-c",
		`COPY (SELECT t1.datname AS db_name, pg_database_size(t1.datname) AS db_size
        FROM pg_database t1 ORDER BY pg_database_size(t1.datname) DESC)
        TO STDOUT WITH CSV;`}
	c := types.ExecConfig{
		AttachStdout: true,
		AttachStderr: false,
		AttachStdin:  false,
		Tty:  false,
		Cmd:          strslice.StrSlice(cmd),
		Env: []string{"PGHOST=" + conf.PgHost,
			"PGUSER=" + conf.PgUser,
			"PGPASSWORD=" + conf.PgPass},
	}
	execResp, err := cli.ContainerExecCreate(ctx, containers[0].ID, c)
	if err != nil {
		log.Println(err)
		return points
	}
	resp, err := cli.ContainerExecAttach(ctx, execResp.ID,
		types.ExecStartCheck{
			Detach: false,
			Tty:    false,
		},
	)
	if err != nil {
		log.Println(err)
		return points
	}
	defer resp.Close()
	var waitCh = make(chan struct{})
	var resCh = make(chan struct {
		content [][]string
		err     error
	})

	go func() {
		close(waitCh)
		defer close(resCh)
		r := csv.NewReader(resp.Reader)
		records, err := r.ReadAll()
		resCh <- struct {
			content [][]string
			err     error
		}{
			content: records,
			err:     err,
		}
	}()

	<-waitCh
	select {
	case <-time.After(10 * time.Second):
		log.Println("Failed to read the content in time")
		return points
	case got := <-resCh:
		if err != nil {
			log.Println(err)
			return points
		}
		for _, row := range got.content {
			dbSize, err := strconv.Atoi(row[1])
			if err != nil {
				log.Println(err)
				return points
			}
			var p = point{measurement: conf.DBMeasurement, host: conf.HostTag,
				name: row[0], value: dbSize}
			points = append(points, p)
		}
	}
	return points
}

```
Server Version: 19.03.6
containerd version: b34a5c8af56e510852c35414db4c1f4fa6172339
runc version: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
init version: fec3683

```
u_str  ESTAB      0      0      /var/run/docker.sock 924112                * 0                     users:(("dockerd",pid=21170,fd=25))
u_str  ESTAB      0      0      /var/run/docker.sock 892325                * 0                     users:(("dockerd",pid=21170,fd=19))
```

After reproducton, hung when try to remove the container forcely or restart the docker service.
 ```
docker rm -f <container-id>
systemctl restart docker.service
```
Only fixed by 
```
 kill -9 $(pidof containerd-shim)
```

As the stack shows, some deadlock happened.

Two new generated goroutine trying to lock (0xc000dc9000)

```
goroutine 423 [semacquire, 74 minutes]:
  sync.runtime_SemacquireMutex(0xc000dc9004, 0x5617d98c1400)
          /usr/local/go/src/runtime/sema.go:71 +0x3f
  sync.(*Mutex).Lock(0xc000dc9000)
          /usr/local/go/src/sync/mutex.go:134 +0x10b
  github.com/docker/docker/pkg/broadcaster.(*Unbuffered).Add(0xc000dc9000, 0x5617d9ade600, 0xc000438180)
          /go/src/github.com/docker/docker/pkg/broadcaster/unbuffered.go:16 +0x33
  github.com/docker/docker/container/stream.(*Config).StdoutPipe(0xc000e72140, 0xc0006d6301, 0xc000dfe240)
          /go/src/github.com/docker/docker/container/stream/streams.go:70 +0x50
  github.com/docker/docker/container/stream.(*Config).AttachStreams(0xc000e72140, 0xc000dfe240)
          /go/src/github.com/docker/docker/container/stream/attach.go:51 +0xaa
  github.com/docker/docker/daemon.(*Daemon).ContainerAttach(0xc00000c3c0, 0xc0004ce097, 0x40, 0xc000a46300, 0x0, 0x0)
          /go/src/github.com/docker/docker/daemon/attach.go:51 +0x459
  github.com/docker/docker/api/server/router/container.(*containerRouter).postContainersAttach(0xc0009cde80, 0x5617d9b0c080, 0xc000a46240, 0x5617d9afb380, 0xc00038c000, 0xc000050500, 0xc000a460f0, 0xc000160a01, 0
          /go/src/github.com/docker/docker/api/server/router/container/container_routes.go:602 +0x357
  github.com/docker/docker/api/server/middleware.ExperimentalMiddleware.WrapHandler.func1(0x5617d9b0c080, 0xc000a46240, 0x5617d9afb380, 0xc00038c000, 0xc000050500, 0xc000a460f0, 0x5617d9b0c080, 0xc000a46240)
          /go/src/github.com/docker/docker/api/server/middleware/experimental.go:26 +0x17b
  github.com/docker/docker/api/server/middleware.VersionMiddleware.WrapHandler.func1(0x5617d9b0c080, 0xc000a46180, 0x5617d9afb380, 0xc00038c000, 0xc000050500, 0xc000a460f0, 0x5617d7095cf5, 0xc0001da0c0)
          /go/src/github.com/docker/docker/api/server/middleware/version.go:62 +0x60d
  github.com/docker/docker/pkg/authorization.(*Middleware).WrapHandler.func1(0x5617d9b0c080, 0xc000a46180, 0x5617d9afb380, 0xc00038c000, 0xc000050500, 0xc000a460f0, 0xc0001da0c0, 0x5617d97ab701)
          /go/src/github.com/docker/docker/pkg/authorization/middleware.go:59 +0x82f
  github.com/docker/docker/api/server/middleware.DebugRequestMiddleware.func1(0x5617d9b0c080, 0xc000a46180, 0x5617d9afb380, 0xc00038c000, 0xc000050500, 0xc000a460f0, 0x5617d9b0c080, 0xc000a46180)
          /go/src/github.com/docker/docker/api/server/middleware/debug.go:53 +0x4a4
  github.com/docker/docker/api/server.(*Server).makeHTTPHandler.func1(0x5617d9afb380, 0xc00038c000, 0xc000050400)
          /go/src/github.com/docker/docker/api/server/server.go:142 +0x225
  net/http.HandlerFunc.ServeHTTP(0xc000c965a0, 0x5617d9afb380, 0xc00038c000, 0xc000050400)
          /usr/local/go/src/net/http/server.go:1995 +0x46
  github.com/docker/docker/vendor/github.com/gorilla/mux.(*Router).ServeHTTP(0xc0005a0a80, 0x5617d9afb380, 0xc00038c000, 0xc00059c100)
          /go/src/github.com/docker/docker/vendor/github.com/gorilla/mux/mux.go:212 +0xe5
  github.com/docker/docker/api/server.(*routerSwapper).ServeHTTP(0xc000e639d0, 0x5617d9afb380, 0xc00038c000, 0xc00059c100)
          /go/src/github.com/docker/docker/api/server/router_swapper.go:29 +0x72
  net/http.serverHandler.ServeHTTP(0xc0007dac30, 0x5617d9afb380, 0xc00038c000, 0xc00059c100)
          /usr/local/go/src/net/http/server.go:2774 +0xaa
  net/http.(*conn).serve(0xc000e340a0, 0x5617d9b0bfc0, 0xc0009cc100)
          /usr/local/go/src/net/http/server.go:1878 +0x853
  created by net/http.(*Server).Serve
          /usr/local/go/src/net/http/server.go:2884 +0x2f6
```
```
  goroutine 467 [semacquire, 4 minutes]:
  sync.runtime_SemacquireMutex(0xc000dc9004, 0x5617d98c1400)
          /usr/local/go/src/runtime/sema.go:71 +0x3f
  sync.(*Mutex).Lock(0xc000dc9000)
          /usr/local/go/src/sync/mutex.go:134 +0x10b
  github.com/docker/docker/pkg/broadcaster.(*Unbuffered).Add(0xc000dc9000, 0x5617d9ade600, 0xc000438380)
          /go/src/github.com/docker/docker/pkg/broadcaster/unbuffered.go:16 +0x33
  github.com/docker/docker/container/stream.(*Config).StdoutPipe(0xc000e72140, 0xc0006d6301, 0xc000dfe510)
          /go/src/github.com/docker/docker/container/stream/streams.go:70 +0x50
  github.com/docker/docker/container/stream.(*Config).AttachStreams(0xc000e72140, 0xc000dfe510)
          /go/src/github.com/docker/docker/container/stream/attach.go:51 +0xaa
  github.com/docker/docker/daemon.(*Daemon).ContainerAttach(0xc00000c3c0, 0xc0004ce117, 0x40, 0xc000988450, 0x0, 0x0)
          /go/src/github.com/docker/docker/daemon/attach.go:51 +0x459
  github.com/docker/docker/api/server/router/container.(*containerRouter).postContainersAttach(0xc0009cde80, 0x5617d9b0c080, 0xc000988390, 0x5617d9afb380, 0xc00038c0e0, 0xc000050700, 0xc0009881b0, 0xc0006de001, 0
          /go/src/github.com/docker/docker/api/server/router/container/container_routes.go:602 +0x357
  github.com/docker/docker/api/server/middleware.ExperimentalMiddleware.WrapHandler.func1(0x5617d9b0c080, 0xc000988390, 0x5617d9afb380, 0xc00038c0e0, 0xc000050700, 0xc0009881b0, 0x5617d9b0c080, 0xc000988390)
          /go/src/github.com/docker/docker/api/server/middleware/experimental.go:26 +0x17b
  github.com/docker/docker/api/server/middleware.VersionMiddleware.WrapHandler.func1(0x5617d9b0c080, 0xc0009882a0, 0x5617d9afb380, 0xc00038c0e0, 0xc000050700, 0xc0009881b0, 0x5617d7095cf5, 0xc0001da260)
          /go/src/github.com/docker/docker/api/server/middleware/version.go:62 +0x60d
  github.com/docker/docker/pkg/authorization.(*Middleware).WrapHandler.func1(0x5617d9b0c080, 0xc0009882a0, 0x5617d9afb380, 0xc00038c0e0, 0xc000050700, 0xc0009881b0, 0xc0001da260, 0x5617d97ab701)
          /go/src/github.com/docker/docker/pkg/authorization/middleware.go:59 +0x82f
  github.com/docker/docker/api/server/middleware.DebugRequestMiddleware.func1(0x5617d9b0c080, 0xc0009882a0, 0x5617d9afb380, 0xc00038c0e0, 0xc000050700, 0xc0009881b0, 0x5617d9b0c080, 0xc0009882a0)
          /go/src/github.com/docker/docker/api/server/middleware/debug.go:53 +0x4a4
  github.com/docker/docker/api/server.(*Server).makeHTTPHandler.func1(0x5617d9afb380, 0xc00038c0e0, 0xc000050600)
          /go/src/github.com/docker/docker/api/server/server.go:142 +0x225
  net/http.HandlerFunc.ServeHTTP(0xc000c965a0, 0x5617d9afb380, 0xc00038c0e0, 0xc000050600)
          /usr/local/go/src/net/http/server.go:1995 +0x46
  github.com/docker/docker/vendor/github.com/gorilla/mux.(*Router).ServeHTTP(0xc0005a0a80, 0x5617d9afb380, 0xc00038c0e0, 0xc000050300)
          /go/src/github.com/docker/docker/vendor/github.com/gorilla/mux/mux.go:212 +0xe5
  github.com/docker/docker/api/server.(*routerSwapper).ServeHTTP(0xc000e639d0, 0x5617d9afb380, 0xc00038c0e0, 0xc000050300)
          /go/src/github.com/docker/docker/api/server/router_swapper.go:29 +0x72
  net/http.serverHandler.ServeHTTP(0xc0007dac30, 0x5617d9afb380, 0xc00038c0e0, 0xc000050300)
          /usr/local/go/src/net/http/server.go:2774 +0xaa
  net/http.(*conn).serve(0xc000e5e000, 0x5617d9b0bfc0, 0xc0004380c0)
          /usr/local/go/src/net/http/server.go:1878 +0x853
  created by net/http.(*Server).Serve
          /usr/local/go/src/net/http/server.go:2884 +0x2f6
```

This goroutine already existed before reproduction.  address (0xc000dc9000) used when pool.Copy.
```
goroutine 438 [sync.Cond.Wait, 82 minutes]:
  runtime.goparkunlock(...)
          /usr/local/go/src/runtime/proc.go:307
  sync.runtime_notifyListWait(0xc000e6a150, 0x561700000000)
          /usr/local/go/src/runtime/sema.go:510 +0xfd
  sync.(*Cond).Wait(0xc000e6a140)
          /usr/local/go/src/sync/cond.go:56 +0xa0
  github.com/docker/docker/pkg/ioutils.(*BytesPipe).Write(0xc000e6a100, 0xc000cee009, 0x72, 0x7ff7, 0x7b, 0x0, 0x0)
          /go/src/github.com/docker/docker/pkg/ioutils/bytespipe.go:89 +0xad
  github.com/docker/docker/pkg/broadcaster.(*Unbuffered).Write(0xc000dc9000, 0xc000cee000, 0x7b, 0x8000, 0x7b, 0x0, 0x0)
          /go/src/github.com/docker/docker/pkg/broadcaster/unbuffered.go:27 +0xdf
  io.copyBuffer(0x5617d9ab0f40, 0xc000dc9000, 0x7ffb16104028, 0xc000097560, 0xc000cee000, 0x8000, 0x8000, 0x5438312d34302d37, 0x0, 0x7ffb16104028)
          /usr/local/go/src/io/io.go:404 +0x1fd
  io.CopyBuffer(...)
          /usr/local/go/src/io/io.go:375
  github.com/docker/docker/pkg/pools.Copy(0x5617d9ab0f40, 0xc000dc9000, 0x7ffb16104028, 0xc000097560, 0xc000097560, 0x1, 0x5617d70bfdc8)
          /go/src/github.com/docker/docker/pkg/pools/pools.go:81 +0xd1
  github.com/docker/docker/container/stream.(*Config).CopyToPipe.func1.1(0x5617d9ab0f40, 0xc000dc9000, 0x7ffb16103f68, 0xc000097560, 0xc000e72140)
          /go/src/github.com/docker/docker/container/stream/streams.go:124 +0x73
  created by github.com/docker/docker/container/stream.(*Config).CopyToPipe.func1
          /go/src/github.com/docker/docker/container/stream/streams.go:123 +0x86
```



dockerd version 18.09.1
containerd version: 9754871865f7fe2f4e74d43e2fc7ccd237edcbce

Find fd leaking in running server, and the stack shows all goroutines waiting for container stopped.
However, most of containers waited are already stopped.

```
Containers: 432
 Running: 402
 Paused: 0
 Stopped: 30
```

```
# ss -xp > ss.0313.log
# cat ss.0313.log | grep 'docker.sock' |  grep '* 0' | wc -l
42699
```

```
# cat /var/run/docker/goroutine-stacks-2020-03-13T153948+0800.log | grep '(*Daemon).Kill' | wc -l
42747
```

```
goroutine 45018 [chan receive, 5940 minutes]:
github.com/docker/docker/daemon.(*Daemon).Kill(0xc4208281e0, 0xc42164a000, 0x0, 0x0)
        /go/src/github.com/docker/docker/daemon/kill.go:162 +0x215
github.com/docker/docker/daemon.(*Daemon).containerStop(0xc4208281e0, 0xc42164a000, 0x1e, 0x0, 0x0)
        /go/src/github.com/docker/docker/daemon/stop.go:80 +0x2b2
github.com/docker/docker/daemon.(*Daemon).ContainerStop(0xc4208281e0, 0xc424e4c397, 0x40, 0xc4225db190, 0xc425a9f4d8, 0xc424b502c0)
        /go/src/github.com/docker/docker/daemon/stop.go:33 +0xc7
github.com/docker/docker/api/server/router/container.(*containerRouter).postContainersStop(0xc4234a9180, 0x5572dedad960, 0xc427dc5860, 0x5572dedabd60, 0xc423118d20, 0xc429611800, 0xc427dc5740, 0x5572de11ecfb, 0x5)
        /go/src/github.com/docker/docker/api/server/router/container/container_routes.go:225 +0xf2
github.com/docker/docker/api/server/router/container.(*containerRouter).(github.com/docker/docker/api/server/router/container.postContainersStop)-fm(0x5572dedad960, 0xc427dc5860, 0x5572dedabd60, 0xc423118d20, 0xc429611800, 0xc427dc5740, 0x5572dcc5f7cc, 0x5572debe98e0)
        /go/src/github.com/docker/docker/api/server/router/container/container.go:53 +0x6b
github.com/docker/docker/api/server/middleware.ExperimentalMiddleware.WrapHandler.func1(0x5572dedad960, 0xc427dc5860, 0x5572dedabd60, 0xc423118d20, 0xc429611800, 0xc427dc5740, 0x5572dedad960, 0xc427dc5860)
        /go/src/github.com/docker/docker/api/server/middleware/experimental.go:26 +0xda
github.com/docker/docker/api/server/middleware.VersionMiddleware.WrapHandler.func1(0x5572dedad960, 0xc427dc5800, 0x5572dedabd60, 0xc423118d20, 0xc429611800, 0xc427dc5740, 0x18, 0xc425fa58f8)
        /go/src/github.com/docker/docker/api/server/middleware/version.go:62 +0x401
github.com/docker/docker/pkg/authorization.(*Middleware).WrapHandler.func1(0x5572dedad960, 0xc427dc5800, 0x5572dedabd60, 0xc423118d20, 0xc429611800, 0xc427dc5740, 0x5572dcb282d7, 0x5572de2f3e20)
        /go/src/github.com/docker/docker/pkg/authorization/middleware.go:59 +0x7ab
github.com/docker/docker/api/server/middleware.DebugRequestMiddleware.func1(0x5572dedad960, 0xc427dc5800, 0x5572dedabd60, 0xc423118d20, 0xc429611800, 0xc427dc5740, 0x5572dedad960, 0xc427dc5800)
        /go/src/github.com/docker/docker/api/server/middleware/debug.go:25 +0x8db
github.com/docker/docker/api/server.(*Server).makeHTTPHandler.func1(0x5572dedabd60, 0xc423118d20, 0xc429611800)
        /go/src/github.com/docker/docker/api/server/server.go:141 +0x19a
net/http.HandlerFunc.ServeHTTP(0xc422154b60, 0x5572dedabd60, 0xc423118d20, 0xc429611800)
        /usr/local/go/src/net/http/server.go:1947 +0x46
github.com/docker/docker/vendor/github.com/gorilla/mux.(*Router).ServeHTTP(0xc42295e910, 0x5572dedabd60, 0xc423118d20, 0xc429611800)
        /go/src/github.com/docker/docker/vendor/github.com/gorilla/mux/mux.go:103 +0x228
github.com/docker/docker/api/server.(*routerSwapper).ServeHTTP(0xc421d0e8e0, 0x5572dedabd60, 0xc423118d20, 0xc429611800)
        /go/src/github.com/docker/docker/api/server/router_swapper.go:29 +0x72
net/http.serverHandler.ServeHTTP(0xc420ab7930, 0x5572dedabd60, 0xc423118d20, 0xc429611800)
        /usr/local/go/src/net/http/server.go:2697 +0xbe
net/http.(*conn).serve(0xc4221c15e0, 0x5572dedad8a0, 0xc423165ac0)
        /usr/local/go/src/net/http/server.go:1830 +0x653
created by net/http.(*Server).Serve
        /usr/local/go/src/net/http/server.go:2798 +0x27d
```
Nice finds!

Hang is here:

https://github.com/moby/moby/blob/master/daemon/kill.go#L161

I wonder if the condition of being stopped is not set properly?


On Fri, Mar 13, 2020 at 4:19 AM richard-tai <notifications@github.com>
wrote:

> dockerd version 18.09.1
> containerd version: 9754871865f7fe2f4e74d43e2fc7ccd237edcbce
>
> Find fd leaking in running server, and the stack shows all goroutines
> waiting for container stopped.
> However, most of containers waited are already stopped.
>
> Containers: 432
>  Running: 402
>  Paused: 0
>  Stopped: 30
>
> # ss -xp > ss.0313.log
> # cat ss.0313.log | grep 'docker.sock' |  grep '* 0' | wc -l
> 42699
>
> # cat /var/run/docker/goroutine-stacks-2020-03-13T153948+0800.log | grep '(*Daemon).Kill' | wc -l
> 42747
>
> goroutine 45018 [chan receive, 5940 minutes]:github.com/docker/docker/daemon.(*Daemon).Kill(0xc4208281e0, 0xc42164a000, 0x0, 0x0)
>         /go/src/github.com/docker/docker/daemon/kill.go:162 +0x215github.com/docker/docker/daemon.(*Daemon).containerStop(0xc4208281e0, 0xc42164a000, 0x1e, 0x0, 0x0)
>         /go/src/github.com/docker/docker/daemon/stop.go:80 +0x2b2github.com/docker/docker/daemon.(*Daemon).ContainerStop(0xc4208281e0, 0xc424e4c397, 0x40, 0xc4225db190, 0xc425a9f4d8, 0xc424b502c0)
>         /go/src/github.com/docker/docker/daemon/stop.go:33 +0xc7github.com/docker/docker/api/server/router/container.(*containerRouter).postContainersStop(0xc4234a9180, 0x5572dedad960, 0xc427dc5860, 0x5572dedabd60, 0xc423118d20, 0xc429611800, 0xc427dc5740, 0x5572de11ecfb, 0x5)
>         /go/src/github.com/docker/docker/api/server/router/container/container_routes.go:225 +0xf2github.com/docker/docker/api/server/router/container.(*containerRouter).(github.com/docker/docker/api/server/router/container.postContainersStop)-fm(0x5572dedad960, 0xc427dc5860, 0x5572dedabd60, 0xc423118d20, 0xc429611800, 0xc427dc5740, 0x5572dcc5f7cc, 0x5572debe98e0)
>         /go/src/github.com/docker/docker/api/server/router/container/container.go:53 +0x6bgithub.com/docker/docker/api/server/middleware.ExperimentalMiddleware.WrapHandler.func1(0x5572dedad960, 0xc427dc5860, 0x5572dedabd60, 0xc423118d20, 0xc429611800, 0xc427dc5740, 0x5572dedad960, 0xc427dc5860)
>         /go/src/github.com/docker/docker/api/server/middleware/experimental.go:26 +0xdagithub.com/docker/docker/api/server/middleware.VersionMiddleware.WrapHandler.func1(0x5572dedad960, 0xc427dc5800, 0x5572dedabd60, 0xc423118d20, 0xc429611800, 0xc427dc5740, 0x18, 0xc425fa58f8)
>         /go/src/github.com/docker/docker/api/server/middleware/version.go:62 +0x401github.com/docker/docker/pkg/authorization.(*Middleware).WrapHandler.func1(0x5572dedad960, 0xc427dc5800, 0x5572dedabd60, 0xc423118d20, 0xc429611800, 0xc427dc5740, 0x5572dcb282d7, 0x5572de2f3e20)
>         /go/src/github.com/docker/docker/pkg/authorization/middleware.go:59 +0x7abgithub.com/docker/docker/api/server/middleware.DebugRequestMiddleware.func1(0x5572dedad960, 0xc427dc5800, 0x5572dedabd60, 0xc423118d20, 0xc429611800, 0xc427dc5740, 0x5572dedad960, 0xc427dc5800)
>         /go/src/github.com/docker/docker/api/server/middleware/debug.go:25 +0x8dbgithub.com/docker/docker/api/server.(*Server).makeHTTPHandler.func1(0x5572dedabd60, 0xc423118d20, 0xc429611800)
>         /go/src/github.com/docker/docker/api/server/server.go:141 +0x19a
> net/http.HandlerFunc.ServeHTTP(0xc422154b60, 0x5572dedabd60, 0xc423118d20, 0xc429611800)
>         /usr/local/go/src/net/http/server.go:1947 +0x46github.com/docker/docker/vendor/github.com/gorilla/mux.(*Router).ServeHTTP(0xc42295e910, 0x5572dedabd60, 0xc423118d20, 0xc429611800)
>         /go/src/github.com/docker/docker/vendor/github.com/gorilla/mux/mux.go:103 +0x228github.com/docker/docker/api/server.(*routerSwapper).ServeHTTP(0xc421d0e8e0, 0x5572dedabd60, 0xc423118d20, 0xc429611800)
>         /go/src/github.com/docker/docker/api/server/router_swapper.go:29 +0x72
> net/http.serverHandler.ServeHTTP(0xc420ab7930, 0x5572dedabd60, 0xc423118d20, 0xc429611800)
>         /usr/local/go/src/net/http/server.go:2697 +0xbe
> net/http.(*conn).serve(0xc4221c15e0, 0x5572dedad8a0, 0xc423165ac0)
>         /usr/local/go/src/net/http/server.go:1830 +0x653
> created by net/http.(*Server).Serve
>         /usr/local/go/src/net/http/server.go:2798 +0x27d
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/moby/moby/issues/37182#issuecomment-598673674>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAAET2ZZLSECATQIBF24GT3RHIJDTANCNFSM4FCR7NTQ>
> .
>

That line https://github.com/moby/moby/blob/da574f93432e600fda561da5e6983e7f69b364a9/daemon/kill.go#L161

Uses https://github.com/moby/moby/blob/ad1b781e44fa1e44b9e654e5078929aec56aed66/container/state.go#L181-L185

Some observations;

It's checking `s.Running`, which is not mutually exclusive with `s.Paused` (could be an issue if the container is paused?); https://github.com/moby/moby/blob/ad1b781e44fa1e44b9e654e5078929aec56aed66/container/state.go#L19-L24

It takes a lock both at the start:

https://github.com/moby/moby/blob/ad1b781e44fa1e44b9e654e5078929aec56aed66/container/state.go#L182-L183

And in a go routine further down;

https://github.com/moby/moby/blob/ad1b781e44fa1e44b9e654e5078929aec56aed66/container/state.go#L225-L230

Is that an issue? 🤔 
Those locks came in the same function after https://github.com/moby/moby/commit/cfdf84d5d04c8ee656e5c4ad3db993c258e52674, which combined `WaitStop` and `WaitWithContext` into a single `Wait` function.

Before that, `WaitStop` did not take a lock; https://github.com/moby/moby/blob/1290ec2d4b3611293cd993cd12fd9e78e268e296/container/state.go#L179

But `WaitWithContext` did; https://github.com/moby/moby/blob/1290ec2d4b3611293cd993cd12fd9e78e268e296/container/state.go#L197
> dockerd version 18.09.1
> containerd version: 9754871865f7fe2f4e74d43e2fc7ccd237edcbce
> 
> Find fd leaking in running server, and the stack shows all goroutines waiting for container stopped.
> However, most of containers waited are already stopped.
> 
> ```
> Containers: 432
>  Running: 402
>  Paused: 0
>  Stopped: 30
> ```
> 
> ```
> # ss -xp > ss.0313.log
> # cat ss.0313.log | grep 'docker.sock' |  grep '* 0' | wc -l
> 42699
> ```
> 
> ```
> # cat /var/run/docker/goroutine-stacks-2020-03-13T153948+0800.log | grep '(*Daemon).Kill' | wc -l
> 42747
> ```
> 
> ```
> goroutine 45018 [chan receive, 5940 minutes]:
> github.com/docker/docker/daemon.(*Daemon).Kill(0xc4208281e0, 0xc42164a000, 0x0, 0x0)
>         /go/src/github.com/docker/docker/daemon/kill.go:162 +0x215
> github.com/docker/docker/daemon.(*Daemon).containerStop(0xc4208281e0, 0xc42164a000, 0x1e, 0x0, 0x0)
>         /go/src/github.com/docker/docker/daemon/stop.go:80 +0x2b2
> github.com/docker/docker/daemon.(*Daemon).ContainerStop(0xc4208281e0, 0xc424e4c397, 0x40, 0xc4225db190, 0xc425a9f4d8, 0xc424b502c0)
>         /go/src/github.com/docker/docker/daemon/stop.go:33 +0xc7
> github.com/docker/docker/api/server/router/container.(*containerRouter).postContainersStop(0xc4234a9180, 0x5572dedad960, 0xc427dc5860, 0x5572dedabd60, 0xc423118d20, 0xc429611800, 0xc427dc5740, 0x5572de11ecfb, 0x5)
>         /go/src/github.com/docker/docker/api/server/router/container/container_routes.go:225 +0xf2
> github.com/docker/docker/api/server/router/container.(*containerRouter).(github.com/docker/docker/api/server/router/container.postContainersStop)-fm(0x5572dedad960, 0xc427dc5860, 0x5572dedabd60, 0xc423118d20, 0xc429611800, 0xc427dc5740, 0x5572dcc5f7cc, 0x5572debe98e0)
>         /go/src/github.com/docker/docker/api/server/router/container/container.go:53 +0x6b
> github.com/docker/docker/api/server/middleware.ExperimentalMiddleware.WrapHandler.func1(0x5572dedad960, 0xc427dc5860, 0x5572dedabd60, 0xc423118d20, 0xc429611800, 0xc427dc5740, 0x5572dedad960, 0xc427dc5860)
>         /go/src/github.com/docker/docker/api/server/middleware/experimental.go:26 +0xda
> github.com/docker/docker/api/server/middleware.VersionMiddleware.WrapHandler.func1(0x5572dedad960, 0xc427dc5800, 0x5572dedabd60, 0xc423118d20, 0xc429611800, 0xc427dc5740, 0x18, 0xc425fa58f8)
>         /go/src/github.com/docker/docker/api/server/middleware/version.go:62 +0x401
> github.com/docker/docker/pkg/authorization.(*Middleware).WrapHandler.func1(0x5572dedad960, 0xc427dc5800, 0x5572dedabd60, 0xc423118d20, 0xc429611800, 0xc427dc5740, 0x5572dcb282d7, 0x5572de2f3e20)
>         /go/src/github.com/docker/docker/pkg/authorization/middleware.go:59 +0x7ab
> github.com/docker/docker/api/server/middleware.DebugRequestMiddleware.func1(0x5572dedad960, 0xc427dc5800, 0x5572dedabd60, 0xc423118d20, 0xc429611800, 0xc427dc5740, 0x5572dedad960, 0xc427dc5800)
>         /go/src/github.com/docker/docker/api/server/middleware/debug.go:25 +0x8db
> github.com/docker/docker/api/server.(*Server).makeHTTPHandler.func1(0x5572dedabd60, 0xc423118d20, 0xc429611800)
>         /go/src/github.com/docker/docker/api/server/server.go:141 +0x19a
> net/http.HandlerFunc.ServeHTTP(0xc422154b60, 0x5572dedabd60, 0xc423118d20, 0xc429611800)
>         /usr/local/go/src/net/http/server.go:1947 +0x46
> github.com/docker/docker/vendor/github.com/gorilla/mux.(*Router).ServeHTTP(0xc42295e910, 0x5572dedabd60, 0xc423118d20, 0xc429611800)
>         /go/src/github.com/docker/docker/vendor/github.com/gorilla/mux/mux.go:103 +0x228
> github.com/docker/docker/api/server.(*routerSwapper).ServeHTTP(0xc421d0e8e0, 0x5572dedabd60, 0xc423118d20, 0xc429611800)
>         /go/src/github.com/docker/docker/api/server/router_swapper.go:29 +0x72
> net/http.serverHandler.ServeHTTP(0xc420ab7930, 0x5572dedabd60, 0xc423118d20, 0xc429611800)
>         /usr/local/go/src/net/http/server.go:2697 +0xbe
> net/http.(*conn).serve(0xc4221c15e0, 0x5572dedad8a0, 0xc423165ac0)
>         /usr/local/go/src/net/http/server.go:1830 +0x653
> created by net/http.(*Server).Serve
>         /usr/local/go/src/net/http/server.go:2798 +0x27d
> ```

D status process in container is the root cause,
and the process can't be terminated by signal.
So dockerd just waiting for ever after issue stopping container command.
find in 18.09.9 if kubelet try to delete containter that main process is defunct, so container stop failed. The kubelet retry to delete this container, then many connection leaked. kill the containerd-shim reslove the issue.

```
# ps -ef |grep 1024916
root     1024916    1620  0 Nov26 ?        00:02:07 containerd-shim -namespace moby -workdir /data/kubernetes/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/75cb6523461e4f5edd2fb52f551ab5f3a38c5241467ef33c53e9916a588d5e98 -address /var/run/docker/containerd/containerd.sock -containerd-binary /usr/local/bin/containerd -runtime-root /var/run/docker/runtime-runc
root     1024937 1024916  9 Nov26 ?        1-06:21:00 [prometheus] <defunct>
root     1042969  712407  0 13:48 pts/0    00:00:00 grep --color=auto 1024916
```

```
 ss -p state established |grep -v Recv-Q | less
u_str  0      0      @/containerd-shim/moby/804c7ff85b5e2201c6c9d5819309c55d63bc645729dccb07775f297a67db10f6/shim.sock@ -1442658938                * -1442661467           users:(("containerd-shim",pid=680421,fd=7))
u_str  0      0      /var/run/docker.sock -1445160328                * 0                     users:(("dockerd",pid=1399,fd=5352))
u_str  0      0      /var/run/docker.sock -1457809475                * 0                     users:(("dockerd",pid=1399,fd=5440))
u_str  0      0      /var/run/docker.sock -1470930645                * 0                     users:(("dockerd",pid=1399,fd=5189))
u_str  0      0      /var/run/docker.sock -1477971491                * 0                     users:(("dockerd",pid=1399,fd=5062))
u_str  0      0      /var/run/docker.sock -1478786239                * 0                     users:(("dockerd",pid=1399,fd=5038))
u_str  0      0      /var/run/docker.sock -1485051053                * 0                     users:(("dockerd",pid=1399,fd=4965))
u_str  0      0      /var/run/docker.sock -1490895638                * 0                     users:(("dockerd",pid=1399,fd=4695))
u_str  0      0      /var/run/docker.sock -1498479204                * 0                     users:(("dockerd",pid=1399,fd=4765))
u_str  0      0      /var/run/docker.sock -1522875673                * 0                     users:(("dockerd",pid=1399,fd=162))
u_str  0      0      /var/run/docker.sock -1557086438                * 0                     users:(("dockerd",pid=1399,fd=931))
u_str  0      0      /var/run/docker.sock -1560995938                * 0                     users:(("dockerd",pid=1399,fd=3833))
u_str  0      0      /var/run/docker.sock -1579938717                * 0                     users:(("dockerd",pid=1399,fd=3532))
u_str  0      0      /var/run/docker.sock -1602443767                * 0                     users:(("dockerd",pid=1399,fd=3198))
u_str  0      0      /var/run/docker.sock -1612112200                * 0                     users:(("dockerd",pid=1399,fd=3041))
u_str  0      0      /var/run/docker.sock -1628669968                * 0                     users:(("dockerd",pid=1399,fd=2773))
u_str  0      0      /var/run/docker.sock -1635379730                * 0                     users:(("dockerd",pid=1399,fd=2685))
u_str  0      0      /var/run/docker.sock -1644389877                * 0                     users:(("dockerd",pid=1399,fd=2571))
u_str  0      0      /var/run/docker.sock -1646173758                * 0                     users:(("dockerd",pid=1399,fd=2533))
u_str  0      0            * -1694637577                * -1694607304           users:(("containerd",pid=1620,fd=174))
u_str  0      0      /var/run/docker.sock -1703647521                * 0                     users:(("dockerd",pid=1399,fd=1616))
u_str  0      0      /var/run/docker.sock -1711287579                * 0                     users:(("dockerd",pid=1399,fd=264))
u_str  0      0            * 24091                      * 24092                 users:(("master",pid=2326,fd=60))
u_str  0      0      /var/run/docker.sock -1427998049                * 0                     users:(("dockerd",pid=1399,fd=5884))
u_str  0      0      /var/run/docker.sock -1440158554                * 0                     users:(("dockerd",pid=1399,fd=4591))
u_str  0      0      /var/run/docker.sock -1444971680                * 0                     users:(("dockerd",pid=1399,fd=5619))
u_str  0      0      /var/run/docker.sock -1449345351                * 0                     users:(("dockerd",pid=1399,fd=1013))
u_str  0      0      /var/run/docker.sock -1456156913                * 0                     users:(("dockerd",pid=1399,fd=5458))
u_str  0      0      /var/run/docker.sock -1468547132                * 0                     users:(("dockerd",pid=1399,fd=5223))
u_str  0      0      /var/run/docker.sock -1472477986                * 0                     users:(("dockerd",pid=1399,fd=5130))
u_str  0      0      /var/run/docker.sock -1521985603                * 0                     users:(("dockerd",pid=1399,fd=4395))
...skipping...
u_str  0      0      @/containerd-shim/moby/75cb6523461e4f5edd2fb52f551ab5f3a38c5241467ef33c53e9916a588d5e98/shim.sock@ -2036027232                * -2036033846           users:(("containerd-shim",pid=1024916,fd=7))
u_str  0      0            * 1056731                    * 1056732               users:(("containerd",pid=1620,fd=411))
u_str  0      0      /var/run/docker.sock -1411489962                * 0                     users:(("dockerd",pid=1399,fd=6140))
u_str  0      0      /var/run/docker.sock -1448687723                * 0                     users:(("dockerd",pid=1399,fd=4858))
u_str  0      0      /var/run/docker.sock -1450468848                * 0                     users:(("dockerd",pid=1399,fd=5546))
u_str  0      0      /var/run/docker.sock -1468025317                * 0                     users:(("dockerd",pid=1399,fd=5242))
u_str  0      0      /var/run/docker.sock -1480843521                * 0                     users:(("dockerd",pid=1399,fd=5020))
u_str  0      0      /var/run/docker.sock -1497948181                * 0                     users:(("dockerd",pid=1399,fd=4764))
u_str  0      0      /var/run/docker.sock -1525701864                * 0                     users:(("dockerd",pid=1399,fd=4351))
u_str  0      0      /var/run/docker.sock -1531970440                * 0                     users:(("dockerd",pid=1399,fd=4254))
u_str  0      0      /var/run/docker.sock -1542344396                * 0                     users:(("dockerd",pid=1399,fd=4097))
u_str  0      0      /var/run/docker.sock -1586881258                * 0                     users:(("dockerd",pid=1399,fd=3435))
u_str  0      0      /var/run/docker.sock -1596263762                * 0                     users:(("dockerd",pid=1399,fd=3291))
u_str  0      0      /var/run/docker.sock -1624366434                * 0                     users:(("dockerd",pid=1399,fd=2710))
u_str  0      0      /var/run/docker.sock -1630769569                * 0                     users:(("dockerd",pid=1399,fd=2732))
u_str  0      0      /var/run/docker.sock -1640559417                * 0                     users:(("dockerd",pid=1399,fd=821))
u_str  0      0      /var/run/docker.sock -1656963197                * 0                     users:(("dockerd",pid=1399,fd=2358))
u_str  0      0      /var/run/docker.sock -1666841009                * 0                     users:(("dockerd",pid=1399,fd=2180))
u_str  0      0      @/containerd-shim/moby/804c7ff85b5e2201c6c9d5819309c55d63bc645729dccb07775f297a67db10f6/shim.sock@ -1442658938  
              * -1442661467           users:(("containerd-shim",pid=680421,fd=7))
u_str  0      0      /var/run/docker.sock -1445160328                * 0                     users:(("dockerd",pid=1399,fd=5352))
u_str  0      0      /var/run/docker.sock -1457809475                * 0                     users:(("dockerd",pid=1399,fd=5440))
u_str  0      0      /var/run/docker.sock -1470930645                * 0                     users:(("dockerd",pid=1399,fd=5189))
```

kubelet log
```
E1202 16:57:53.919198 1029994 cpu_manager.go:418] [cpumanager] reconcileState: failed to update container (pod: prometheus-79dfbb765c
-qjmth, container: prometheus, container id: 75cb6523461e4f5edd2fb52f551ab5f3a38c5241467ef33c53e9916a588d5e98, cpuset: "0-31", error:
 rpc error: code = Unknown desc = failed to update container "75cb6523461e4f5edd2fb52f551ab5f3a38c5241467ef33c53e9916a588d5e98": Erro
r response from daemon: Cannot update container 75cb6523461e4f5edd2fb52f551ab5f3a38c5241467ef33c53e9916a588d5e98: runc did not termin
ate sucessfully: container not running
: unknown)
I1202 16:57:53.941201 1029994 container_manager_linux.go:470] [ContainerManager]: Discovered runtime cgroups name: /system.slice/dock
er.service
E1202 16:57:56.617679 1029994 remote_runtime.go:243] StopContainer "75cb6523461e4f5edd2fb52f551ab5f3a38c5241467ef33c53e9916a588d5e98"
 from runtime service failed: rpc error: code = Unknown desc = operation timeout: context deadline exceeded
E1202 16:57:56.617718 1029994 kuberuntime_container.go:652] Container "docker://75cb6523461e4f5edd2fb52f551ab5f3a38c5241467ef33c53e99
16a588d5e98" termination failed with gracePeriod 30: rpc error: code = Unknown desc = operation timeout: context deadline exceeded
E1202 16:57:56.854644 1029994 kubelet.go:1576] error killing pod: failed to "KillContainer" for "prometheus" with KillContainerError:
 "rpc error: code = Unknown desc = operation timeout: context deadline exceeded"
E1202 16:57:56.854674 1029994 pod_workers.go:191] Error syncing pod d455027d-7ffa-4403-8e60-7eccd5000eb4 ("prometheus-79dfbb765c-qjmt
h_prometheus(d455027d-7ffa-4403-8e60-7eccd5000eb4)"), skipping: error killing pod: failed to "KillContainer" for "prometheus" with Ki
llContainerError: "rpc error: code = Unknown desc = operation timeout: context deadline exceeded"
```

dockerd log

```
Nov 26 18:38:47 sh-saas-k8s1-node-qa-06 dockerd[1399]: time="2020-11-26T18:38:47.094516634+08:00" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/75cb6523461e4f5edd2fb52f551ab5f3a38c5241467ef33c53e9916a588d5e98/shim.sock" debug=false pid=1024916
Dec 02 16:56:26 sh-saas-k8s1-node-qa-06 dockerd[1399]: time="2020-12-02T16:56:26.626575595+08:00" level=info msg="Container 75cb6523461e4f5edd2fb52f551ab5f3a38c5241467ef33c53e9916a588d5e98 failed to exit within 30 seconds of signal 15 - using the force"
Dec 02 16:56:38 sh-saas-k8s1-node-qa-06 dockerd[1399]: time="2020-12-02T16:56:38.964532744+08:00" level=error msg="Handler for POST /containers/75cb6523461e4f5edd2fb52f551ab5f3a38c5241467ef33c53e9916a588d5e98/update returned error: Cannot update container 75cb6523461e4f5edd2fb52f551ab5f3a38c5241467ef33c53e9916a588d5e98: runc did not terminate sucessfully: container not running\n: unknown"
Dec 02 16:56:52 sh-saas-k8s1-node-qa-06 dockerd[1399]: time="2020-12-02T16:56:52.288153031+08:00" level=error msg="Handler for POST /containers/75cb6523461e4f5edd2fb52f551ab5f3a38c5241467ef33c53e9916a588d5e98/update returned error: Cannot update container 75cb6523461e4f5edd2fb52f551ab5f3a38c5241467ef33c53e9916a588d5e98: runc did not terminate sucessfully: container not running\n: unknown"
Dec 02 16:57:04 sh-saas-k8s1-node-qa-06 dockerd[1399]: time="2020-12-02T16:57:04.404765270+08:00" level=error msg="Handler for POST /containers/75cb6523461e4f5edd2fb52f551ab5f3a38c5241467ef33c53e9916a588d5e98/update returned error: Cannot update container 75cb6523461e4f5edd2fb52f551ab5f3a38c5241467ef33c53e9916a588d5e98: runc did not terminate sucessfully: container not running\n: unknown"
Dec 02 16:57:17 sh-saas-k8s1-node-qa-06 dockerd[1399]: time="2020-12-02T16:57:17.137132720+08:00" level=error msg="Handler for POST /containers/75cb6523461e4f5edd2fb52f551ab5f3a38c5241467ef33c53e9916a588d5e98/update returned error: Cannot update container 75cb652346

Dec 09 13:42:56 sh-saas-k8s1-node-qa-06 dockerd[1399]: time="2020-12-09T13:42:56.301194278+08:00" level=info msg="Container 75cb65234
61e4f5edd2fb52f551ab5f3a38c5241467ef33c53e9916a588d5e98 failed to exit within 30 seconds of signal 15 - using the force"
Dec 09 13:43:06 sh-saas-k8s1-node-qa-06 dockerd[1399]: time="2020-12-09T13:43:06.321949778+08:00" level=info msg="Container 75cb65234
61e failed to exit within 10 seconds of kill - trying direct SIGKILL"
```
On Tue Dec 8, 2020 at 11:42 PM PST, wu0407 wrote:
> find in 18.09.9 if kubelet try to delete containter that main process is
> <defunct>, so container stop failed. The kubelet retry to delete this
> container, then many connection leaked. kill the containerd-shim reslove
> the issue.

Sounds like something isn't waiting at the right time.

I think https://github.com/moby/moby/pull/42019 will solve the fd leaking issue.
Feel free to give it a try.
Is there any movement on this issue?
dockerd kept leaking file descriptors and eating up memory on our server over a period of 2 months.
I think https://github.com/docker/docker-ce/commit/3a6c414d8b42540a63b2dac448756a5c454dbf4d will solve the fd leaking issue.

v20.10.10
Thanks for your comments.
Unfortunately, I am using version 20.10.17, so the response you gave me does not seem to cure the file descriptor leak.

```
# docker version
Client:
 Version:           20.10.17
 API version:       1.41
 Go version:        go1.18.3
 Git commit:        100c701
 Built:             Thu Jun 16 20:08:47 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          20.10.17
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.3
  Git commit:       a89b842
  Built:            Thu Jun 16 20:09:24 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.6
  GitCommit:        10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1
 runc:
  Version:          1.1.3
  GitCommit:        1e7bb5b773162b57333d57f612fd72e3f8612d94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
```

Fix patch:
diff --git a/api/server/router/container/container_routes.go b/api/server/router/container/container_routes.go
index 3872028652f86..2b1ec0d5f05de 100644
--- a/api/server/router/container/container_routes.go
+++ b/api/server/router/container/container_routes.go
@@ -775,7 +775,7 @@ func (s *containerRouter) postContainersAttach(ctx context.Context, w http.Respo
 	}
 
 	contentType := types.MediaTypeRawStream
-	setupStreams := func(multiplexed bool) (io.ReadCloser, io.Writer, io.Writer, error) {
+	setupStreams := func(multiplexed bool, cancel func()) (io.ReadCloser, io.Writer, io.Writer, error) {
 		conn, _, err := hijacker.Hijack()
 		if err != nil {
 			return nil, nil, nil, err
@@ -793,6 +793,8 @@ func (s *containerRouter) postContainersAttach(ctx context.Context, w http.Respo
 			fmt.Fprintf(conn, "HTTP/1.1 200 OK\r\nContent-Type: application/vnd.docker.raw-stream\r\n\r\n")
 		}
 
+		go notifyClosed(ctx, conn, cancel)
+
 		closer := func() error {
 			httputils.CloseStreams(conn)
 			return nil
@@ -841,7 +843,7 @@ func (s *containerRouter) wsContainersAttach(ctx context.Context, w http.Respons
 
 	version := httputils.VersionFromContext(ctx)
 
-	setupStreams := func(multiplexed bool) (io.ReadCloser, io.Writer, io.Writer, error) {
+	setupStreams := func(multiplexed bool, cancel func()) (io.ReadCloser, io.Writer, io.Writer, error) {
 		wsChan := make(chan *websocket.Conn)
 		h := func(conn *websocket.Conn) {
 			wsChan <- conn
@@ -860,6 +862,8 @@ func (s *containerRouter) wsContainersAttach(ctx context.Context, w http.Respons
 		if versions.GreaterThanOrEqualTo(version, "1.28") {
 			conn.PayloadType = websocket.BinaryFrame
 		}
+
+		// TODO: Close notifications
 		return conn, conn, conn, nil
 	}
 
diff --git a/api/server/router/container/notify_linux.go b/api/server/router/container/notify_linux.go
new file mode 100644
index 0000000000000..5ec02bebf4712
--- /dev/null
+++ b/api/server/router/container/notify_linux.go
@@ -0,0 +1,54 @@
+package container
+
+import (
+	"context"
+	"net"
+	"syscall"
+
+	"github.com/containerd/log"
+	"github.com/docker/docker/internal/unix_noeintr"
+	"golang.org/x/sys/unix"
+)
+
+func notifyClosed(ctx context.Context, conn net.Conn, notify func()) {
+	sc, ok := conn.(syscall.Conn)
+	if !ok {
+		log.G(ctx).Debug("notifyClosed: conn does not support close notifications")
+		return
+	}
+
+	rc, err := sc.SyscallConn()
+	if err != nil {
+		log.G(ctx).WithError(err).Warn("notifyClosed: failed get raw conn for close notifications")
+		return
+	}
+
+	epFd, err := unix_noeintr.EpollCreate()
+	if err != nil {
+		log.G(ctx).WithError(err).Warn("notifyClosed: failed to create epoll fd")
+		return
+	}
+	defer unix.Close(epFd)
+
+	err = rc.Control(func(fd uintptr) {
+		err := unix_noeintr.EpollCtl(epFd, unix.EPOLL_CTL_ADD, int(fd), &unix.EpollEvent{
+			Events: unix.EPOLLHUP,
+			Fd:     int32(fd),
+		})
+		if err != nil {
+			log.G(ctx).WithError(err).Warn("notifyClosed: failed to register fd for close notifications")
+			return
+		}
+
+		events := make([]unix.EpollEvent, 1)
+		if _, err := unix_noeintr.EpollWait(epFd, events, -1); err != nil {
+			log.G(ctx).WithError(err).Warn("notifyClosed: failed to wait for close notifications")
+			return
+		}
+		notify()
+	})
+	if err != nil {
+		log.G(ctx).WithError(err).Warn("notifyClosed: failed to register for close notifications")
+		return
+	}
+}
diff --git a/api/server/router/container/notify_unsupported.go b/api/server/router/container/notify_unsupported.go
new file mode 100644
index 0000000000000..8d80afcf33009
--- /dev/null
+++ b/api/server/router/container/notify_unsupported.go
@@ -0,0 +1,10 @@
+//go:build !linux
+
+package container
+
+import (
+	"context"
+	"net"
+)
+
+func notifyClosed(ctx context.Context, conn net.Conn, notify func()) {}
diff --git a/api/types/backend/backend.go b/api/types/backend/backend.go
index e4e760905d01c..d6ec041663c5a 100644
--- a/api/types/backend/backend.go
+++ b/api/types/backend/backend.go
@@ -30,7 +30,7 @@ type ContainerRmConfig struct {
 
 // ContainerAttachConfig holds the streams to use when connecting to a container to view logs.
 type ContainerAttachConfig struct {
-	GetStreams func(multiplexed bool) (io.ReadCloser, io.Writer, io.Writer, error)
+	GetStreams func(multiplexed bool, cancel func()) (io.ReadCloser, io.Writer, io.Writer, error)
 	UseStdin   bool
 	UseStdout  bool
 	UseStderr  bool
diff --git a/container/stream/attach.go b/container/stream/attach.go
index 6079d20d1e56b..c0b0302f46b17 100644
--- a/container/stream/attach.go
+++ b/container/stream/attach.go
@@ -150,6 +150,14 @@ func (c *Config) CopyStreams(ctx context.Context, cfg *AttachConfig) <-chan erro
 				cfg.CStderr.Close()
 			}
 
+			if cfg.Stdin != nil {
+				// In this case, `cfg.Stdin` is a stream from the client.
+				// The way `io.Copy` works we may get stuck waiting to read from `cfg.Stdin` even if the container has exited.
+				// This will cause the `io.Copy` to never return and the `group.Wait()` to never return.
+				// By closing cfg.Stdin we will cause the `io.Copy` to return and the `group.Wait()` to return.
+				cfg.Stdin.Close()
+			}
+
 			// Now with these closed, wait should return.
 			if err := group.Wait(); err != nil {
 				errs <- err
diff --git a/daemon/attach.go b/daemon/attach.go
index 9809671134a74..f233cdca862af 100644
--- a/daemon/attach.go
+++ b/daemon/attach.go
@@ -18,13 +18,13 @@ import (
 )
 
 // ContainerAttach attaches to logs according to the config passed in. See ContainerAttachConfig.
-func (daemon *Daemon) ContainerAttach(prefixOrName string, c *backend.ContainerAttachConfig) error {
+func (daemon *Daemon) ContainerAttach(prefixOrName string, req *backend.ContainerAttachConfig) error {
 	keys := []byte{}
 	var err error
-	if c.DetachKeys != "" {
-		keys, err = term.ToBytes(c.DetachKeys)
+	if req.DetachKeys != "" {
+		keys, err = term.ToBytes(req.DetachKeys)
 		if err != nil {
-			return errdefs.InvalidParameter(errors.Errorf("Invalid detach keys (%s) provided", c.DetachKeys))
+			return errdefs.InvalidParameter(errors.Errorf("Invalid detach keys (%s) provided", req.DetachKeys))
 		}
 	}
 
@@ -42,20 +42,37 @@ func (daemon *Daemon) ContainerAttach(prefixOrName string, c *backend.ContainerA
 	}
 
 	cfg := stream.AttachConfig{
-		UseStdin:   c.UseStdin,
-		UseStdout:  c.UseStdout,
-		UseStderr:  c.UseStderr,
+		UseStdin:   req.UseStdin,
+		UseStdout:  req.UseStdout,
+		UseStderr:  req.UseStderr,
 		TTY:        ctr.Config.Tty,
 		CloseStdin: ctr.Config.StdinOnce,
 		DetachKeys: keys,
 	}
 	ctr.StreamConfig.AttachStreams(&cfg)
 
-	multiplexed := !ctr.Config.Tty && c.MuxStreams
-	inStream, outStream, errStream, err := c.GetStreams(multiplexed)
+	multiplexed := !ctr.Config.Tty && req.MuxStreams
+
+	clientCtx, closeNotify := context.WithCancel(context.Background())
+	defer closeNotify()
+	go func() {
+		<-clientCtx.Done()
+		// The client has disconnected
+		// In this case we need to close the container's output streams so that the goroutines used to copy
+		// to the client streams are unblocked and can exit.
+		if cfg.CStdout != nil {
+			cfg.CStdout.Close()
+		}
+		if cfg.CStderr != nil {
+			cfg.CStderr.Close()
+		}
+	}()
+
+	inStream, outStream, errStream, err := req.GetStreams(multiplexed, closeNotify)
 	if err != nil {
 		return err
 	}
+
 	defer inStream.Close()
 
 	if multiplexed {
@@ -73,7 +90,7 @@ func (daemon *Daemon) ContainerAttach(prefixOrName string, c *backend.ContainerA
 		cfg.Stderr = errStream
 	}
 
-	if err := daemon.containerAttach(ctr, &cfg, c.Logs, c.Stream); err != nil {
+	if err := daemon.containerAttach(ctr, &cfg, req.Logs, req.Stream); err != nil {
 		fmt.Fprintf(outStream, "Error attaching: %s\n", err)
 	}
 	return nil
diff --git a/integration/internal/system/goroutines.go b/integration/internal/system/goroutines.go
new file mode 100644
index 0000000000000..09f2ddb47842a
--- /dev/null
+++ b/integration/internal/system/goroutines.go
@@ -0,0 +1,78 @@
+package system
+
+import (
+	"context"
+	"time"
+
+	"github.com/docker/docker/client"
+	"gotest.tools/v3/poll"
+)
+
+// WaitForStableGoroutineCount polls the daemon Info API and returns the reported goroutine count
+// after multiple calls return the same number.
+func WaitForStableGoroutineCount(ctx context.Context, t poll.TestingT, apiClient client.SystemAPIClient, opts ...poll.SettingOp) int {
+	var out int
+	// Use a longish delay to make sure the goroutine count is actually stable.
+	defaults := []poll.SettingOp{poll.WithTimeout(time.Minute), poll.WithDelay(time.Second)}
+	opts = append(defaults, opts...)
+
+	poll.WaitOn(t, StableGoroutineCount(ctx, apiClient, &out), opts...)
+	return out
+}
+
+// StableGoroutineCount is a [poll.Check] that polls the daemon info API until the goroutine count is the same for 3 iterations.
+func StableGoroutineCount(ctx context.Context, apiClient client.SystemAPIClient, count *int) poll.Check {
+	var (
+		numStable int
+		nRoutines int
+	)
+
+	return func(t poll.LogT) poll.Result {
+		n, err := getGoroutineNumber(ctx, apiClient)
+		if err != nil {
+			return poll.Error(err)
+		}
+
+		last := nRoutines
+
+		if nRoutines == n {
+			numStable++
+		} else {
+			numStable = 0
+			nRoutines = n
+		}
+
+		if numStable > 3 {
+			*count = n
+			return poll.Success()
+		}
+		return poll.Continue("goroutine count is not stable: last %d, current %d, stable iters: %d", last, n, numStable)
+	}
+}
+
+// CheckGoroutineCount returns a [poll.Check] that polls the daemon info API until the expected number of goroutines is hit.
+func CheckGoroutineCount(ctx context.Context, apiClient client.SystemAPIClient, expected int) poll.Check {
+	first := true
+	return func(t poll.LogT) poll.Result {
+		n, err := getGoroutineNumber(ctx, apiClient)
+		if err != nil {
+			return poll.Error(err)
+		}
+		if n > expected {
+			if first {
+				t.Log("Waiting for goroutines to stabilize")
+				first = false
+			}
+			return poll.Continue("exepcted %d goroutines, got %d", expected, n)
+		}
+		return poll.Success()
+	}
+}
+
+func getGoroutineNumber(ctx context.Context, apiClient client.SystemAPIClient) (int, error) {
+	info, err := apiClient.Info(ctx)
+	if err != nil {
+		return 0, err
+	}
+	return info.NGoroutines, nil
+}
diff --git a/internal/unix_noeintr/epoll_linux.go b/internal/unix_noeintr/epoll_linux.go
new file mode 100644
index 0000000000000..ef691ab3ebbf6
--- /dev/null
+++ b/internal/unix_noeintr/epoll_linux.go
@@ -0,0 +1,37 @@
+package unix_noeintr
+
+import (
+	"errors"
+
+	"golang.org/x/sys/unix"
+)
+
+func EpollCreate() (int, error) {
+	for {
+		fd, err := unix.EpollCreate1(unix.EPOLL_CLOEXEC)
+		if errors.Is(err, unix.EINTR) {
+			continue
+		}
+		return fd, err
+	}
+}
+
+func EpollCtl(epFd int, op int, fd int, event *unix.EpollEvent) error {
+	for {
+		err := unix.EpollCtl(epFd, op, fd, event)
+		if errors.Is(err, unix.EINTR) {
+			continue
+		}
+		return err
+	}
+}
+
+func EpollWait(epFd int, events []unix.EpollEvent, msec int) (int, error) {
+	for {
+		n, err := unix.EpollWait(epFd, events, msec)
+		if errors.Is(err, unix.EINTR) {
+			continue
+		}
+		return n, err
+	}
+}


Test patch:
diff --git a/container/stream/attach_test.go b/container/stream/attach_test.go
new file mode 100644
index 0000000000000..e1f7f87b944a6
--- /dev/null
+++ b/container/stream/attach_test.go
@@ -0,0 +1,102 @@
+package stream
+
+import (
+	"context"
+	"io"
+	"testing"
+	"time"
+
+	"gotest.tools/v3/assert"
+)
+
+// Make sure when there is no I/O on a stream that the goroutines do not get blocked after the container exits.
+func TestAttachNoIO(t *testing.T) {
+	t.Run("stdin only", func(t *testing.T) {
+		stdinR, _ := io.Pipe()
+		defer stdinR.Close()
+		testStreamCopy(t, stdinR, nil, nil)
+	})
+
+	t.Run("stdout only", func(t *testing.T) {
+		_, w := io.Pipe()
+		defer w.Close()
+		testStreamCopy(t, nil, w, nil)
+	})
+
+	t.Run("stderr only", func(t *testing.T) {
+		_, w := io.Pipe()
+		defer w.Close()
+		testStreamCopy(t, nil, nil, w)
+	})
+
+	t.Run("stdout+stderr", func(t *testing.T) {
+		_, stdoutW := io.Pipe()
+		defer stdoutW.Close()
+		_, stderrW := io.Pipe()
+		defer stderrW.Close()
+
+		testStreamCopy(t, nil, stdoutW, stderrW)
+	})
+
+	t.Run("stdin+stdout", func(t *testing.T) {
+		stdin, _ := io.Pipe()
+		defer stdin.Close()
+		_, stdout := io.Pipe()
+		defer stdout.Close()
+
+		testStreamCopy(t, stdin, stdout, nil)
+	})
+
+	t.Run("stdin+stderr", func(t *testing.T) {
+		stdin, _ := io.Pipe()
+		defer stdin.Close()
+		_, stderr := io.Pipe()
+		defer stderr.Close()
+
+		testStreamCopy(t, stdin, nil, stderr)
+	})
+
+	t.Run("stdin+stdout+stderr", func(t *testing.T) {
+		stdinR, _ := io.Pipe()
+		defer stdinR.Close()
+		stdoutR, stdoutW := io.Pipe()
+		defer stdoutR.Close()
+		stderrR, stderrW := io.Pipe()
+		defer stderrR.Close()
+		testStreamCopy(t, stdinR, stdoutW, stderrW)
+	})
+}
+
+func testStreamCopy(t *testing.T, stdin io.ReadCloser, stdout, stderr io.WriteCloser) {
+	cfg := AttachConfig{
+		UseStdin:  stdin != nil,
+		UseStdout: stdout != nil,
+		UseStderr: stderr != nil,
+		Stdin:     stdin,
+		Stdout:    stdout,
+		Stderr:    stderr,
+	}
+
+	sc := NewConfig()
+	sc.AttachStreams(&cfg)
+	defer sc.CloseStreams()
+
+	ctx, cancel := context.WithCancel(context.Background())
+	defer cancel()
+	chErr := sc.CopyStreams(ctx, &cfg)
+
+	select {
+	case err := <-chErr:
+		assert.NilError(t, err)
+	default:
+	}
+
+	cancel()
+
+	select {
+	case err := <-chErr:
+		assert.ErrorIs(t, err, context.Canceled)
+	case <-time.After(10 * time.Second):
+		t.Fatal("timeout waiting for CopyStreams to exit")
+	}
+}
diff --git a/integration/container/attach_test.go b/integration/container/attach_test.go
index b761615de5be8..da31453ec291a 100644
--- a/integration/container/attach_test.go
+++ b/integration/container/attach_test.go
@@ -2,13 +2,18 @@ package container // import "github.com/docker/docker/integration/container"
 
 import (
 	"testing"
+	"time"
 
 	"github.com/docker/docker/api/types"
 	"github.com/docker/docker/api/types/container"
 	"github.com/docker/docker/api/types/network"
+	systemutil "github.com/docker/docker/integration/internal/system"
 	"github.com/docker/docker/testutil"
+	"github.com/docker/docker/testutil/daemon"
 	"gotest.tools/v3/assert"
 	is "gotest.tools/v3/assert/cmp"
+	"gotest.tools/v3/poll"
+	"gotest.tools/v3/skip"
 )
 
 func TestAttach(t *testing.T) {
@@ -59,3 +64,58 @@ func TestAttach(t *testing.T) {
 		})
 	}
 }
+
+// Regression test for #37182
+func TestAttachDisconnectLeak(t *testing.T) {
+	skip.If(t, testEnv.DaemonInfo.OSType != "linux", "Bug still exists on Windows")
+	t.Parallel()
+
+	ctx := testutil.StartSpan(baseContext, t)
+
+	// Use a new daemon to make sure stuff from other tests isn't affecting the
+	// goroutine count.
+	d := daemon.New(t)
+	defer d.Cleanup(t)
+
+	d.StartWithBusybox(ctx, t, "--iptables=false")
+
+	client := d.NewClientT(t)
+
+	resp, err := client.ContainerCreate(ctx,
+		&container.Config{
+			Image: "busybox",
+			Cmd:   []string{"/bin/sh", "-c", "while true; usleep 100000; done"},
+		},
+		&container.HostConfig{},
+		&network.NetworkingConfig{},
+		nil,
+		"",
+	)
+	assert.NilError(t, err)
+	cID := resp.ID
+	defer client.ContainerRemove(ctx, cID, container.RemoveOptions{
+		Force: true,
+	})
+
+	nGoroutines := systemutil.WaitForStableGoroutineCount(ctx, t, client)
+
+	attach, err := client.ContainerAttach(ctx, cID, container.AttachOptions{
+		Stdout: true,
+	})
+	assert.NilError(t, err)
+	defer attach.Close()
+
+	poll.WaitOn(t, func(_ poll.LogT) poll.Result {
+		count := systemutil.WaitForStableGoroutineCount(ctx, t, client)
+		if count > nGoroutines {
+			return poll.Success()
+		}
+		return poll.Continue("waiting for goroutines to increase from %d, current: %d", nGoroutines, count)
+	},
+		poll.WithTimeout(time.Minute),
+	)
+
+	attach.Close()
+
+	poll.WaitOn(t, systemutil.CheckGoroutineCount(ctx, client, nGoroutines), poll.WithTimeout(time.Minute))
+}