Repository: moby/moby (language: go)

Issue:
dockerd panic: runtime error at monitor.go:189
### Description

Looks like part of the code that was refactored "recently"
- https://github.com/moby/moby/pull/43564

```
Dockerd panic:
[2023-06-15T13:13:22.069699364Z][dockerd][I] panic: runtime error: invalid memory address or nil pointer dereference
[2023-06-15T13:13:22.069710675Z][dockerd][I] [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x3588ae4]
[2023-06-15T13:13:22.069742125Z][dockerd][I] 
[2023-06-15T13:13:22.069742615Z][dockerd][I] goroutine 5158404 [running]:
[2023-06-15T13:13:22.069747822Z][dockerd][I] github.com/docker/docker/daemon.(*Daemon).ProcessEvent.func1()
[2023-06-15T13:13:22.070073145Z][dockerd][I] 	/go/src/github.com/docker/docker/daemon/monitor.go:190 +0x44
[2023-06-15T13:13:22.070077880Z][dockerd][I] created by github.com/docker/docker/daemon.(*Daemon).ProcessEvent
[2023-06-15T13:13:22.070093744Z][dockerd][I] 	/go/src/github.com/docker/docker/daemon/monitor.go:189 +0x55d
[2023-06-15T13:13:23.030707964Z][dockerd][I] EOF
```

Docker Desktop diagnostic ID: 379DAAF8-E0CF-4906-AF4C-7BED92CAF01F/20230615153018

### Reproduce

Unknown

### Expected behavior

_No response_

### docker version

```bash
{
  "ApiVersion": "1.43",
  "Arch": "amd64",
  "BuildTime": "2023-06-02T15:58:50.000000000+00:00",
  "Components": [
    {
      "Details": {
        "ApiVersion": "1.43",
        "Arch": "amd64",
        "BuildTime": "2023-06-02T15:58:50.000000000+00:00",
        "Experimental": "false",
        "GitCommit": "8e70a1b23e965d86ec8c2feb77605196ae124630",
        "GoVersion": "go1.20.4",
        "KernelVersion": "5.15.49-linuxkit-pr",
        "MinAPIVersion": "1.12",
        "Os": "linux"
      },
      "Name": "Engine",
      "Version": "24.0.2-38-g8e70a1b23e"
    },
    {
      "Details": {
        "GitCommit": "3dce8eb055cbb6872793272b4f20ed16117344f8"
      },
      "Name": "containerd",
      "Version": "1.6.21"
    },
    {
      "Details": {
        "GitCommit": "v1.1.7-0-g860f061"
      },
      "Name": "runc",
      "Version": "1.1.7"
    },
    {
      "Details": {
        "GitCommit": "de40ad0"
      },
      "Name": "docker-init",
      "Version": "0.19.0"
    }
  ],
  "GitCommit": "8e70a1b23e965d86ec8c2feb77605196ae124630",
  "GoVersion": "go1.20.4",
  "KernelVersion": "5.15.49-linuxkit-pr",
  "MinAPIVersion": "1.12",
  "Os": "linux",
  "Platform": {
    "Name": "Docker Engine - Community"
  },
  "Version": "24.0.2-38-g8e70a1b23e"
```


### docker info

```bash
Client:
 Version:    24.0.2
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.10.4
    Path:     /Users/milosgajdos/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.17.3
    Path:     /Users/milosgajdos/.docker/cli-plugins/docker-compose
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.0
    Path:     /Users/milosgajdos/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.19
    Path:     /Users/milosgajdos/.docker/cli-plugins/docker-extension
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v0.1.0-beta.4
    Path:     /Users/milosgajdos/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/milosgajdos/.docker/cli-plugins/docker-sbom
  scan: Docker Scan (Docker Inc.)
    Version:  v0.26.0
    Path:     /Users/milosgajdos/.docker/cli-plugins/docker-scan
  scout: Command line tool for Docker Scout (Docker Inc.)
    Version:  v0.12.0
    Path:     /Users/milosgajdos/.docker/cli-plugins/docker-scout

Server:
 Containers: 1
  Running: 0
  Paused: 0
  Stopped: 1
 Images: 48
 Server Version: 24.0.2-38-g8e70a1b23e
 Storage Driver: stargz
  driver-type: io.containerd.snapshotter.v1
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 3dce8eb055cbb6872793272b4f20ed16117344f8
 runc version: v1.1.7-0-g860f061
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.15.49-linuxkit-pr
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 7.773GiB
 Name: docker-desktop
 ID: 96b8b534-8e84-4d6e-8a6f-5be60219a133
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Experimental: false
 Insecure Registries:
  host.docker.internal:5000
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false
```


### Additional Info

_No response_


Discussion / hints:
cc @corhere
Looks similar to #45641. It seems [my claim that `c.ExecCommands` only contains execs in the Running state](https://github.com/moby/moby/blob/659604f9ee60f147020bdd444b26e4b5c636dc28/daemon/monitor.go#L171-L173) is incorrect. It will also contain created execs, and, as evidenced by this issue, sometimes execs which failed to start. While there is an easy fix for the panic (check that `execConfig.Process != nil` before calling `Delete()` on it) I suspect that this is just the most prominent symptom of deeper issues with the lifecycle of an exec.

Fix patch:
diff --git a/daemon/monitor.go b/daemon/monitor.go
index 6dd5dbab15f5b..1897c533e36dc 100644
--- a/daemon/monitor.go
+++ b/daemon/monitor.go
@@ -179,7 +179,7 @@ func (daemon *Daemon) ProcessEvent(id string, e libcontainerdtypes.EventType, ei
 			// Remove the exec command from the container's store only and not the
 			// daemon's store so that the exec command can be inspected. Remove it
 			// before mutating execConfig to maintain the invariant that
-			// c.ExecCommands only contain execs in the Running state.
+			// c.ExecCommands only contains execs that have not exited.
 			c.ExecCommands.Delete(execConfig.ID)
 
 			execConfig.ExitCode = &ec
@@ -195,15 +195,25 @@ func (daemon *Daemon) ProcessEvent(id string, e libcontainerdtypes.EventType, ei
 
 			exitCode = ec
 
-			go func() {
-				if _, err := execConfig.Process.Delete(context.Background()); err != nil {
-					logrus.WithFields(logrus.Fields{
-						logrus.ErrorKey: err,
-						"container":     ei.ContainerID,
-						"process":       ei.ProcessID,
-					}).Warn("failed to delete process")
-				}
-			}()
+			// If the exec failed at start in such a way that containerd
+			// publishes an exit event for it, we will race processing the event
+			// with daemon.ContainerExecStart() removing the exec from
+			// c.ExecCommands. If we win the race, we will find that there is no
+			// process to clean up. (And ContainerExecStart will clobber the
+			// exit code we set.) Prevent a nil-dereferenc panic in that
+			// situation to restore the status quo where this is merely a
+			// logical race condition.
+			if execConfig.Process != nil {
+				go func() {
+					if _, err := execConfig.Process.Delete(context.Background()); err != nil {
+						logrus.WithFields(logrus.Fields{
+							logrus.ErrorKey: err,
+							"container":     ei.ContainerID,
+							"process":       ei.ProcessID,
+						}).Warn("failed to delete process")
+					}
+				}()
+			}
 		}
 		attributes := map[string]string{
 			"execID":   ei.ProcessID,


Test patch: