Repository: moby/moby (language: go)

Issue:
30s HealthStartInterval being applied to containers created with API 1.43 client
### Description

If any non-zero `health-start-period` is provided to `docker run`, no `health-cmd` outcome will affect the container's health status (regardless of success or failure) until the first run after `30s` has elapsed or the container is stopped/exits.


### Reproduce

###  Case 1
- `--health-interval=2s`
- `--health-start-period=2s`
- container is "running" for 35s (`sleep 35`)
- EXPECTED: "healthy" after ~2-4s
- ACTUAL: "healthy" after ~30-32s
```sh
docker run --name=case-1 --health-cmd 'echo hello' --health-interval=2s --health-start-period=2s busybox sleep 35
```

###  Case 2
- `--health-interval=2s`
- `--health-start-period=15s`
- container is "running" for 35s (`sleep 35`)
- EXPECTED: "healthy" after ~2-4s
- ACTUAL: "healthy" after 32s
```sh
docker run --name=case-2 --health-cmd 'echo hello' --health-interval=2s --health-start-period=15s busybox sleep 35
```

###  Case 3
- `--health-interval=2s`
- container is "running" for 5s (`sleep 5`)
- EXPECTED: "healthy" after ~2s
- ACTUAL: "healthy" after ~2s **(works as expected)**
```sh
docker run --name=case-3 --health-cmd 'echo hello' --health-interval=2s busybox sleep 5
```

###  Case 4
- `--health-interval=2s`
- `--health-start-period=15s`
- container is "running" for 20s (`sleep 20`)
- EXPECTED: "unhealthy" after ~15-17s
- ACTUAL: "unhealthy" w/o any health logs after 20s, so we don't know when or how the status changed.
    - In this case, inspecting the container doesn't show any `log` of healthcheck results, despite the fact that there should have been ~5s during which the container was running outside of the `start-period`.
        - Extend the sleep command to `35` and you will see that some healthcheck test outcomes are logged, but only after 30s (which is not what the `start-period` was configured as).
    - I believe this case demonstrates that some 30s threshold is being respected as the `start-period` instead of the user-provided value.
```sh
docker run --name=case-4 --health-cmd 'fake' --health-interval=2s --health-start-period=15s 
busybox sleep 20
```


### Expected behavior

Based on [the docs](https://docs.docker.com/engine/reference/builder/#healthcheck) I expected:
- Any successful health command execution within a specified `start-period` to allow the container to be marked "healthy" (and any subsequent failure, even if within the start-period, would mark it as "unhealthy")
- The first `health-cmd` failure result to occur _after_ the specified `start-period` should result in the container being immediately marked as unhealthy.

### docker version

```bash
Client:
 Cloud integration: v1.0.35+desktop.5
 Version:           24.0.6
 API version:       1.43
 Go version:        go1.20.7
 Git commit:        ed223bc
 Built:             Mon Sep  4 12:28:49 2023
 OS/Arch:           darwin/arm64
 Context:           desktop-linux

Server: Docker Desktop 4.24.2 (124339)
 Engine:
  Version:          dev
  API version:      1.44 (minimum version 1.12)
  Go version:       go1.20.8
  Git commit:       HEAD
  Built:            Tue Sep 26 11:52:32 2023
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.6.22
  GitCommit:        8165feabfdfe38c65b599c4993d227328c231fca
 runc:
  Version:          1.1.8
  GitCommit:        v1.1.8-0-g82f18fe
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
```


### docker info

```bash
Client:
 Version:    24.0.6
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.11.2-desktop.5
    Path:     /Users/tyler/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.22.0-desktop.2
    Path:     /Users/tyler/.docker/cli-plugins/docker-compose
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.0
    Path:     /Users/tyler/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.20
    Path:     /Users/tyler/.docker/cli-plugins/docker-extension
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v0.1.0-beta.8
    Path:     /Users/tyler/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/tyler/.docker/cli-plugins/docker-sbom
  scan: Docker Scan (Docker Inc.)
    Version:  v0.26.0
    Path:     /Users/tyler/.docker/cli-plugins/docker-scan
  scout: Docker Scout (Docker Inc.)
    Version:  v1.0.7
    Path:     /Users/tyler/.docker/cli-plugins/docker-scout

Server:
 Containers: 30
  Running: 24
  Paused: 0
  Stopped: 6
 Images: 77
 Server Version: dev
 Storage Driver: stargz
  driver-type: io.containerd.snapshotter.v1
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 8165feabfdfe38c65b599c4993d227328c231fca
 runc version: v1.1.8-0-g82f18fe
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
  cgroupns
 Kernel Version: 6.4.16-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 8
 Total Memory: 19.52GiB
 Name: docker-desktop
 ID: 36252aca-a483-4b9d-b39c-ea24c2c0d207
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: daemon is not using the default seccomp profile
```


### Additional Info

_No response_


Discussion / hints:
I am unable to reproduce cases 1 or 2 on Docker Desktop for Mac v4.24.2, ostensibly the same version as you are running. Both containers become healthy within two seconds, as expected.

<details><summary>My <tt>docker version</tt></summary>
<p>

Client:
 Cloud integration: v1.0.35+desktop.5
 Version:           24.0.6
 API version:       1.43
 Go version:        go1.20.7
 Git commit:        ed223bc
 Built:             Mon Sep  4 12:28:49 2023
 OS/Arch:           darwin/amd64
 Context:           desktop-linux

Server: Docker Desktop 4.24.2 (124339)
 Engine:
  Version:          24.0.6
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.7
  Git commit:       1a79695
  Built:            Mon Sep  4 12:32:16 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.22
  GitCommit:        8165feabfdfe38c65b599c4993d227328c231fca
 runc:
  Version:          1.1.8
  GitCommit:        v1.1.8-0-g82f18fe
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0


</p>
</details> 

Oddly, it looks like you are running some random development version of of the v25 branch?
```
Server: Docker Desktop 4.24.2 (124339)
 Engine:
  Version:          dev
  API version:      1.44 (minimum version 1.12)
  Go version:       go1.20.8
  Git commit:       HEAD
  Built:            Tue Sep 26 11:52:32 2023
  OS/Arch:          linux/arm64
  Experimental:     false
```

Note the API version.

------

Your expectations for Case 4 are wrong. A container configured with a health check can only reach healthy status after a health check succeeds. Health checks cannot succeed when the health-check command is bogus, so it is expected that the container will never reach healthy status. What you observed, the container transitioning from `starting` status to `unhealthy` after the start period has elapsed, is the expected behaviour.

> Your expectations for Case 4 are wrong. A container configured with a health check can only reach healthy status after a health check succeeds. Health checks cannot succeed when the health-check command is bogus, so it is expected that the container will never reach healthy status. What you observed, the container transitioning from `starting` status to `unhealthy` after the start period has elapsed, is the expected behaviour.

You're absolutely right. I made a mistake in describing that case. I've corrected it in the main issue description. It should have read:

###  Case 4
- `--health-interval=2s`
- `--health-start-period=15s`
- EXPECTED: "unhealthy" after ~15-17s
- ACTUAL: "unhealthy" w/o any health logs after 20s, so we don't know when or how the status changed.
    - In this case, inspecting the container doesn't show any `log` of healthcheck results, despite the fact that there should have been ~5s during which the container was running outside of the `start-period`.
        - Extend the sleep command to `35` and you will see that some healthcheck test outcomes are logged, but only after 30s (which is not what the `start-period` was configured as).
    - I believe this case demonstrates that some 30s threshold is being respected as the `start-period` instead of the user-provided value.
```sh
docker run --name=case-4 --health-cmd 'fake' --health-interval=2s --health-start-period=15s 
busybox sleep 20
```

<details>
<summary>Inspect output</summary>

```
"State": {
        "Status": "exited",
        "Running": false,
        "Paused": false,
        "Restarting": false,
        "OOMKilled": false,
        "Dead": false,
        "Pid": 0,
        "ExitCode": 0,
        "Error": "",
        "StartedAt": "2023-10-30T20:38:11.254493502Z",
        "FinishedAt": "2023-10-30T20:38:46.259828588Z",
        "Health": {
            "Status": "unhealthy",
            "FailingStreak": 3,
            "Log": [
                {
                    "Start": "2023-10-30T20:38:41.257489585Z",
                    "End": "2023-10-30T20:38:41.29221346Z",
                    "ExitCode": 127,
                    "Output": "/bin/sh: fake: not found\n"
                },
                {
                    "Start": "2023-10-30T20:38:43.297019961Z",
                    "End": "2023-10-30T20:38:43.34983692Z",
                    "ExitCode": 127,
                    "Output": "/bin/sh: fake: not found\n"
                },
                {
                    "Start": "2023-10-30T20:38:45.356054171Z",
                    "End": "2023-10-30T20:38:45.407804421Z",
                    "ExitCode": 127,
                    "Output": "/bin/sh: fake: not found\n"
                }
            ]
        }
    },
```

</details> 

Your deduction about there being a hidden 30-second threshold is correct.
- https://github.com/moby/moby/pull/40894

I have been informed that when you opt into the containerd store on Docker Desktop, you are also opting into a development build of the daemon. Thank you for alpha-testing v25!

The default value for the new health-check start interval configuration is 30 seconds. It is supposed to only apply to API version >= 1.44 clients but apparently there's a bug in our backward-compatibility logic. And how we handle the case when `StartInterval > StartPeriod` is clearly not intuitive. @cpuguy83 

Fix patch:
diff --git a/daemon/health.go b/daemon/health.go
index e454caf526a80..b046d0441ec5e 100644
--- a/daemon/health.go
+++ b/daemon/health.go
@@ -248,7 +248,7 @@ func handleProbeResult(d *Daemon, c *container.Container, result *types.Healthch
 // There is never more than one monitor thread running per container at a time.
 func monitor(d *Daemon, c *container.Container, stop chan struct{}, probe probe) {
 	probeInterval := timeoutWithDefault(c.Config.Healthcheck.Interval, defaultProbeInterval)
-	startInterval := timeoutWithDefault(c.Config.Healthcheck.StartInterval, defaultProbeInterval)
+	startInterval := timeoutWithDefault(c.Config.Healthcheck.StartInterval, probeInterval)
 	startPeriod := timeoutWithDefault(c.Config.Healthcheck.StartPeriod, defaultStartPeriod)
 
 	c.Lock()


Test patch: