Repository: moby/moby (language: go)

Issue:
Docker version 24.0.xx: Containers exiting with 0 status code on hard reboot
### Description

Recently we noticed with Docker version **24.0.xx**, the containers are exiting with 0 status code on hard reboot as seen below. This behavior was observed for containers running in both SWARM and COMPOSE mode. 

![image (2)](https://github.com/distribution/distribution/assets/137250645/44973402-6a8c-4563-8a16-d15978d5c6c2)

In docker versions **< 24.0.xx**, the containers were exiting with non-zero status code. 
![image (3)](https://github.com/distribution/distribution/assets/137250645/07d4e526-34a9-4954-a659-88251563cca8)

This change in behavior with docker version 24.0.xx results in container not booting up after a hard reboot if restart policy is set to on-failure.

### Reproduce

1. Install docker version 24.0.2 in Ubuntu
2. Run a sample container with restart-policy ***on-failure**
3. Hard reboot the device
4. Verify container is not restarted (but docker daemon is up)
5. Check the exit code using ```docker container ls -a```. The exit code would be 0.

### Expected behavior

1. Container should exit with non-zero status code on hard reboot
2. If restart-policy of the container is set to "on-failure", container should restart after a hard reboot

### docker version

```bash
Client: Docker Engine - Community
 Version:           24.0.2
 API version:       1.43
 Go version:        go1.20.4
 Git commit:        cb74dfc
 Built:             Thu May 25 21:52:22 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          24.0.2
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.4
  Git commit:       659604f
  Built:            Thu May 25 21:52:22 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.21
  GitCommit:        3dce8eb055cbb6872793272b4f20ed16117344f8
 runc:
  Version:          1.1.7
  GitCommit:        v1.1.7-0-g860f061
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
```


### docker info

```bash
Client: Docker Engine - Community
 Version:    24.0.2
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.10.5
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.18.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 2
  Running: 1
  Paused: 0
  Stopped: 1
 Images: 1
 Server Version: 24.0.2
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: active
  NodeID: nzif6vnxqpx5sr0svdmf5hcm3
  Is Manager: true
  ClusterID: wv0pvxeu1rf32tckcz3rfl4g8
  Managers: 1
  Nodes: 1
  Default Address Pool: 10.0.0.0/8
  SubnetSize: 24
  Data Path Port: 4789
  Orchestration:
   Task History Retention Limit: 5
  Raft:
   Snapshot Interval: 10000
   Number of Old Snapshots to Retain: 0
   Heartbeat Tick: 1
   Election Tick: 10
  Dispatcher:
   Heartbeat Period: 5 seconds
  CA Configuration:
   Expiry Duration: 3 months
   Force Rotate: 0
  Autolock Managers: false
  Root Rotation In Progress: false
  Node Address: 127.0.0.1
  Manager Addresses:
   127.0.0.1:2377
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 3dce8eb055cbb6872793272b4f20ed16117344f8
 runc version: v1.1.7-0-g860f061
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
 Kernel Version: 5.4.0-150-lowlatency
 Operating System: Ubuntu 20.04.4 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 5
 Total Memory: 15.51GiB
 Name: MDT4-AutoSlam-101
 ID: 7fb4077d-5f78-4553-ae97-3c7b728fdb09
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
```


### Additional Info

_No response_


Discussion / hints:
Are those containers expected to have a non-zero exit on a clean shutdown? I'm wondering if this actually is the side effect of a fix where (potentially) older versions didn't properly do a graceful shutdown of containers when the docker service was shut down
Since the devices were hard rebooted i.e the devices lost power immediately (the device power cord was pulled), the docker container would not be able to go through the clean shutdown (graceful shutdown) process. So the issue here is that previously when the host device suddenly lost power, the containers that were shut down as a result of this, were considered to be exited with a non-zero status code. This behavior has been modified in docker version 24.0.xx. I would expect docker container that were exited due to the host device losing power, to have a non-zero exit.
I have used the command ` docker run --restart=on-failure ubuntu` but it did not get started  automatically after reboot.

![image](https://github.com/moby/moby/assets/94950988/6a7f7bfa-d22d-43d0-9dc6-616bc1059e9a)
![image](https://github.com/moby/moby/assets/94950988/82065a27-e1a3-4d12-b4da-2ba7c9833b6f)


@satyampsoni looks like you're running docker engine 20.10.24 (see the "server" section), which means that your issue should be separate from this discussion.

Fix patch:
diff --git a/daemon/daemon.go b/daemon/daemon.go
index bd20e158f0b35..c90c8d91c2376 100644
--- a/daemon/daemon.go
+++ b/daemon/daemon.go
@@ -447,6 +447,8 @@ func (daemon *Daemon) restore(cfg *configStore) error {
 					if es != nil {
 						ces.ExitCode = int(es.ExitCode())
 						ces.ExitedAt = es.ExitTime()
+					} else {
+						ces.ExitCode = 255
 					}
 					c.SetStopped(&ces)
 					daemon.Cleanup(c)


Test patch:
diff --git a/integration/container/daemon_linux_test.go b/integration/container/daemon_linux_test.go
index 5189c6bf53793..d1d6c61a9df5d 100644
--- a/integration/container/daemon_linux_test.go
+++ b/integration/container/daemon_linux_test.go
@@ -2,10 +2,8 @@ package container // import "github.com/docker/docker/integration/container"
 
 import (
 	"context"
-	"encoding/json"
 	"fmt"
 	"os"
-	"path/filepath"
 	"strconv"
 	"strings"
 	"testing"
@@ -19,6 +17,7 @@ import (
 	"golang.org/x/sys/unix"
 	"gotest.tools/v3/assert"
 	is "gotest.tools/v3/assert/cmp"
+	"gotest.tools/v3/assert/opt"
 	"gotest.tools/v3/skip"
 )
 
@@ -204,21 +203,10 @@ func TestRestartDaemonWithRestartingContainer(t *testing.T) {
 
 	d.Stop(t)
 
-	configPath := filepath.Join(d.Root, "containers", id, "config.v2.json")
-	configBytes, err := os.ReadFile(configPath)
-	assert.NilError(t, err)
-
-	var c realcontainer.Container
-
-	assert.NilError(t, json.Unmarshal(configBytes, &c))
-
-	c.State = realcontainer.NewState()
-	c.SetRestarting(&realcontainer.ExitStatus{ExitCode: 1})
-	c.HasBeenStartedBefore = true
-
-	configBytes, err = json.Marshal(&c)
-	assert.NilError(t, err)
-	assert.NilError(t, os.WriteFile(configPath, configBytes, 0600))
+	d.TamperWithContainerConfig(t, id, func(c *realcontainer.Container) {
+		c.SetRestarting(&realcontainer.ExitStatus{ExitCode: 1})
+		c.HasBeenStartedBefore = true
+	})
 
 	d.Start(t)
 
@@ -231,3 +219,71 @@ func TestRestartDaemonWithRestartingContainer(t *testing.T) {
 		assert.NilError(t, err)
 	}
 }
+
+// TestHardRestartWhenContainerIsRunning simulates a case where dockerd is
+// killed while a container is running, and the container's task no longer
+// exists when dockerd starts back up. This can happen if the system is
+// hard-rebooted, for example.
+//
+// Regression test for moby/moby#45788
+func TestHardRestartWhenContainerIsRunning(t *testing.T) {
+	skip.If(t, testEnv.IsRemoteDaemon, "cannot start daemon on remote test run")
+	skip.If(t, testEnv.DaemonInfo.OSType == "windows")
+
+	t.Parallel()
+
+	d := daemon.New(t)
+	defer d.Cleanup(t)
+
+	d.StartWithBusybox(t, "--iptables=false")
+	defer d.Stop(t)
+
+	ctx := context.Background()
+	client := d.NewClientT(t)
+
+	// Just create the containers, no need to start them.
+	// We really want to make sure there is no process running when docker starts back up.
+	// We will manipulate the on disk state later.
+	nopolicy := container.Create(ctx, t, client, container.WithCmd("/bin/sh", "-c", "exit 1"))
+	onfailure := container.Create(ctx, t, client, container.WithRestartPolicy("on-failure"), container.WithCmd("/bin/sh", "-c", "sleep 60"))
+
+	d.Stop(t)
+
+	for _, id := range []string{nopolicy, onfailure} {
+		d.TamperWithContainerConfig(t, id, func(c *realcontainer.Container) {
+			c.SetRunning(nil, nil, true)
+			c.HasBeenStartedBefore = true
+		})
+	}
+
+	d.Start(t)
+
+	t.Run("RestartPolicy=none", func(t *testing.T) {
+		ctx, cancel := context.WithTimeout(ctx, 5*time.Second)
+		defer cancel()
+		inspect, err := client.ContainerInspect(ctx, nopolicy)
+		assert.NilError(t, err)
+		assert.Check(t, is.Equal(inspect.State.Status, "exited"))
+		assert.Check(t, is.Equal(inspect.State.ExitCode, 255))
+		finishedAt, err := time.Parse(time.RFC3339Nano, inspect.State.FinishedAt)
+		if assert.Check(t, err) {
+			assert.Check(t, is.DeepEqual(finishedAt, time.Now(), opt.TimeWithThreshold(time.Minute)))
+		}
+	})
+
+	t.Run("RestartPolicy=on-failure", func(t *testing.T) {
+		ctx, cancel := context.WithTimeout(ctx, 5*time.Second)
+		defer cancel()
+		inspect, err := client.ContainerInspect(ctx, onfailure)
+		assert.NilError(t, err)
+		assert.Check(t, is.Equal(inspect.State.Status, "running"))
+		assert.Check(t, is.Equal(inspect.State.ExitCode, 0))
+		finishedAt, err := time.Parse(time.RFC3339Nano, inspect.State.FinishedAt)
+		if assert.Check(t, err) {
+			assert.Check(t, is.DeepEqual(finishedAt, time.Now(), opt.TimeWithThreshold(time.Minute)))
+		}
+
+		stopTimeout := 0
+		assert.Assert(t, client.ContainerStop(ctx, onfailure, containerapi.StopOptions{Timeout: &stopTimeout}))
+	})
+}
diff --git a/testutil/daemon/daemon.go b/testutil/daemon/daemon.go
index 2a1f5b3dc8298..98230960c6da0 100644
--- a/testutil/daemon/daemon.go
+++ b/testutil/daemon/daemon.go
@@ -16,6 +16,7 @@ import (
 	"github.com/docker/docker/api/types"
 	"github.com/docker/docker/api/types/events"
 	"github.com/docker/docker/client"
+	"github.com/docker/docker/container"
 	"github.com/docker/docker/pkg/ioutils"
 	"github.com/docker/docker/pkg/stringid"
 	"github.com/docker/docker/testutil/request"
@@ -825,6 +826,23 @@ func (d *Daemon) Info(t testing.TB) types.Info {
 	return info
 }
 
+// TamperWithContainerConfig modifies the on-disk config of a container.
+func (d *Daemon) TamperWithContainerConfig(t testing.TB, containerID string, tamper func(*container.Container)) {
+	t.Helper()
+
+	configPath := filepath.Join(d.Root, "containers", containerID, "config.v2.json")
+	configBytes, err := os.ReadFile(configPath)
+	assert.NilError(t, err)
+
+	var c container.Container
+	assert.NilError(t, json.Unmarshal(configBytes, &c))
+	c.State = container.NewState()
+	tamper(&c)
+	configBytes, err = json.Marshal(&c)
+	assert.NilError(t, err)
+	assert.NilError(t, os.WriteFile(configPath, configBytes, 0600))
+}
+
 // cleanupRaftDir removes swarmkit wal files if present
 func cleanupRaftDir(t testing.TB, d *Daemon) {
 	t.Helper()