Repository: moby/moby (language: go)

Issue:
Restarting dockerd breaks the embedded DNS server
### Description

Ubuntu Unattended-upgrades upgraded docker-ce and friends. As part of the upgrade, dockerd was restarted. Subsequently, running containers could not resolve the name of peers on the same host (Docker bridge networking in use).

Log has many pairs of these two messages

```
Resolver Start failed for container .... setting up IP table rules failed:  (iptables failed: iptables --wait -t nat -N DOCKER_OUTPUT: iptables: Chain already exists
failed to populate fields for osl sandbox
```


### Reproduce

Create a clean VM, updata all packages, install docker, add ubuntu user to the docker group then reboot
```
multipass -c 2 -m 4G -d 20G jam jammy
multipass shell jam
sudo apt update
sudo apt upgrade -y
sudo apt install ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg
echo   "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
"$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" |   sudo tee /etc/apt/sources.list.d/docker.list
sudo apt install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
sudo nano /etc/docker/daemon.json
sudo usermod -a -G docker ubuntu
sudo reboot
```
daemon.json contents, note "live-restore": true
```
{
    "icc": false,
    "log-driver": "syslog",
    "log-opts": {
        "syslog-facility": "local1",
        "tag": "{{.Name}}"
    },
    "live-restore": true,
    "userland-proxy": false,
    "userns-remap": "default",
    "default-address-pools":
        [
            {"base":"192.168.255.0/24","size":29}
        ],
    "bip": "192.168.255.253/30"
}
```
Following the reboot, start two containers using Docker compose.
```
docker pull bash:latest
nano dc.yml
docker compose -f dc.yml -p t up -d
```
Contents of dc.yml
```
services:
cona:
    entrypoint: ["/bin/sleep","1d"]
    hostname: hosta
    image: bash:latest
    restart: unless-stopped
    networks:
    - neta
    - netb
    security_opt:
    - no-new-privileges:true
conb:
    entrypoint: ["/bin/sleep","1d"]
    hostname: hostb
    image: bash:latest
    restart: unless-stopped
    networks:
    - neta
    - netb
    security_opt:
    - no-new-privileges:true
networks:
neta:
netb:
```
Prove that the embedded DNS server works
```
ubuntu@jam:~$ docker exec t-cona-1 ping conb
PING conb (192.168.255.26): 56 data bytes
64 bytes from 192.168.255.26: seq=0 ttl=64 time=0.224 ms
64 bytes from 192.168.255.26: seq=1 ttl=64 time=0.204 ms
^C
ubuntu@jam:~$ docker exec t-conb-1 ping cona
PING cona (192.168.255.27): 56 data bytes
64 bytes from 192.168.255.27: seq=0 ttl=64 time=0.118 ms
64 bytes from 192.168.255.27: seq=1 ttl=64 time=0.159 ms
^C
ubuntu@jam:~$
```
Restart dockerd and observe the logs
```
ubuntu@jam:~$ sudo systemctl restart docker
ubuntu@jam:~$ sudo systemctl status docker
● docker.service - Docker Application Container Engine
    Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
    Active: active (running) since Tue 2023-05-30 09:18:16 AEST; 5s ago
TriggeredBy: ● docker.socket
    Docs: https://docs.docker.com
Main PID: 1479 (dockerd)
    Tasks: 9
    Memory: 26.3M
        CPU: 267ms
    CGroup: /system.slice/docker.service
            └─1479 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock

May 30 09:18:16 jam dockerd[1479]: time="2023-05-30T09:18:16.369133373+10:00" level=error msg="Resolver Start failed for container b5ceaf056e1a2603abde826490cf3ac2d18d8af6095c64a6716e5e7b544bae86, \"setting up I>
May 30 09:18:16 jam dockerd[1479]: time="2023-05-30T09:18:16.369164615+10:00" level=error msg="failed to populate fields for osl sandbox 09d0460ebab447eb650884ff0e4371b86125185b8776db0ef8ba4903e915258a"
May 30 09:18:16 jam dockerd[1479]: time="2023-05-30T09:18:16.371111533+10:00" level=error msg="Resolver Start failed for container 11881e3c01dfb4f8244342ec0cc241d7c18250d1d9956d69725a9b4e1d868f0d, \"setting up I>
May 30 09:18:16 jam dockerd[1479]: time="2023-05-30T09:18:16.371129838+10:00" level=error msg="failed to populate fields for osl sandbox 0ef5358411a1bbe4d96d4d318b3a31243109436e47a1c6324ec54322be9a63da"
May 30 09:18:16 jam dockerd[1479]: time="2023-05-30T09:18:16.371269276+10:00" level=info msg="there are running containers, updated network configuration will not take affect"
May 30 09:18:16 jam dockerd[1479]: time="2023-05-30T09:18:16.371445983+10:00" level=info msg="Loading containers: done."
May 30 09:18:16 jam dockerd[1479]: time="2023-05-30T09:18:16.385157565+10:00" level=info msg="Docker daemon" commit=659604f graphdriver=overlay2 version=24.0.2
May 30 09:18:16 jam dockerd[1479]: time="2023-05-30T09:18:16.385214548+10:00" level=info msg="Daemon has completed initialization"
May 30 09:18:16 jam dockerd[1479]: time="2023-05-30T09:18:16.402196640+10:00" level=info msg="API listen on /run/docker.sock"
May 30 09:18:16 jam systemd[1]: Started Docker Application Container Engine.
ubuntu@jam:~$
```
Observe that the embedded DNS server is broken
```
ubuntu@jam:~$ docker exec t-cona-1 ping conb
ping: bad address 'conb'
ubuntu@jam:~$ docker exec t-conb-1 ping cona
ping: bad address 'cona'
ubuntu@jam:~$
```

### Expected behavior

With `live-restore=true` in play, the embedded DNS server should have operated correctly after the docker daemon restart

### docker version

```bash
Client: Docker Engine - Community
 Version:           24.0.2
 API version:       1.43
 Go version:        go1.20.4
 Git commit:        cb74dfc
 Built:             Thu May 25 21:51:00 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          24.0.2
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.4
  Git commit:       659604f
  Built:            Thu May 25 21:51:00 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.21
  GitCommit:        3dce8eb055cbb6872793272b4f20ed16117344f8
 runc:
  Version:          1.1.7
  GitCommit:        v1.1.7-0-g860f061
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
```


### docker info

```bash
Client: Docker Engine - Community
 Version:    24.0.2
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.10.5
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.18.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 2
  Running: 2
  Paused: 0
  Stopped: 0
 Images: 1
 Server Version: 24.0.2
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: syslog
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 3dce8eb055cbb6872793272b4f20ed16117344f8
 runc version: v1.1.7-0-g860f061
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  userns
  cgroupns
 Kernel Version: 5.15.0-72-generic
 Operating System: Ubuntu 22.04.2 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 3.828GiB
 Name: jam
 ID: 68a7b3d7-8a6d-4204-921e-ef786a0e7ac3
 Docker Root Dir: /var/lib/docker/165536.165536
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  corunastylis:5000
  127.0.0.0/8
 Live Restore Enabled: true
 Default Address Pools:
   Base: 192.168.255.0/24, Size: 29
```


### Additional Info

We have placed an APT `hold` on `docker-ce docker-ce-cli docker-ce-rootless-extras docker-compose-plugin docker-buildx-plugin` until this bug is resolved.


Discussion / hints:
We have a similar issue after upgrade to 24.0.x and with live-restore enabled.

`docker stats` doesn't report net I/O stats until a container is restarted.
All values are reported as  0B / 0B, after the upgrade.

We didn't had this issue before 24.0.

To double-check; both @bruceesmith and @zas - this issue only reproduces with "live-restore" enabled? (I know there's some code-paths that have a special case for "live-restore", so if it's only with that feature enabled, that would help narrow down what code-paths to look into)
Same problem here since the update to 24.0 (also live-restore enabled)
/cc @akerouanton 
I don't think @zas issue is the same as what the OP reported. OP's issue might be related to  https://github.com/moby/moby/commit/50a4951ddc9c4f6d5aee9865b83ccf212e06664e:

- https://github.com/moby/moby/pull/44491
cc @corhere 
I can't reproduce the "Resolver Start failed" logs on master or 24.0 in a dev environment. Though given where those logs could have come from, it does suggest that the resolver is not running in the container's network namespace. I don't think #44491 is to blame here. If anything, 8f5a9a741b70852bc6c9d675c6e0c4944022b467 would be the most likely cause.

Despite that, I can reproduce the issue at hand, that containers attached to a non-default bridge network cannot resolve other containers on the network after a daemon restart with live-restore.

<details><summary>Our resolver is forwarding the query externally</summary>
<p>

```
DEBU[2023-05-30T15:30:52.501039035Z] Name To resolve: pause2.
DEBU[2023-05-30T15:30:52.501123746Z] Name To resolve: pause2.
DEBU[2023-05-30T15:30:52.501303903Z] [resolver] forwarding query                   client-addr="udp:172.19.0.2:60886" dns-server="udp:192.168.65.7:53" question=";pause2.\tIN\t A"
DEBU[2023-05-30T15:30:52.501340047Z] [resolver] forwarding query                   client-addr="udp:172.19.0.2:36411" dns-server="udp:192.168.65.7:53" question=";pause2.\tIN\t AAAA"
DEBU[2023-05-30T15:30:52.513749203Z] [resolver] external DNS udp:192.168.65.7 returned response with no answers:
;; opcode: QUERY, status: NXDOMAIN, id: 64694
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;pause2.	IN	 AAAA
DEBU[2023-05-30T15:30:52.513968068Z] [resolver] external DNS udp:192.168.65.7 returned response with no answers:
;; opcode: QUERY, status: NXDOMAIN, id: 64504
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;pause2.	IN	 A
ping: bad address 'pause2'
```

</p>
</details> 

I'll keep digging.
The `failed to populate fields for osl sandbox` logs are directly related. And they helpfully fail to log the actual error, because libnetwork.
https://github.com/moby/moby/blob/be220af9fb36e9baa9a75bbc41f784260aa6f96e/libnetwork/sandbox_store.go#L262-L265

After modifying it to actually log the error, it is revealed to be
> wrong iface name in restore osl sandbox interface: vethbbeeeaceth

https://github.com/moby/moby/blob/d5dc675d373de98ea181e0bec8023ba746d82ab2/libnetwork/osl/namespace_linux.go#L472-L478

Oh, libnetwork.

Anyway, the root cause is that f0be4d126dacb19a35046536b041a5989ff0dc95 accidentally [introduced a functional change](https://github.com/moby/moby/commit/f0be4d126dacb19a35046536b041a5989ff0dc95#diff-bb85ab0df0fcb23d25c96e7eea50b11f884be602452b720e0201d52911706344L819-R821) which broke sandbox restore.
By the way, we also have "failed to populate fields for osl sandbox" errors in logs.

We use iptables=false in our setup along live-restore=true.
We noticed a failure because we have some alerts based on docker net I/O, and metrics are clearly incorrect (=zero) just at the time of the upgrade.

We first thought it was due to upgrade from 23.x to 24.x with live-restore, so we restarted containers and it worked again. But after that we did a 24.0.1 to 24.0.2 upgrade (which should be supported by live-restore since that's a minor version upgrade), and it broke again.

> Anyway, the root cause is that https://github.com/moby/moby/commit/f0be4d126dacb19a35046536b041a5989ff0dc95 accidentally [introduced a functional change](https://github.com/moby/moby/commit/f0be4d126dacb19a35046536b041a5989ff0dc95#diff-bb85ab0df0fcb23d25c96e7eea50b11f884be602452b720e0201d52911706344L819-R821) which broke sandbox restore.

🤦 Right... had to be a literal `+` in between 🙈 

```go
Ifaces[i.srcName+"+"+i.dstPrefix] = ifaceOptions
```

@zas please file a separate issue for `docker stats`.
> @zas please file a separate issue for `docker stats`.

https://github.com/moby/moby/issues/45653

Fix patch:
diff --git a/libnetwork/resolver_unix.go b/libnetwork/resolver_unix.go
index 7b0511bcff493..466a89db087ca 100644
--- a/libnetwork/resolver_unix.go
+++ b/libnetwork/resolver_unix.go
@@ -38,8 +38,7 @@ func (r *Resolver) setupIPTable() error {
 		iptable := iptables.GetIptable(iptables.IPv4)
 
 		// insert outputChain and postroutingchain
-		err := iptable.RawCombinedOutputNative("-t", "nat", "-C", "OUTPUT", "-d", resolverIP, "-j", outputChain)
-		if err == nil {
+		if iptable.ExistsNative("nat", "OUTPUT", "-d", resolverIP, "-j", outputChain) {
 			if err := iptable.RawCombinedOutputNative("-t", "nat", "-F", outputChain); err != nil {
 				setupErr = err
 				return
@@ -55,8 +54,7 @@ func (r *Resolver) setupIPTable() error {
 			}
 		}
 
-		err = iptable.RawCombinedOutputNative("-t", "nat", "-C", "POSTROUTING", "-d", resolverIP, "-j", postroutingChain)
-		if err == nil {
+		if iptable.ExistsNative("nat", "POSTROUTING", "-d", resolverIP, "-j", postroutingChain) {
 			if err := iptable.RawCombinedOutputNative("-t", "nat", "-F", postroutingChain); err != nil {
 				setupErr = err
 				return


Test patch: