Repository: moby/moby (language: go)

Issue:
docker stats net I/O do not work after 24.0.x upgrade, works after container restart
### Description

We run docker with `live-restore` enabled and `iptables` disabled.

With pre-24 versions we were used to make upgrades without issue, but when we upgraded to 24.0.1, our monitoring systems failed to retrieve docker containers net I/O stats.

After investigation, it appears `docker stats` do not show them either.

We also noticed errors in logs:

```
May 27 10:33:39 trille dockerd[622073]: time="2023-05-27T10:33:39.243194146Z" level=error msg="failed to populate fields for osl sandbox 67abfd820a74ce968552ae633d70594dd45e72915dbc79365ed1e97e7a387362"
May 27 10:33:39 trille dockerd[622073]: time="2023-05-27T10:33:39.243596763Z" level=error msg="failed to populate fields for osl sandbox 8189efe2c5341321692d45fd69a2dc6fc12140b398919e3dc78ec97055d2ae7d"
May 27 10:33:39 trille dockerd[622073]: time="2023-05-27T10:33:39.243972135Z" level=error msg="failed to populate fields for osl sandbox ae7c36e8222b5adecd6b8abe22894f50d5822b6cff2909b98cb4b83cb50c0599"
May 27 10:33:39 trille dockerd[622073]: time="2023-05-27T10:33:39.244346826Z" level=error msg="failed to populate fields for osl sandbox d733fd814745bfa09df3c8b3109b413788d5d96b4a7622adbd4f63714ede86a3"
May 27 10:33:39 trille dockerd[622073]: time="2023-05-27T10:33:39.244727886Z" level=error msg="failed to populate fields for osl sandbox 208c1f938e071ad10384b7e85b3b17438d589fc3d14d67022a625c0c9810f574"
May 27 10:33:39 trille dockerd[622073]: time="2023-05-27T10:33:39.245092720Z" level=error msg="failed to populate fields for osl sandbox 3f855669b02f7642b472080249c2a580c5fccba08b39362a4d8d19526359d42e"
May 27 10:33:39 trille dockerd[622073]: time="2023-05-27T10:33:39.245489716Z" level=error msg="failed to populate fields for osl sandbox 360f7f8af8121c78bbcc718e3dfc74a94ca8cd6cd9beae785813d911242daf33"
May 27 10:33:39 trille dockerd[622073]: time="2023-05-27T10:33:39.245856577Z" level=error msg="failed to populate fields for osl sandbox 39271985b3b9e8b4ceadce0ad42741ff2858873a3cc199611b421099dc75b550"
May 27 10:33:39 trille dockerd[622073]: time="2023-05-27T10:33:39.246958805Z" level=error msg="failed to populate fields for osl sandbox cfae9b4cd4ed57c869513cfd502ab484b1d523560b18c52e7861aca9905e586e"
May 27 10:33:39 trille dockerd[622073]: time="2023-05-27T10:33:39.247697784Z" level=error msg="failed to populate fields for osl sandbox 1700f904e5cf331891916a871d2e97b2341c904ac317c660efa43687255da9b5"
May 27 10:33:39 trille dockerd[622073]: time="2023-05-27T10:33:39.248084187Z" level=error msg="failed to populate fields for osl sandbox 1f71901cf67d829a19db7a2c404952247418c6846c15b4a0d725612b61d950b3"
May 27 10:33:39 trille dockerd[622073]: time="2023-05-27T10:33:39.248472626Z" level=error msg="failed to populate fields for osl sandbox 9649704effa323dc45215a7808ec09d18a1104548ef173fcb90b91b2fb30f4fd"
May 27 10:33:39 trille dockerd[622073]: time="2023-05-27T10:33:39.248855398Z" level=error msg="failed to populate fields for osl sandbox a48e709fd327c98f7b031dcee623e4b5354ccc9160ca013fce504945b7d1674d"
May 27 10:33:39 trille dockerd[622073]: time="2023-05-27T10:33:39.249654180Z" level=error msg="failed to populate fields for osl sandbox 440cd65de19a83813a6fc35862120ab934d9fe4261d533926195c7bcfd439d8d"
May 27 10:33:39 trille dockerd[622073]: time="2023-05-27T10:33:39.250046212Z" level=error msg="failed to populate fields for osl sandbox 4c2fa2735c5fba0d558d06a4daa8e6f150a8a0366d7cd4f4283091e37f52e4de"
May 27 10:33:39 trille dockerd[622073]: time="2023-05-27T10:33:39.250781711Z" level=error msg="failed to populate fields for osl sandbox ee60047aa69c6efa813b02d4f79c5bc1721eadcb1d18eabb7125c3b89b642d1d"
May 27 10:33:39 trille dockerd[622073]: time="2023-05-27T10:33:39.251210452Z" level=error msg="failed to populate fields for osl sandbox 3b7edff711835f691adf3d3e0dd4446a545132b378054682671149d23cde98e8"
May 27 10:33:39 trille dockerd[622073]: time="2023-05-27T10:33:39.251604818Z" level=error msg="failed to populate fields for osl sandbox 4ecf6f9887014db6fdfc6ede7c639cf4f2843b88bd7c9383035f0c74c0337885"
```

That's for all containers running on this node, just after the upgrade (Ubuntu 20.04).




### Reproduce

Runs `systemctl restart docker` with live-restore = true

### Expected behavior

Net I/O should work even after the upgrade.

### docker version

```bash
Client: Docker Engine - Community
 Version:           24.0.2
 API version:       1.43
 Go version:        go1.20.4
 Git commit:        cb74dfc
 Built:             Thu May 25 21:52:22 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          24.0.2
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.4
  Git commit:       659604f
  Built:            Thu May 25 21:52:22 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.21
  GitCommit:        3dce8eb055cbb6872793272b4f20ed16117344f8
 runc:
  Version:          1.1.7
  GitCommit:        v1.1.7-0-g860f061
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
```


### docker info

```bash
Client: Docker Engine - Community
 Version:    24.0.2
 Context:    default
 Debug Mode: false

Server:
 Containers: 25
  Running: 24
  Paused: 0
  Stopped: 1
 Images: 124
 Server Version: 24.0.2
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 3dce8eb055cbb6872793272b4f20ed16117344f8
 runc version: v1.1.7-0-g860f061
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.15.0-72-generic
 Operating System: Ubuntu 20.04.6 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 62.59GiB
 Name: trille
 ID: LDN5:JIPJ:YCQZ:ZTE2:6JHV:BGAJ:T3QP:2YJ6:HSCY:IZZZ:IN3C:P6XD
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: true

WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
```


### Additional Info

_No response_


Discussion / hints:

Fix patch:
diff --git a/libnetwork/osl/namespace_linux.go b/libnetwork/osl/namespace_linux.go
index 9f22d8077208b..d7d2fe2d63769 100644
--- a/libnetwork/osl/namespace_linux.go
+++ b/libnetwork/osl/namespace_linux.go
@@ -470,16 +470,10 @@ func (n *networkNamespace) Destroy() error {
 }
 
 // Restore restore the network namespace
-func (n *networkNamespace) Restore(ifsopt map[string][]IfaceOption, routes []*types.StaticRoute, gw net.IP, gw6 net.IP) error {
+func (n *networkNamespace) Restore(ifsopt map[Iface][]IfaceOption, routes []*types.StaticRoute, gw net.IP, gw6 net.IP) error {
 	// restore interfaces
 	for name, opts := range ifsopt {
-		if !strings.Contains(name, "+") {
-			return fmt.Errorf("wrong iface name in restore osl sandbox interface: %s", name)
-		}
-		seps := strings.Split(name, "+")
-		srcName := seps[0]
-		dstPrefix := seps[1]
-		i := &nwIface{srcName: srcName, dstName: dstPrefix, ns: n}
+		i := &nwIface{srcName: name.SrcName, dstName: name.DstPrefix, ns: n}
 		i.processInterfaceOptions(opts...)
 		if i.master != "" {
 			i.dstMaster = n.findDst(i.master, true)
@@ -531,7 +525,7 @@ func (n *networkNamespace) Restore(ifsopt map[string][]IfaceOption, routes []*ty
 			}
 
 			var index int
-			indexStr := strings.TrimPrefix(i.dstName, dstPrefix)
+			indexStr := strings.TrimPrefix(i.dstName, name.DstPrefix)
 			if indexStr != "" {
 				index, err = strconv.Atoi(indexStr)
 				if err != nil {
@@ -540,8 +534,8 @@ func (n *networkNamespace) Restore(ifsopt map[string][]IfaceOption, routes []*ty
 			}
 			index++
 			n.Lock()
-			if index > n.nextIfIndex[dstPrefix] {
-				n.nextIfIndex[dstPrefix] = index
+			if index > n.nextIfIndex[name.DstPrefix] {
+				n.nextIfIndex[name.DstPrefix] = index
 			}
 			n.iFaces = append(n.iFaces, i)
 			n.Unlock()
diff --git a/libnetwork/osl/sandbox.go b/libnetwork/osl/sandbox.go
index 84833167e645e..9af7e46d234b6 100644
--- a/libnetwork/osl/sandbox.go
+++ b/libnetwork/osl/sandbox.go
@@ -17,6 +17,10 @@ const (
 	SandboxTypeLoadBalancer = iota
 )
 
+type Iface struct {
+	SrcName, DstPrefix string
+}
+
 // IfaceOption is a function option type to set interface options.
 type IfaceOption func(i *nwIface)
 
@@ -89,7 +93,7 @@ type Sandbox interface {
 	Destroy() error
 
 	// Restore restores the sandbox.
-	Restore(ifsopt map[string][]IfaceOption, routes []*types.StaticRoute, gw net.IP, gw6 net.IP) error
+	Restore(ifsopt map[Iface][]IfaceOption, routes []*types.StaticRoute, gw net.IP, gw6 net.IP) error
 
 	// ApplyOSTweaks applies operating system specific knobs on the sandbox.
 	ApplyOSTweaks([]SandboxType)
diff --git a/libnetwork/sandbox.go b/libnetwork/sandbox.go
index 194844ca7b804..aceb7b9be5b81 100644
--- a/libnetwork/sandbox.go
+++ b/libnetwork/sandbox.go
@@ -765,7 +765,7 @@ func (sb *Sandbox) restoreOslSandbox() error {
 	var routes []*types.StaticRoute
 
 	// restore osl sandbox
-	Ifaces := make(map[string][]osl.IfaceOption)
+	Ifaces := make(map[osl.Iface][]osl.IfaceOption)
 	for _, ep := range sb.endpoints {
 		ep.mu.Lock()
 		joinInfo := ep.joinInfo
@@ -790,7 +790,7 @@ func (sb *Sandbox) restoreOslSandbox() error {
 		if len(i.llAddrs) != 0 {
 			ifaceOptions = append(ifaceOptions, sb.osSbox.InterfaceOptions().LinkLocalAddresses(i.llAddrs))
 		}
-		Ifaces[i.srcName+i.dstPrefix] = ifaceOptions
+		Ifaces[osl.Iface{SrcName: i.srcName, DstPrefix: i.dstPrefix}] = ifaceOptions
 		if joinInfo != nil {
 			routes = append(routes, joinInfo.StaticRoutes...)
 		}
diff --git a/libnetwork/sandbox_store.go b/libnetwork/sandbox_store.go
index 92eaf7a1554ee..fc6b0071ff7f2 100644
--- a/libnetwork/sandbox_store.go
+++ b/libnetwork/sandbox_store.go
@@ -260,7 +260,7 @@ func (c *Controller) sandboxCleanup(activeSandboxes map[string]interface{}) {
 		// reconstruct osl sandbox field
 		if !sb.config.useDefaultSandBox {
 			if err := sb.restoreOslSandbox(); err != nil {
-				logrus.Errorf("failed to populate fields for osl sandbox %s", sb.ID())
+				logrus.Errorf("failed to populate fields for osl sandbox %s: %v", sb.ID(), err)
 				continue
 			}
 		} else {


Test patch: