Repository: moby/moby (language: go)

Issue:
Rootful-in-Rootless dind doesn't work since systemd v250 (due to oom score adj): `runc create failed: unable to start container process: can't get final child's PID from pipe: EOF: unknown.`
### Description

Rootful-in-Rootless dind doesn't work since systemd v250 (Dec 2021): 

```console
$ docker run -d --name dind --privileged docker:24.0.6-dind

$ docker exec dind docker run --rm hello-world
docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: can't get final child's PID from pipe: EOF: unknown.
```

This is because systemd v250 bumped up the default oom scores for user sessions:
- https://github.com/systemd/systemd/pull/20893

### Reproduce

See above

### Expected behavior

It should work

### docker version (parent)

```bash
Client: Docker Engine - Community
 Version:           24.0.6
 API version:       1.43
 Go version:        go1.20.7
 Git commit:        ed223bc
 Built:             Mon Sep  4 12:31:40 2023
 OS/Arch:           linux/amd64
 Context:           rootless

Server: Docker Engine - Community
 Engine:
  Version:          24.0.6
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.7
  Git commit:       1a79695
  Built:            Mon Sep  4 12:31:40 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.24
  GitCommit:        61f9fd88f79f081d64d6fa3bb1a0dc71ec870523
 runc:
  Version:          1.1.9
  GitCommit:        v1.1.9-0-gccaecfc
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
 rootlesskit:
  Version:          1.1.1
  ApiVersion:       1.1.1
  NetworkDriver:    slirp4netns
  PortDriver:       builtin
  StateDir:         /tmp/rootlesskit396465062
 slirp4netns:
  Version:          1.2.0
  GitCommit:        656041d45cfca7a4176f6b7eed9e4fe6c11e8383
```


### docker info (parent)

```bash
Client: Docker Engine - Community
 Version:    24.0.6
 Context:    rootless
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.11.2
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.21.0
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 1
  Running: 1
  Paused: 0
  Stopped: 0
 Images: 2
 Server Version: 24.0.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: false
  userxattr: true
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 61f9fd88f79f081d64d6fa3bb1a0dc71ec870523
 runc version: v1.1.9-0-gccaecfc
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  rootless
  cgroupns
 Kernel Version: 6.2.0-33-generic
 Operating System: Ubuntu 23.04
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 3.809GiB
 Name: lima-docker
 ID: 58287749-998e-420b-8a8f-93b483ed5102
 Docker Root Dir: /home/suda.linux/.local/share/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
```


### Additional Info

## Workaround

Set `--oom-score-adj=100` in the child `docker run` (when the container's `oom_score_adj` is set to 200)

```console
$ docker exec dind cat /proc/1/oom_score_adj
200

$ docker exec dind docker run --rm --oom-score-adj=100 hello-world

Hello from Docker!
...
```


Discussion / hints:
I'm slightly confused; do we set an oom-score-adjust by default? (And if not, does that mean that it now must be set always)?
> I'm slightly confused; do we set an oom-score-adjust by default? (And if not, does that mean that it now must be set always)?

Set to `0` by default, and there is no way to unset it currently, as `HostConfig` can't distinguish zero value from "unset".

https://github.com/moby/moby/blob/e2c23ed57de3bc768b928d0350b9c65d03d45de2/api/types/container/hostconfig.go#L442

Fix patch:
diff --git a/daemon/oci_linux.go b/daemon/oci_linux.go
index fc19ca04e012a..45f19026aa183 100644
--- a/daemon/oci_linux.go
+++ b/daemon/oci_linux.go
@@ -110,6 +110,15 @@ func withRootless(daemon *Daemon, daemonCfg *dconfig.Config) coci.SpecOpts {
 	}
 }
 
+// withRootfulInRootless is used for "rootful-in-rootless" dind;
+// the daemon is running in UserNS but has no access to RootlessKit API socket, host filesystem, etc.
+func withRootfulInRootless(daemon *Daemon, daemonCfg *dconfig.Config) coci.SpecOpts {
+	return func(_ context.Context, _ coci.Client, _ *containers.Container, s *coci.Spec) error {
+		specconv.ToRootfulInRootless(s)
+		return nil
+	}
+}
+
 // WithOOMScore sets the oom score
 func WithOOMScore(score *int) coci.SpecOpts {
 	return func(ctx context.Context, _ coci.Client, _ *containers.Container, s *coci.Spec) error {
@@ -1126,6 +1135,8 @@ func (daemon *Daemon) createSpec(ctx context.Context, daemonCfg *configStore, c
 	}
 	if daemonCfg.Rootless {
 		opts = append(opts, withRootless(daemon, &daemonCfg.Config))
+	} else if userns.RunningInUserNS() {
+		opts = append(opts, withRootfulInRootless(daemon, &daemonCfg.Config))
 	}
 
 	var snapshotter, snapshotKey string
diff --git a/pkg/rootless/specconv/specconv_linux.go b/pkg/rootless/specconv/specconv_linux.go
index d66b67f381b53..9a8db2da01629 100644
--- a/pkg/rootless/specconv/specconv_linux.go
+++ b/pkg/rootless/specconv/specconv_linux.go
@@ -13,6 +13,20 @@ import (
 	specs "github.com/opencontainers/runtime-spec/specs-go"
 )
 
+// ToRootfulInRootless is used for "rootful-in-rootless" dind;
+// the daemon is running in UserNS but has no access to RootlessKit API socket, host filesystem, etc.
+//
+// This fuction does:
+// * Fix up OOMScoreAdj (needed since systemd v250: https://github.com/moby/moby/issues/46563)
+func ToRootfulInRootless(spec *specs.Spec) {
+	if spec.Process == nil || spec.Process.OOMScoreAdj == nil {
+		return
+	}
+	if currentOOMScoreAdj := getCurrentOOMScoreAdj(); *spec.Process.OOMScoreAdj < currentOOMScoreAdj {
+		*spec.Process.OOMScoreAdj = currentOOMScoreAdj
+	}
+}
+
 // ToRootless converts spec to be compatible with "rootless" runc.
 // * Remove non-supported cgroups
 // * Fix up OOMScoreAdj


Test patch: