Repository: moby/moby (language: go)

Issue:
v25 Regression: Docker-Rootless Fails to Enable Plugins
### Description

Docker Rootless fails to enable plugins on Docker v25.0.1 (with Debian12).
I have replicated this error with:
- ~~vieux/sshfs~~ (this plugin doesn't show this issue, see https://github.com/moby/moby/issues/47248#issuecomment-1926403623)
- ghcr.io/ibm/docker-logdna

```
chris@debian:~$ docker plugin install vieux/sshfs
Plugin "vieux/sshfs" is requesting the following privileges:
 - network: [host]
 - mount: [/var/lib/docker/plugins/]
 - mount: []
 - device: [/dev/fuse]
 - capabilities: [CAP_SYS_ADMIN]
Do you grant the above permissions? [y/N] y
latest: Pulling from vieux/sshfs
Digest: sha256:1d3c3e42c12138da5ef7873b97f7f32cf99fb6edde75fa4f0bcf9ed277855811
52d435ada6a4: Complete
Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/var/lib/docker/plugins/" to rootfs at "/mnt/state": stat /var/lib/docker/plugins/: permission denied: unknown
```

This error also occurs when you perform a `docker plugin enable vieux/sshfs` afterwards. So this is an issue with the enabling process, not the install process.

You can see more of the error with journalctl:
```
chris@debian:~$ journalctl --user -fu docker
Jan 30 09:17:43 debian dockerd-rootless.sh[725]: time="2024-01-30T09:17:43.243597136+01:00" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1
Jan 30 09:17:43 debian dockerd-rootless.sh[725]: time="2024-01-30T09:17:43.243929909+01:00" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1
Jan 30 09:17:43 debian dockerd-rootless.sh[725]: time="2024-01-30T09:17:43.243941735+01:00" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
Jan 30 09:17:43 debian dockerd-rootless.sh[725]: time="2024-01-30T09:17:43.244399134+01:00" level=info msg="starting signal loop" namespace=plugins.moby path=/run/.ro3508243726/user/1000/docker/containerd/daemon/io.containerd.runtime.v2.task/plugins.moby/6fddc473ff1bea75de4335bbe59954be87f866647327e9a3e2cb7f8a97fd025a pid=951 runtime=io.containerd.runc.v2
Jan 30 09:17:43 debian dockerd-rootless.sh[725]: time="2024-01-30T09:17:43.326335231+01:00" level=info msg="shim disconnected" id=6fddc473ff1bea75de4335bbe59954be87f866647327e9a3e2cb7f8a97fd025a
Jan 30 09:17:43 debian dockerd-rootless.sh[725]: time="2024-01-30T09:17:43.326443552+01:00" level=warning msg="cleaning up after shim disconnected" id=6fddc473ff1bea75de4335bbe59954be87f866647327e9a3e2cb7f8a97fd025a namespace=plugins.moby
Jan 30 09:17:43 debian dockerd-rootless.sh[725]: time="2024-01-30T09:17:43.326460424+01:00" level=info msg="cleaning up dead shim"
Jan 30 09:17:43 debian dockerd-rootless.sh[725]: time="2024-01-30T09:17:43.338050307+01:00" level=warning msg="cleanup warnings time=\"2024-01-30T09:17:43+01:00\" level=info msg=\"starting signal loop\" namespace=plugins.moby pid=980 runtime=io.containerd.runc.v2\ntime=\"2024-01-30T09:17:43+01:00\" level=warning msg=\"failed to read init pid file\" error=\"open /run/.ro3508243726/user/1000/docker/containerd/daemon/io.containerd.runtime.v2.task/plugins.moby/6fddc473ff1bea75de4335bbe59954be87f866647327e9a3e2cb7f8a97fd025a/init.pid: no such file or directory\"runtime=io.containerd.runc.v2\n"
Jan 30 09:17:43 debian dockerd-rootless.sh[725]: time="2024-01-30T09:17:43.338428072+01:00" level=error msg="copy shim log" error="read /proc/self/fd/14: file already closed"
Jan 30 09:17:43 debian dockerd-rootless.sh[699]: time="2024-01-30T09:17:43.344743825+01:00" level=error msg="Handler for POST /v1.44/plugins/vieux/sshfs:latest/enable returned error: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting \"/var/lib/docker/plugins/\" to rootfs at \"/mnt/state\": stat /var/lib/docker/plugins/: permission denied: unknown"
```

I looked inside of the `/run/user/1000/docker/containerd/daemon/io.containerd.runtime.v2.task/plugins.moby/6fddc473ff1bea75de4335bbe59954be87f866647327e9a3e2cb7f8a97fd025a` directory (before it got deleted after the plugin fails to enable) and the `init.pid` indeed does not exist.

The problem does not exist on the same machine with Docker Rootfull.

### Reproduce

I reproduced this bug on a clean Debian12 VM (using `debian-12.4.0-amd64-netinst.iso`). This is the entire history:
```
    1  ls
    2  sudo apt-get update
    3  sudo apt-get install ca-certificates curl
    4  sudo install -m 0755 -d /etc/apt/keyrings
    5  sudo curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc
    6  sudo chmod a+r /etc/apt/keyrings/docker.asc
    7  echo   "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian \
    8    $(. /etc/os-release && echo "$VERSION_CODENAME") stable" |   sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
    9  sudo apt-get update
   10  sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
   11  sudo docker run --rm hello-world
   12  sudo apt-get install -y dbus-user-session
   13  sudo apt-get install -y fuse-overlayfs
   14  slirp4netns --version
   15  sudo systemctl disable --now docker.service docker.socket
   16  dockerd-rootless-setuptool.sh install
   17  sudo apt-get install uidmap
   18  dockerd-rootless-setuptool.sh install
   19  systemctl --user start docker
   20  systemctl --user enable docker
   21  sudo loginctl enable-linger $(whoami)
   22  docker run --rm hello-world
   23  docker plugin install ghcr.io/ibm/docker-logdna:1.0.1
   24  docker context use rootless
   25  docker plugin enable ghcr.io/ibm/docker-logdna:1.0.1
   26  reboot
   27  sudo reboot
   28  docker plugin install vieux/sshfs
   29*
   30  docker plugin install vieux/sshfs
   31  journalctl --user -fu docker
   32  history
```

I performed the official [Debian12 install instructions](https://docs.docker.com/engine/install/debian) and the [Docker Rootless installation instructions](https://docs.docker.com/engine/security/rootless).

### Expected behavior

`docker plugin enable vieux/sshfs` should not fail and properly enable the plugin.

### docker version

```bash
Version:           25.0.1
 API version:       1.44
 Go version:        go1.21.6
 Git commit:        29cf629
 Built:             Tue Jan 23 23:09:46 2024
 OS/Arch:           linux/amd64
 Context:           rootless

Server: Docker Engine - Community
 Engine:
  Version:          25.0.1
  API version:      1.44 (minimum version 1.24)
  Go version:       go1.21.6
  Git commit:       71fa3ab
  Built:            Tue Jan 23 23:09:46 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.27
  GitCommit:        a1496014c916f9e62104b33d1bb5bd03b0858e59
 runc:
  Version:          1.1.11
  GitCommit:        v1.1.11-0-g4bccb38
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
 rootlesskit:
  Version:          2.0.0
  ApiVersion:       1.1.1
  NetworkDriver:    slirp4netns
  PortDriver:       builtin
  StateDir:         /run/user/1000/dockerd-rootless
 slirp4netns:
  Version:          1.2.0
  GitCommit:        656041d45cfca7a4176f6b7eed9e4fe6c11e8383
```


### docker info

```bash
Client: Docker Engine - Community
 Version:    25.0.1
 Context:    rootless
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.12.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.24.2
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 1
 Server Version: 25.0.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: false
  userxattr: true
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: a1496014c916f9e62104b33d1bb5bd03b0858e59
 runc version: v1.1.11-0-g4bccb38
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  rootless
  cgroupns
 Kernel Version: 6.1.0-17-amd64
 Operating System: Debian GNU/Linux 12 (bookworm)
 OSType: linux
 Architecture: x86_64
 CPUs: 12
 Total Memory: 30.4GiB
 Name: debian
 ID: e90b8494-9305-4e46-9206-e397cdea0c59
 Docker Root Dir: /home/chris/.local/share/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No cpuset support
WARNING: No io.weight support
WARNING: No io.weight (per device) support
WARNING: No io.max (rbps) support
WARNING: No io.max (wbps) support
WARNING: No io.max (riops) support
WARNING: No io.max (wiops) support
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
```


### Additional Info

_No response_


Discussion / hints:
Hmm, looks like the vieux/sshfs plugin [is authored incorrectly](https://github.com/vieux/docker-volume-sshfs/blob/1e0cd2fcb72d6af0a2ad4689faed8c312124517c/config.json#L32-L44), making it incompatible with rootless docker. Managed plugins should not be requesting a bind-mount of the (default path to the rootful) [_unmanaged_ plugin directory](https://docs.docker.com/engine/extend/plugin_api/#plugin-discovery).

This is not a v25 regression; that plugin appears to never have been compatible with rootless mode. I can reproduce the issue on `docker:23.0.6-dind-rootless`.

```console
/ $ docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.11.0
    Path:     /usr/local/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.19.0
    Path:     /usr/local/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 23.0.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: false
  userxattr: true
 Logging Driver: json-file
 Cgroup Driver: none
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 3dce8eb055cbb6872793272b4f20ed16117344f8
 runc version: v1.1.7-0-g860f061
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  rootless
  cgroupns
 Kernel Version: 6.6.12-linuxkit
 Operating System: Alpine Linux v3.18
 OSType: linux
 Architecture: x86_64
 CPUs: 12
 Total Memory: 5.787GiB
 Name: ef1780590d4f
 ID: c81669c8-7d5e-48aa-9f21-6aa19e8aa773
 Docker Root Dir: /home/rootless/.local/share/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

WARNING: Running in rootless-mode without cgroups. Systemd is required to enable cgroups in rootless-mode.
/ $ docker plugin install vieux/sshfs
Plugin "vieux/sshfs" is requesting the following privileges:
 - network: [host]
 - mount: [/var/lib/docker/plugins/]
 - mount: []
 - device: [/dev/fuse]
 - capabilities: [CAP_SYS_ADMIN]
Do you grant the above permissions? [y/N] y
latest: Pulling from vieux/sshfs
Digest: sha256:1d3c3e42c12138da5ef7873b97f7f32cf99fb6edde75fa4f0bcf9ed277855811
52d435ada6a4: Complete
Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/var/lib/docker/plugins/" to rootfs at "/mnt/state": stat /var/lib/docker/plugins/: no such file or directory: unknown
```

There is nothing we can do.
Thanks for checking that, @corhere!

I tested it myself and vieux/sshfs does not show the problem, though ghcr.io/ibm/docker-logdna does. It is authored correctly. See: https://github.com/IBM/docker_logdna/blob/main/plugin/config.json
```json
{
    "description": "logdna plugin",
    "documentation": "https://github.com/ibm/docker_logdna",
    "entrypoint": [
        "/docker_logdna"
    ],
    "network": {
        "type": "host"
    },
    "interface": {
        "types": [
            "docker.logdriver/1.0"
        ],
        "socket": "logdna.sock"
    }
}
```

Installing the plugin on Debian 12 with Docker v24.0.9 works. Installing it on Docker 25.0.1 does not (as tested in the issue description). This is the log for v24.0.9:
```
chris@debian:~$ docker plugin install ghcr.io/ibm/docker-logdna:1.0.1
Plugin "ghcr.io/ibm/docker-logdna:1.0.1" is requesting the following privileges:
 - network: [host]
Do you grant the above permissions? [y/N] y
1.0.1: Pulling from ghcr.io/ibm/docker-logdna
Digest: sha256:b4b73b2e41dd9a23cc5279b0a9aca53e0e54f9084600de1bc53cb193fd585d26
cd173d625410: Complete
Installed plugin ghcr.io/ibm/docker-logdna:1.0.1
```

Here is the history for v24.0.9:
```
    1  ps --version
    2  iptables --version
    3  sudo apt install iptables
    4  iptables --version
    5  sudo iptables --version
    6  git --version
    7  sudo apt install git
    8  wget
    9  wget https://download.docker.com/linux/static/stable/x86_64/docker-24.0.9.tgz
   10  tar xzvf docker-24.0.9.tgz
   11  sudo cp docker/* /usr/bin/
   12  sudo dockerd &
   13  sudo docker run --rm hello-world
   14  top
   15  pgrep dockerd
   16  sudo reboot
   17  sudo apt install -y uidmap
   18  sudo apt install -y dbus-user-session
   19  sudo apt install -y fuse-overlayfs
   20  slirp4netns --version
   21  sudo apt install slirp4netns
   22  slirp4netns --version
   23  sudo apt install docker-ce-rootless-extras
   24  wget https://download.docker.com/linux/static/stable/x86_64/docker-rootless-extras-24.0.9.tgz
   25  ls
   26  tar xzvf docker-rootless-extras-24.0.9.tgz
   27  docker-rootless-extras/dockerd-rootless-setuptool.sh install
   28  sudo cp docker-rootless-extras/* /usr/bin/
   29  dockerd-rootless-setuptool.sh install
   30  systemctl --user start docker
   31  systemctl --user enable docker
   32  sudo loginctl enable-linger $(whoami)
   33  sudo reboot
   34  docker run --rm hello-world
   35  docker plugin install ghcr.io/ibm/docker-logdna:1.0.1
   36  docker plugin install vieux/sshfs
   37  docker --version
   38  history
```
Installing vieux/sshfs indeed fails.

This is the docker version:
```
Client:
 Version:           24.0.9
 API version:       1.43
 Go version:        go1.20.13
 Git commit:        2936816
 Built:             Thu Feb  1 00:47:46 2024
 OS/Arch:           linux/amd64
 Context:           rootless

Server: Docker Engine - Community
 Engine:
  Version:          24.0.9
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.13
  Git commit:       fca702d
  Built:            Thu Feb  1 00:49:16 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.7.13
  GitCommit:        7c3aca7a610df76212171d200ca3811ff6096eb8
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
 rootlesskit:
  Version:          1.1.1
  ApiVersion:       1.1.1
  NetworkDriver:    slirp4netns
  PortDriver:       builtin
  StateDir:         /tmp/rootlesskit499189580
 slirp4netns:
  Version:          1.2.0
  GitCommit:        656041d45cfca7a4176f6b7eed9e4fe6c11e8383
```

And this is docker info:
```
Client:
 Version:    24.0.9
 Context:    rootless
 Debug Mode: false

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 1
 Server Version: 24.0.9
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: false
  userxattr: true
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7c3aca7a610df76212171d200ca3811ff6096eb8
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  rootless
  cgroupns
 Kernel Version: 6.1.0-17-amd64
 Operating System: Debian GNU/Linux 12 (bookworm)
 OSType: linux
 Architecture: x86_64
 CPUs: 12
 Total Memory: 30.4GiB
 Name: debian
 ID: 5ebb78f4-3851-40e7-9897-19130167d40e
 Docker Root Dir: /home/chris/.local/share/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

WARNING: No cpuset support
WARNING: No io.weight support
WARNING: No io.weight (per device) support
WARNING: No io.max (rbps) support
WARNING: No io.max (wbps) support
WARNING: No io.max (riops) support
WARNING: No io.max (wiops) support
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
```

If there is anything I could test or do to help, just let me know.

Thanks
Chris
I stumbled through the moby contribution guides and tried a few more combinations. So far I haven't managed to reproduce the problem in the moby dev container. I also haven't managed to get enabling the plugin to work on the debian12 vm. These are a few versions I've tried:


### Doesn't enable on Debian12 VM
```
Client: Docker Engine - Community
 Version:           25.0.1
 API version:       1.44
 Go version:        go1.21.6
 Git commit:        29cf629
 Built:             Tue Jan 23 23:09:46 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          25.0.1
  API version:      1.44 (minimum version 1.24)
  Go version:       go1.21.6
  Git commit:       71fa3ab
  Built:            Tue Jan 23 23:09:46 2024
  OS/Arch:          linux/amd64
  Experimental:     true
 containerd:
  Version:          v1.7.13
  GitCommit:        7c3aca7a610df76212171d200ca3811ff6096eb8
 runc:
  Version:          1.1.11
  GitCommit:        v1.1.11-0-g4bccb38
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
 rootlesskit:
  Version:          2.0.0
  ApiVersion:       1.1.1
  NetworkDriver:    slirp4netns
  PortDriver:       slirp4netns
  StateDir:         /run/user/1000/dockerd-rootless
 slirp4netns:
  Version:          1.2.0
  GitCommit:        656041d45cfca7a4176f6b7eed9e4fe6c11e8383
```

### Doesn't enable on Debian12 VM
```
Client: Docker Engine - Community
 Version:           25.0.1
 API version:       1.44
 Go version:        go1.21.6
 Git commit:        29cf629
 Built:             Tue Jan 23 23:09:46 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          25.0.1
  API version:      1.44 (minimum version 1.24)
  Go version:       go1.21.6
  Git commit:       71fa3ab
  Built:            Tue Jan 23 23:09:46 2024
  OS/Arch:          linux/amd64
  Experimental:     true
 containerd:
  Version:          v1.7.12
  GitCommit:        71909c1814c544ac47ab91d2e8b84718e517bb99
 runc:
  Version:          1.1.11
  GitCommit:        v1.1.11-0-g4bccb38
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
 rootlesskit:
  Version:          2.0.0
  ApiVersion:       1.1.1
  NetworkDriver:    slirp4netns
  PortDriver:       slirp4netns
  StateDir:         /run/user/1000/dockerd-rootless
 slirp4netns:
  Version:          1.2.0
  GitCommit:        656041d45cfca7a4176f6b7eed9e4fe6c11e8383
```

### Works on moby dev container
```
Client:
 Version:           25.0.0
 API version:       1.44
 Go version:        go1.21.6
 Git commit:        e758fe5
 Built:             Thu Jan 18 17:09:01 2024
 OS/Arch:           linux/amd64
 Context:           default

Server:
 Engine:
  Version:          dev
  API version:      1.44 (minimum version 1.24)
  Go version:       go1.21.6
  Git commit:       71fa3ab079ec13d17257f86fa92db8d7f24802f1
  Built:            Mon Feb  5 14:13:34 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.7.12
  GitCommit:        71909c1814c544ac47ab91d2e8b84718e517bb99
 runc:
  Version:          1.1.11
  GitCommit:        v1.1.11-0-g4bccb38
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
 rootlesskit:
  Version:          2.0.0
  ApiVersion:       1.1.1
  NetworkDriver:    slirp4netns
  PortDriver:       builtin
  StateDir:         /home/test-user/.docker/run/dockerd-rootless
 slirp4netns:
  Version:          1.2.0
  GitCommit:        656041d45cfca7a4176f6b7eed9e4fe6c11e8383
```

### Works on moby dev container
```
Client:
 Version:           25.0.2.m
 API version:       1.44
 Go version:        go1.21.6
 Git commit:        29cf629222
 Built:             Mon Feb  5 13:06:34 2024
 OS/Arch:           linux/amd64
 Context:           default

Server:
 Engine:
  Version:          dev
  API version:      1.44 (minimum version 1.24)
  Go version:       go1.21.6
  Git commit:       71fa3ab079ec13d17257f86fa92db8d7f24802f1
  Built:            Mon Feb  5 13:22:21 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.7.12
  GitCommit:        71909c1814c544ac47ab91d2e8b84718e517bb99
 runc:
  Version:          1.1.11
  GitCommit:        v1.1.11-0-g4bccb38
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
 rootlesskit:
  Version:          2.0.0
  ApiVersion:       1.1.1
  NetworkDriver:    vpnkit
  PortDriver:       builtin
  StateDir:         /home/test-user/.docker/run/dockerd-rootless
 vpnkit:
  Version:          7f0eff0dd99b576c5474de53b4454a157c642834
```

I tried different dockerd versions, different docker cli versions, different containerd versions and using vpnkit vs slirp4netns. But the problem stays.
I don't know if that is any help, but it's all I got for now.
Just to rule out any problems with apt or the Debian repos I tried installing v25.0.1 without apt and the plugin can't be installed. So this is not a Debian issue.

```
chris@debian:~$ docker plugin install ghcr.io/ibm/docker-logdna:1.0.1
Plugin "ghcr.io/ibm/docker-logdna:1.0.1" is requesting the following privileges:
 - network: [host]
Do you grant the above permissions? [y/N] y
1.0.1: Pulling from ghcr.io/ibm/docker-logdna
Digest: sha256:b4b73b2e41dd9a23cc5279b0a9aca53e0e54f9084600de1bc53cb193fd585d26
cd173d625410: Complete
Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/etc/hosts" to rootfs at "/etc/hosts": mount /etc/hosts:/etc/hosts (via /proc/self/fd/6), flags: 0x5021: operation not permitted: unknown
```

History:
```
    1  ls
    2  history
    3  sudo apt install iptables
    4  sudo apt install git
    5  wget https://download.docker.com/linux/static/stable/x86_64/docker-25.0.1.tgz
    6  wget https://download.docker.com/linux/static/stable/x86_64/docker-rootless-extras-25.0.1.tgz
    7  ls
    8  tar xf docker-25.0.1.tgz
    9  tar xf docker-rootless-extras-25.0.1.tgz
   10  sudo cp docker/* /usr/bin/
   11  sudo docker-rootless-extras/* /usr/bin
   12  sudo cp docker-rootless-extras/* /usr/bin
   13  sudo dockerd &
   14  sudo docker run --rm hello-world
   15  sudo apt install -y uidmap
   16  sudo apt install -y dbus-user-session fuse-overlayfs
   17  sudo apt install slirp4netns -y
   18  dockerd-rootless-setuptool.sh install
   19  docker run --rm hello-world
   20  docker plugin install ghcr.io/ibm/docker-logdna:1.0.1
   21  history
```
> mount /etc/hosts:/etc/hosts (via /proc/self/fd/6), flags: 0x5021: operation not permitted

Interesting... What happens if you do `docker run --rm --net=host hello-world` on the affected install?
> Interesting... What happens if you do docker run --rm --net=host hello-world on the affected install?

The normal hello-world container starts as expected.

I'm currently in the process of doing a git bisect by compiling docker in the dev container and then copying the binary to a separate vm. That actually works and this is my progress:

- good: v24.0.9 fca702de7f71362c8d103073c7e4a1d0a467fadd
- bad: v25.0.1 71fa3ab079ec13d17257f86fa92db8d7f24802f1
- good: 8d9a40a820349860405e142249c55215aa4da8e3

Out of curiosity, what is the output of `sudo cat /proc/sys/kernel/unprivileged_userns_clone`
> Out of curiosity, what is the output of sudo cat /proc/sys/kernel/unprivileged_userns_clone
```
chris@debian:~$ sudo cat /proc/sys/kernel/unprivileged_userns_clone
[sudo] password for chris:
1
chris@debian:~$
```
The daemon managed-plugin code hasn't really changed between v24 and v25. I did notice a couple differences between the `docker version` output of the working v24.0.9 and broken v25 installs which might be pertinent.
| Version | runc | rootlesskit |
|--------|--------|--------|
| 24.0.9 | 1.1.12 | 1.1.1 |
| 25.0.1 | 1.1.11 | 2.0.0 | 

The only difference between runc 1.1.11 and 1.1.12 is the CVE fix. It's possible, albeit unlikely, that the change also unintentionally fixed an issue with bind-mounting /etc/hosts in rootless. Should be easy enough to rule out by trying to install the plugin on rootless v25 with runc 1.1.12 installed.

I have a feeling that [the upgrade of rootlesskit to v2.0.0 is related.](https://github.com/rootless-containers/rootlesskit/pull/415)

(cc @AkihiroSuda)
git bisect is done:
7d08d84b039d2f4661a2242e765a141e65943920 is the first bad commit
commit 7d08d84b039d2f4661a2242e765a141e65943920
Author: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
Date:   Tue Jan 16 23:56:04 2024 +0900

    dockerd-rootless.sh: set `rootlesskit --state-dir=DIR`

    Now the state dir is set to `${XDG_RUNTIME_DIR}/dockerd-rootless`.

    This is similar to `${XDG_RUNTIME_DIR}/containerd-rootless` used in nerdctl:
    https://github.com/containerd/nerdctl/blob/v1.7.2/extras/rootless/containerd-rootless.sh#L35

    Prior to this commit, the state dir was unset and a random dir under `/tmp` was used.
    (e.g., `/tmp/rootlesskit1869901982`)

    Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>

 contrib/dockerd-rootless.sh | 3 +++
 1 file changed, 3 insertions(+)
These are the tests I ran:
- good fca702de7f71362c8d103073c7e4a1d0a467fadd
- bad  71fa3ab079ec13d17257f86fa92db8d7f24802f1
- good 8d9a40a820349860405e142249c55215aa4da8e3
- good f10e182ca7b3a5facfe653f5a5e21529927e409e
- good f7ddfdbae6cd9ad7e820e296cc4e58687eadc853
- good afe281964d6699e152ef4dfe6c822dc99922e5a5
- good b7042cb767888a3ef249da1b535e18ecb9498c75
- good 60ed73bb917170e1005b141cf47d992cc9b35564
- bad  d25aa32c215b5a23873d186a45c3c015b25ebeae
- bad  66325f72718a7e4df4f27658cfee47e40e9e0179
- good 1fd682930ac11ada52dbd46d20aa7a53f879ab26
- bad  83de55b37040f21034f198d8e8afc763b78d4d6e
- bad  7d08d84b039d2f4661a2242e765a141e65943920
- good ae9fbcb82c6ef34270a90b4aaca0ed7b0ad3d5b3

> Should be easy enough to rule out by trying to install the plugin on rootless v25 with runc 1.1.12 installed.

I just ruled that out. The problem persists with these versions:
```
Client: Docker Engine - Community
 Version:           25.0.1
 API version:       1.44
 Go version:        go1.21.6
 Git commit:        29cf629
 Built:             Tue Jan 23 23:09:46 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          25.0.1
  API version:      1.44 (minimum version 1.24)
  Go version:       go1.21.6
  Git commit:       71fa3ab
  Built:            Tue Jan 23 23:09:46 2024
  OS/Arch:          linux/amd64
  Experimental:     true
 containerd:
  Version:          1.6.27
  GitCommit:        a1496014c916f9e62104b33d1bb5bd03b0858e59
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e946
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
 rootlesskit:
  Version:          2.0.0
  ApiVersion:       1.1.1
  NetworkDriver:    slirp4netns
  PortDriver:       slirp4netns
  StateDir:         /run/user/1000/dockerd-rootless
 slirp4netns:
  Version:          1.2.0
  GitCommit:        656041d45cfca7a4176f6b7eed9e4fe6c11e8383
```
> ghcr.io/ibm/docker-logdna

What is this plugin? Where is the source? Who built it?

Does it work if you downgrade RootlessKit binary to v1.1 ?

> What is this plugin? Where is the source? Who built it?

All that information has been provided in https://github.com/moby/moby/issues/47248#issuecomment-1926403623

Though I don't think it matters. The error logs clearly show that runc can't bind-mount the host's `/etc/hosts` file in a rootless environment. Managed plugins with `"network": {"type": "host"}` happen to be affected but unless there's some special-casing for the `moby` containerd namespace somewhere it seems unlikely that only plugins are affected, let alone a specific plugin.
> Does it work if you downgrade RootlessKit binary to v1.1 ?

Can confirm, that it works with Rootlesskit v1.1.1

```
 Client: Docker Engine - Community
 Version:           25.0.3
 API version:       1.44
 Go version:        go1.21.6
 Git commit:        4debf41
 Built:             Tue Feb  6 21:14:26 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          25.0.3
  API version:      1.44 (minimum version 1.24)
  Go version:       go1.21.6
  Git commit:       f417435
  Built:            Tue Feb  6 21:14:26 2024
  OS/Arch:          linux/amd64
  Experimental:     true
 containerd:
  Version:          1.6.28
  GitCommit:        ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
 rootlesskit:
  Version:          1.1.1
  ApiVersion:       1.1.1
  NetworkDriver:    slirp4netns
  PortDriver:       slirp4netns
  StateDir:         /tmp/rootlesskit2257322869
 slirp4netns:
  Version:          1.0.1
  GitCommit:        6a7b16babc95b6a3056b33fb45b74a6f62262dd4
```


Fix patch:
diff --git a/daemon/oci_linux.go b/daemon/oci_linux.go
index c7fdedcb31f10..a865dc9398fce 100644
--- a/daemon/oci_linux.go
+++ b/daemon/oci_linux.go
@@ -19,6 +19,7 @@ import (
 	"github.com/docker/docker/container"
 	dconfig "github.com/docker/docker/daemon/config"
 	"github.com/docker/docker/errdefs"
+	"github.com/docker/docker/internal/rootless/mountopts"
 	"github.com/docker/docker/oci"
 	"github.com/docker/docker/oci/caps"
 	"github.com/docker/docker/pkg/idtools"
@@ -31,7 +32,6 @@ import (
 	"github.com/opencontainers/runc/libcontainer/cgroups"
 	specs "github.com/opencontainers/runtime-spec/specs-go"
 	"github.com/pkg/errors"
-	"golang.org/x/sys/unix"
 )
 
 const inContainerInitPath = "/sbin/" + dconfig.DefaultInitBinary
@@ -468,38 +468,6 @@ func ensureSharedOrSlave(path string) error {
 	return nil
 }
 
-// Get the set of mount flags that are set on the mount that contains the given
-// path and are locked by CL_UNPRIVILEGED. This is necessary to ensure that
-// bind-mounting "with options" will not fail with user namespaces, due to
-// kernel restrictions that require user namespace mounts to preserve
-// CL_UNPRIVILEGED locked flags.
-func getUnprivilegedMountFlags(path string) ([]string, error) {
-	var statfs unix.Statfs_t
-	if err := unix.Statfs(path, &statfs); err != nil {
-		return nil, err
-	}
-
-	// The set of keys come from https://github.com/torvalds/linux/blob/v4.13/fs/namespace.c#L1034-L1048.
-	unprivilegedFlags := map[uint64]string{
-		unix.MS_RDONLY:     "ro",
-		unix.MS_NODEV:      "nodev",
-		unix.MS_NOEXEC:     "noexec",
-		unix.MS_NOSUID:     "nosuid",
-		unix.MS_NOATIME:    "noatime",
-		unix.MS_RELATIME:   "relatime",
-		unix.MS_NODIRATIME: "nodiratime",
-	}
-
-	var flags []string
-	for mask, flag := range unprivilegedFlags {
-		if uint64(statfs.Flags)&mask == mask {
-			flags = append(flags, flag)
-		}
-	}
-
-	return flags, nil
-}
-
 var (
 	mountPropagationMap = map[string]int{
 		"private":  mount.PRIVATE,
@@ -723,7 +691,7 @@ func withMounts(daemon *Daemon, daemonCfg *configStore, c *container.Container)
 			// when runc sets up the root filesystem, it is already inside a user
 			// namespace, and thus cannot change any flags that are locked.
 			if daemonCfg.RemappedRoot != "" || userns.RunningInUserNS() {
-				unprivOpts, err := getUnprivilegedMountFlags(m.Source)
+				unprivOpts, err := mountopts.UnprivilegedMountFlags(m.Source)
 				if err != nil {
 					return err
 				}
diff --git a/internal/rootless/mountopts/mountopts_linux.go b/internal/rootless/mountopts/mountopts_linux.go
new file mode 100644
index 0000000000000..f4ecf710c8983
--- /dev/null
+++ b/internal/rootless/mountopts/mountopts_linux.go
@@ -0,0 +1,39 @@
+package mountopts
+
+import (
+	"golang.org/x/sys/unix"
+)
+
+// UnprivilegedMountFlags gets the set of mount flags that are set on the mount that contains the given
+// path and are locked by CL_UNPRIVILEGED. This is necessary to ensure that
+// bind-mounting "with options" will not fail with user namespaces, due to
+// kernel restrictions that require user namespace mounts to preserve
+// CL_UNPRIVILEGED locked flags.
+//
+// TODO: Move to github.com/moby/sys/mount, and update BuildKit copy of this code as well (https://github.com/moby/buildkit/blob/v0.13.0/util/rootless/mountopts/mountopts_linux.go#L11-L18)
+func UnprivilegedMountFlags(path string) ([]string, error) {
+	var statfs unix.Statfs_t
+	if err := unix.Statfs(path, &statfs); err != nil {
+		return nil, err
+	}
+
+	// The set of keys come from https://github.com/torvalds/linux/blob/v4.13/fs/namespace.c#L1034-L1048.
+	unprivilegedFlags := map[uint64]string{
+		unix.MS_RDONLY:     "ro",
+		unix.MS_NODEV:      "nodev",
+		unix.MS_NOEXEC:     "noexec",
+		unix.MS_NOSUID:     "nosuid",
+		unix.MS_NOATIME:    "noatime",
+		unix.MS_RELATIME:   "relatime",
+		unix.MS_NODIRATIME: "nodiratime",
+	}
+
+	var flags []string
+	for mask, flag := range unprivilegedFlags {
+		if uint64(statfs.Flags)&mask == mask {
+			flags = append(flags, flag)
+		}
+	}
+
+	return flags, nil
+}
diff --git a/plugin/v2/plugin_linux.go b/plugin/v2/plugin_linux.go
index 82f973ffc9d95..746afde8d5e67 100644
--- a/plugin/v2/plugin_linux.go
+++ b/plugin/v2/plugin_linux.go
@@ -1,3 +1,6 @@
+// FIXME(thaJeztah): remove once we are a module; the go:build directive prevents go from downgrading language version to go1.16:
+//go:build go1.19
+
 package v2 // import "github.com/docker/docker/plugin/v2"
 
 import (
@@ -6,7 +9,10 @@ import (
 	"runtime"
 	"strings"
 
+	"github.com/containerd/containerd/pkg/userns"
 	"github.com/docker/docker/api/types"
+	"github.com/docker/docker/internal/rootless/mountopts"
+	"github.com/docker/docker/internal/sliceutil"
 	"github.com/docker/docker/oci"
 	specs "github.com/opencontainers/runtime-spec/specs-go"
 	"github.com/pkg/errors"
@@ -136,5 +142,35 @@ func (p *Plugin) InitSpec(execRoot string) (*specs.Spec, error) {
 		p.modifyRuntimeSpec(&s)
 	}
 
+	// Rootless mode requires modifying the mount flags
+	// https://github.com/moby/moby/issues/47248#issuecomment-1927776700
+	// https://github.com/moby/moby/pull/47558
+	if userns.RunningInUserNS() {
+		for i := range s.Mounts {
+			m := &s.Mounts[i]
+			for _, o := range m.Options {
+				switch o {
+				case "bind", "rbind":
+					if _, err := os.Lstat(m.Source); err != nil {
+						if errors.Is(err, os.ErrNotExist) {
+							continue
+						}
+						return nil, err
+					}
+					// UnprivilegedMountFlags gets the set of mount flags that are set on the mount that contains the given
+					// path and are locked by CL_UNPRIVILEGED. This is necessary to ensure that
+					// bind-mounting "with options" will not fail with user namespaces, due to
+					// kernel restrictions that require user namespace mounts to preserve
+					// CL_UNPRIVILEGED locked flags.
+					unpriv, err := mountopts.UnprivilegedMountFlags(m.Source)
+					if err != nil {
+						return nil, errors.Wrapf(err, "failed to get unprivileged mount flags for %+v", m)
+					}
+					m.Options = sliceutil.Dedup(append(m.Options, unpriv...))
+				}
+			}
+		}
+	}
+
 	return &s, nil
 }


Test patch: