Repository: moby/moby (language: go)

Issue:
[BUG] Rootless docker - plugin discovery uses wrong path
**Description**

I'm running the docker engine in rootless mode in parallel with another rootfull engine.
When calling `docker volume ls`, the rootless daemon returns an error message that it failed to read the rootfull daemon's plugins folder.

TL;DR: It seems the rootless daemon's  plugin discovery is using the wrong path.

**Steps to reproduce the issue:**
1. Install rootfull docker ([see here](https://docs.docker.com/engine/install/ubuntu/))
2. Install rootless docker ([see here](https://docs.docker.com/engine/security/rootless/))
3. Rootless cli: `docker volume create test`
2. Rootless cli: `docker volume ls`
3. See daemon response: 
   ```
   Error response from daemon: list: error listing plugins: legacy plugin: error reading dir entries: open /etc/docker/plugins: permission denied
   ```

**Describe the results you received:**
Error response from daemon, that it failed to read the protected rootfull daemon's plugin folder at `/etc/docker/plugins`.

**Describe the results you expected:**
List of volumes (that contains one item named "test").

I would expect that the plugins in the rootless docker directory ("Docker Root Dir") at `/home/rootless/.local/share/docker/plugins` are used.

**Additional information you deem important (e.g. issue happens only occasionally):**

The hard-coded paths are used in rootless mode. 
Shouldn't the plugin path be selected by the mode (rootfull or rootless)? 
Or shouldn't the plugin path depend on the docker root directory?

Location in code:
https://github.com/moby/moby/blob/94e00d09da6a12b32161889e9a15c6c2da7bc0d2/pkg/plugins/discovery_unix.go#L6
https://github.com/moby/moby/blob/94e00d09da6a12b32161889e9a15c6c2da7bc0d2/pkg/plugins/discovery.go#L33


**Output of `docker version`:**

```
$ docker version
Client:
 Version:           20.10.12
 API version:       1.41
 Go version:        go1.16.12
 Git commit:        e91ed57
 Built:             Mon Dec 13 11:40:57 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.12
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.12
  Git commit:       459d0df
  Built:            Mon Dec 13 11:46:12 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.4.12
  GitCommit:        7b11cfaabd73bb80907dd23182b9347b4245eb5d
 runc:
  Version:          1.0.2
  GitCommit:        v1.0.2-0-g52b36a2d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
```

**Output of `docker info`:**

```
$ docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Docker Buildx (Docker Inc., v0.7.1-docker)
  compose: Docker Compose (Docker Inc., v2.0.1)

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 20.10.12
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: false
  userxattr: true
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7b11cfaabd73bb80907dd23182b9347b4245eb5d
 runc version: v1.0.2-0-g52b36a2d
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
  rootless
  cgroupns
 Kernel Version: 5.13.0-22-generic
 Operating System: Ubuntu 21.10
 OSType: linux
 Architecture: x86_64
 CPUs: 3
 Total Memory: 3.74GiB
 Name: ubuntu-8gb-nbg1-dc3-1
 ID: ****
 Docker Root Dir: /home/rootless/.local/share/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support
WARNING: No cpu shares support
WARNING: No cpuset support
WARNING: No io.weight support
WARNING: No io.weight (per device) support
WARNING: No io.max (rbps) support
WARNING: No io.max (wbps) support
WARNING: No io.max (riops) support
WARNING: No io.max (wiops) support

```

**Additional environment details (AWS, VirtualBox, physical, etc.):**
Hosted Ubuntu on a VM machine.


Discussion / hints:
Sidenote: The other volume sub-commands are working just fine. I can create, delete and inspect volumes.
Quick-fix: `sudo chmod +rx /etc/docker`
I'm seeing similar error when creating a rootless image with octoprint (other images work fine):

```
unprivileged-docker@host:~/octoprint$ docker-compose up
ERROR: get octoprint_octoprint: error listing plugins: legacy plugin: error reading dir entries: open /etc/docker/plugins: permission denied
unprivileged-docker@host:~/octoprint$
```

Docker compose yaml:

```
version: '2.4'

services:
  octoprint:
    image: octoprint/octoprint
    restart: unless-stopped
    ports:
      - 10.0.7.204:80:80
    volumes:
     - octoprint:/octoprint
```

my workaround:

```
1) remove /etc/docker if it already exists (my had some stuff from root (not rootless) installation
2)
# mkdir -p /etc/docker/plugins
# chown unprivileged-docker:unprivileged-docker -R /etc/docker
# chmod 600 -R /etc/docker
``` 
Any new on this?
:+1:  Is this fixed or is it expected to get fixed. Changing folder permissions seams hacky. 

Fix patch:
diff --git a/cmd/dockerd/config_unix.go b/cmd/dockerd/config_unix.go
index 89148d81c4ec8..e76c9624220c4 100644
--- a/cmd/dockerd/config_unix.go
+++ b/cmd/dockerd/config_unix.go
@@ -10,8 +10,8 @@ import (
 	"github.com/docker/docker/daemon/config"
 	"github.com/docker/docker/opts"
 	"github.com/docker/docker/pkg/homedir"
+	"github.com/docker/docker/pkg/rootless"
 	"github.com/docker/docker/registry"
-	"github.com/docker/docker/rootless"
 	"github.com/spf13/pflag"
 )
 
diff --git a/cmd/dockerd/daemon.go b/cmd/dockerd/daemon.go
index 4ced1361ac760..8389378174cbc 100644
--- a/cmd/dockerd/daemon.go
+++ b/cmd/dockerd/daemon.go
@@ -46,10 +46,10 @@ import (
 	"github.com/docker/docker/pkg/jsonmessage"
 	"github.com/docker/docker/pkg/pidfile"
 	"github.com/docker/docker/pkg/plugingetter"
+	"github.com/docker/docker/pkg/rootless"
 	"github.com/docker/docker/pkg/sysinfo"
 	"github.com/docker/docker/pkg/system"
 	"github.com/docker/docker/plugin"
-	"github.com/docker/docker/rootless"
 	"github.com/docker/docker/runconfig"
 	"github.com/docker/go-connections/tlsconfig"
 	"github.com/moby/buildkit/session"
diff --git a/cmd/dockerd/docker.go b/cmd/dockerd/docker.go
index 41123fa9499aa..3c594a0878046 100644
--- a/cmd/dockerd/docker.go
+++ b/cmd/dockerd/docker.go
@@ -8,7 +8,7 @@ import (
 	"github.com/docker/docker/dockerversion"
 	"github.com/docker/docker/pkg/jsonmessage"
 	"github.com/docker/docker/pkg/reexec"
-	"github.com/docker/docker/rootless"
+	"github.com/docker/docker/pkg/rootless"
 	"github.com/moby/buildkit/util/apicaps"
 	"github.com/moby/term"
 	"github.com/sirupsen/logrus"
diff --git a/daemon/config/config_linux.go b/daemon/config/config_linux.go
index 70d1efda8d11f..13b06752bcb3d 100644
--- a/daemon/config/config_linux.go
+++ b/daemon/config/config_linux.go
@@ -11,7 +11,7 @@ import (
 	"github.com/docker/docker/api/types/container"
 	"github.com/docker/docker/opts"
 	"github.com/docker/docker/pkg/homedir"
-	"github.com/docker/docker/rootless"
+	"github.com/docker/docker/pkg/rootless"
 	units "github.com/docker/go-units"
 	"github.com/pkg/errors"
 )
diff --git a/daemon/info_unix.go b/daemon/info_unix.go
index b9affb62a9ce5..12e9180366b7a 100644
--- a/daemon/info_unix.go
+++ b/daemon/info_unix.go
@@ -12,8 +12,8 @@ import (
 
 	"github.com/docker/docker/api/types"
 	containertypes "github.com/docker/docker/api/types/container"
+	"github.com/docker/docker/pkg/rootless"
 	"github.com/docker/docker/pkg/sysinfo"
-	"github.com/docker/docker/rootless"
 	"github.com/pkg/errors"
 	"github.com/sirupsen/logrus"
 )
diff --git a/daemon/oci_linux.go b/daemon/oci_linux.go
index 2806abfaaebed..59c07941ef8c8 100644
--- a/daemon/oci_linux.go
+++ b/daemon/oci_linux.go
@@ -22,8 +22,8 @@ import (
 	"github.com/docker/docker/oci"
 	"github.com/docker/docker/oci/caps"
 	"github.com/docker/docker/pkg/idtools"
+	"github.com/docker/docker/pkg/rootless/specconv"
 	"github.com/docker/docker/pkg/stringid"
-	"github.com/docker/docker/rootless/specconv"
 	volumemounts "github.com/docker/docker/volume/mounts"
 	"github.com/moby/sys/mount"
 	"github.com/moby/sys/mountinfo"
diff --git a/libnetwork/iptables/iptables.go b/libnetwork/iptables/iptables.go
index 9423d6247e4ca..86e0c364b2dc2 100644
--- a/libnetwork/iptables/iptables.go
+++ b/libnetwork/iptables/iptables.go
@@ -13,7 +13,7 @@ import (
 	"sync"
 	"time"
 
-	"github.com/docker/docker/rootless"
+	"github.com/docker/docker/pkg/rootless"
 	"github.com/sirupsen/logrus"
 )
 
diff --git a/pkg/homedir/homedir_linux.go b/pkg/homedir/homedir_linux.go
index 5e6310fdcd6bb..7df039b4cf78f 100644
--- a/pkg/homedir/homedir_linux.go
+++ b/pkg/homedir/homedir_linux.go
@@ -91,3 +91,12 @@ func GetConfigHome() (string, error) {
 	}
 	return filepath.Join(home, ".config"), nil
 }
+
+// GetLibHome returns $HOME/.local/lib
+func GetLibHome() (string, error) {
+	home := os.Getenv("HOME")
+	if home == "" {
+		return "", errors.New("could not get HOME")
+	}
+	return filepath.Join(home, ".local/lib"), nil
+}
diff --git a/pkg/plugins/discovery.go b/pkg/plugins/discovery.go
index 9d972b3c219fb..61a58980cbecb 100644
--- a/pkg/plugins/discovery.go
+++ b/pkg/plugins/discovery.go
@@ -49,7 +49,7 @@ func Scan() ([]string, error) {
 		}
 	}
 
-	for _, p := range specsPaths {
+	for _, p := range SpecsPaths() {
 		dirEntries, err := os.ReadDir(p)
 		if err != nil && !os.IsNotExist(err) {
 			return nil, errors.Wrap(err, "error reading dir entries")
@@ -93,7 +93,7 @@ func (l *localRegistry) Plugin(name string) (*Plugin, error) {
 	}
 
 	var txtspecpaths []string
-	for _, p := range specsPaths {
+	for _, p := range SpecsPaths() {
 		txtspecpaths = append(txtspecpaths, pluginPaths(p, name, ".spec")...)
 		txtspecpaths = append(txtspecpaths, pluginPaths(p, name, ".json")...)
 	}
diff --git a/pkg/plugins/discovery_unix.go b/pkg/plugins/discovery_unix.go
index d645da8ce4219..46d57d31e7467 100644
--- a/pkg/plugins/discovery_unix.go
+++ b/pkg/plugins/discovery_unix.go
@@ -2,5 +2,45 @@
 // +build !windows
 
 package plugins // import "github.com/docker/docker/pkg/plugins"
+import (
+	"path/filepath"
 
-var specsPaths = []string{"/etc/docker/plugins", "/usr/lib/docker/plugins"}
+	"github.com/docker/docker/pkg/homedir"
+	"github.com/docker/docker/pkg/rootless"
+)
+
+const globalConfigPluginsPath = "/etc/docker/plugins"
+const globalLibPluginsPath = "/usr/lib/docker/plugins"
+
+var globalSpecsPaths = []string{globalConfigPluginsPath, globalLibPluginsPath}
+
+func rootlessConfigPluginsPath() string {
+	configHome, err := homedir.GetConfigHome()
+	if err == nil {
+		return filepath.Join(configHome, "docker/plugins")
+	}
+
+	return globalConfigPluginsPath
+}
+
+func rootlessLibPluginsPath() string {
+	libHome, err := homedir.GetLibHome()
+	if err == nil {
+		return filepath.Join(libHome, "docker/plugins")
+	}
+
+	return globalLibPluginsPath
+}
+
+// SpecsPaths returns
+// { "%programdata%\docker\plugins" } on Windows,
+// { "/etc/docker/plugins", "/usr/lib/docker/plugins" } on Unix in non-rootless mode,
+// { "$XDG_CONFIG_HOME/docker/plugins", "$HOME/.local/lib/docker/plugins" } on Unix in rootless mode
+// with fallback to the corresponding path in non-rootless mode if $XDG_CONFIG_HOME or $HOME is not set.
+func SpecsPaths() []string {
+	if rootless.RunningWithRootlessKit() {
+		return []string{rootlessConfigPluginsPath(), rootlessLibPluginsPath()}
+	}
+
+	return globalSpecsPaths
+}
diff --git a/pkg/plugins/discovery_windows.go b/pkg/plugins/discovery_windows.go
index f0af3477f42f7..6e978c94fae39 100644
--- a/pkg/plugins/discovery_windows.go
+++ b/pkg/plugins/discovery_windows.go
@@ -5,4 +5,13 @@ import (
 	"path/filepath"
 )
 
-var specsPaths = []string{filepath.Join(os.Getenv("programdata"), "docker", "plugins")}
+var globalSpecsPaths = []string{filepath.Join(os.Getenv("programdata"), "docker", "plugins")}
+
+// SpecsPaths returns
+// { "%programdata%\docker\plugins" } on Windows,
+// { "/etc/docker/plugins", "/usr/lib/docker/plugins" } on Unix in non-rootless mode,
+// { "$XDG_CONFIG_HOME/docker/plugins", "$HOME/.local/lib/docker/plugins" } on Unix in rootless mode
+// with fallback to the corresponding path in non-rootless mode if $XDG_CONFIG_HOME or $HOME is not set.
+func SpecsPaths() []string {
+	return globalSpecsPaths
+}
diff --git a/rootless/rootless.go b/pkg/rootless/rootless.go
similarity index 92%
rename from rootless/rootless.go
rename to pkg/rootless/rootless.go
index 4cda7cf6fd216..f407f0fc42d12 100644
--- a/rootless/rootless.go
+++ b/pkg/rootless/rootless.go
@@ -1,4 +1,4 @@
-package rootless // import "github.com/docker/docker/rootless"
+package rootless // import "github.com/docker/docker/pkg/rootless"
 
 import (
 	"os"
diff --git a/rootless/specconv/specconv_linux.go b/pkg/rootless/specconv/specconv_linux.go
similarity index 97%
rename from rootless/specconv/specconv_linux.go
rename to pkg/rootless/specconv/specconv_linux.go
index 4e542818c24d4..16200e22e93fb 100644
--- a/rootless/specconv/specconv_linux.go
+++ b/pkg/rootless/specconv/specconv_linux.go
@@ -1,4 +1,4 @@
-package specconv // import "github.com/docker/docker/rootless/specconv"
+package specconv // import "github.com/docker/docker/pkg/rootless/specconv"
 
 import (
 	"os"


Test patch:
diff --git a/pkg/plugins/discovery_test.go b/pkg/plugins/discovery_test.go
index f162fe662a1d2..544b1f8cb57ca 100644
--- a/pkg/plugins/discovery_test.go
+++ b/pkg/plugins/discovery_test.go
@@ -13,7 +13,7 @@ func Setup(t *testing.T) (string, func()) {
 	}
 	backup := socketsPath
 	socketsPath = tmpdir
-	specsPaths = []string{tmpdir}
+	globalSpecsPaths = []string{tmpdir}
 
 	return tmpdir, func() {
 		socketsPath = backup