Repository: moby/moby (language: go)

Issue:
Rootless Docker Service doesn't use native overlayfs (` Native Overlay Diff: false`)
**Rootless Docker Service doesn't use native overlayfs**

Rootless Docker Service doesn't use native overlayfs despite modprobe options:
```options overlay metacopy=off redirect_dir=off```
whereas, the root Docker service works as expected.


**Steps to reproduce the issue:**
1. export DOCKER_HOST=unix://$XDG_RUNTIME_DIR/docker.sock
2. systemctl --user start docker.service
3. docker info | grep Native

**Describe the results you received:**
  Native Overlay Diff: false


**Describe the results you expected:**
  Native Overlay Diff: true


**Output of `docker version`:**

```
Docker version 20.10.16, build aa7e414fdc
```

**Output of `docker info`:**

```
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc., v0.8.2-docker)
  compose: Docker Compose (Docker Inc., 2.5.1)

Server:
 Containers: 6
  Running: 2
  Paused: 0
  Stopped: 4
 Images: 4
 Server Version: 20.10.16
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: false
  userxattr: true
 Logging Driver: json-file
 Cgroup Driver: none
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 212e8b6fa2f44b9c21b2798135fc6fb7c53efc16.m
 runc version: 
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
  rootless
  cgroupns
 Kernel Version: 5.15.40-1-lts
 Operating System: Arch Linux
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 62.47GiB
 Name: phantom
 ID: SMDW:PBEU:YHFR:NAVA:5WGK:ELUL:YRF3:SIHN:JEZB:FV52:LQZD:GUSE
 Docker Root Dir: /home/sighoya/vm/docker
 Debug Mode: true
  File Descriptors: 40
  Goroutines: 59
  System Time: 2022-05-21T16:02:20.585738277+02:00
  EventsListeners: 0
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: true
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: Running in rootless-mode without cgroups. Systemd is required to enable cgroups in rootless-mode.

```

**Additional environment details (AWS, VirtualBox, physical, etc.):**
ArchLinux Linux phantom 5.15.40-1-lts #1 SMP Sun, 15 May 2022 19:07:59 +0000 x86_64 GNU/Linux


See also: https://github.com/moby/moby/issues/41802



Discussion / hints:
We also hit this with `rootless-mode` docker on `Ubuntu 18.04 4.15.0-196-generic` and `overlay2` storage. This causes some renamed/moved files/folders to exists from previous layers (afaik).

Any workaround or fixes? Thanks!
Unfortunately this problem still exists for me after almost 2 years.
I get the following warning:
```
Not using native diff for overlay2, this may cause degraded performance for building images: running in a user namespace
```
It seems like the Native Overlay Diff detection is always forced to be `false` when in user namespace, since there is an early return in the detection function: https://github.com/moby/moby/blob/8b79278316b532d396048bc8c2fa015a85d53a53/daemon/graphdriver/overlay2/check.go#L28
Which then directly results in a warning message and the disabling of native overlay diff here: https://github.com/moby/moby/blob/8b79278316b532d396048bc8c2fa015a85d53a53/daemon/graphdriver/overlay2/overlay.go#L250

I think a possible fix could be: 
- Remove this user namespace check at https://github.com/moby/moby/blob/8b79278316b532d396048bc8c2fa015a85d53a53/daemon/graphdriver/overlay2/check.go#L28
- When rootless, use `user.overlay.opaque` instead of `trusted.overlay.opaque` at https://github.com/moby/moby/blob/8b79278316b532d396048bc8c2fa015a85d53a53/daemon/graphdriver/overlay2/check.go#L63 and at https://github.com/moby/moby/blob/8b79278316b532d396048bc8c2fa015a85d53a53/daemon/graphdriver/overlay2/check.go#L83
- When rootless, use `user.overlay.redirect` instead of `trusted.overlay.redirect` at https://github.com/moby/moby/blob/8b79278316b532d396048bc8c2fa015a85d53a53/daemon/graphdriver/overlay2/check.go#L100
- When rootless, add `userxattr` to the mount options at https://github.com/moby/moby/blob/8b79278316b532d396048bc8c2fa015a85d53a53/daemon/graphdriver/overlay2/check.go#L67

I might attempt a PR but I'm an absolute beginner in Go and also don't know anything about this code or overlayfs, except from what I've learned by looking at the [podman equivalent code](https://github.com/containers/storage/blob/b0c3d9650bfb3b32a312640ae581f9fc7811bb87/drivers/overlay/check.go#L29). I know for a fact that the Native Overlay Diff works when using podman, not sure what docker does differently...

If I may, I will also mention a few people that were involved in earlier related issues/PRs (#41802 and #42235) for feedback on my little proposal above: @AkihiroSuda, @thaJeztah and @tiborvass 
Yes, and in addition the kernel version has to be checked to use userxattr
> @AkihiroSuda wrote: Yes, and in addition the kernel version has to be checked to use userxattr

Okay, so I guess it needs a `if kernel.CheckKernelVersion(5, 11, 0) {...}` somewhere (not sure if 5.11 is the right version). Should I attempt a PR or are you already looking into this?
Hard to me to craft a PR during KubeCon, so submitting a PR would be much appreciated 🙏 
@AkihiroSuda do you know if there's properties we can detect if it can be used instead of kernel version? (curious if we can avoid kernel version for this, in cases where kernel versions "lie" and the distro is carrying patches). If not, that's fine, mostly curious ☺️
> @thaJeztah wrote: (at)AkihiroSuda do you know if there's properties we can detect if it can be used instead of kernel version? (curious if we can avoid kernel version for this, in cases where kernel versions "lie" and the distro is carrying patches). If not, that's fine, mostly curious ☺️

I just noticed that the `usingMetacopy` function already has a check whether userxattr is supported or not. I will try to adapt it for the `doesSupportNativeDiff` function 🤔 
https://github.com/moby/moby/blob/8b79278316b532d396048bc8c2fa015a85d53a53/daemon/graphdriver/overlay2/check.go#L129-L138

Fix patch:
diff --git a/daemon/graphdriver/overlay2/check.go b/daemon/graphdriver/overlay2/check.go
index d9c35dac620e4..0b1a0ce4b2e7f 100644
--- a/daemon/graphdriver/overlay2/check.go
+++ b/daemon/graphdriver/overlay2/check.go
@@ -22,11 +22,21 @@ import (
 // directory or the kernel enable CONFIG_OVERLAY_FS_REDIRECT_DIR.
 // When these exist naive diff should be used.
 //
-// When running in a user namespace, returns errRunningInUserNS
-// immediately.
+// When running in a user namespace it either immediately returns with an error
+// when userxattr is not supported, or performs the native diff checks as usual
+// with some minor user namespace differences, like adding userxattr.
 func doesSupportNativeDiff(d string) error {
+	userxattr := false
 	if userns.RunningInUserNS() {
-		return errors.New("running in a user namespace")
+		needed, err := overlayutils.NeedsUserXAttr(d)
+		if err != nil {
+			return err
+		}
+		if needed {
+			userxattr = true
+		} else {
+			return errors.New("native diff is not supported in user namespace, consider updating to kernel 5.11 or later to fix")
+		}
 	}
 
 	td, err := os.MkdirTemp(d, "opaque-bug-check")
@@ -60,11 +70,16 @@ func doesSupportNativeDiff(d string) error {
 	}
 
 	// Mark l2/d as opaque
-	if err := system.Lsetxattr(filepath.Join(td, "l2", "d"), "trusted.overlay.opaque", []byte("y"), 0); err != nil {
+	if err := system.Lsetxattr(filepath.Join(td, "l2", "d"), overlayutils.GetOverlayXattr("opaque"), []byte("y"), 0); err != nil {
 		return errors.Wrap(err, "failed to set opaque flag on middle layer")
 	}
 
-	opts := fmt.Sprintf("lowerdir=%s:%s,upperdir=%s,workdir=%s", path.Join(td, "l2"), path.Join(td, "l1"), path.Join(td, "l3"), path.Join(td, workDirName))
+	mountFlags := "lowerdir=%s:%s,upperdir=%s,workdir=%s"
+	if userxattr {
+		mountFlags = mountFlags + ",userxattr"
+	}
+
+	opts := fmt.Sprintf(mountFlags, path.Join(td, "l2"), path.Join(td, "l1"), path.Join(td, "l3"), path.Join(td, workDirName))
 	if err := unix.Mount("overlay", filepath.Join(td, mergedDirName), "overlay", 0, opts); err != nil {
 		return errors.Wrap(err, "failed to mount overlay")
 	}
@@ -80,12 +95,12 @@ func doesSupportNativeDiff(d string) error {
 	}
 
 	// Check l3/d does not have opaque flag
-	xattrOpaque, err := system.Lgetxattr(filepath.Join(td, "l3", "d"), "trusted.overlay.opaque")
+	xattrOpaque, err := system.Lgetxattr(filepath.Join(td, "l3", "d"), overlayutils.GetOverlayXattr("opaque"))
 	if err != nil {
 		return errors.Wrap(err, "failed to read opaque flag on upper layer")
 	}
 	if string(xattrOpaque) == "y" {
-		return errors.New("opaque flag erroneously copied up, consider update to kernel 4.8 or later to fix")
+		return errors.New("opaque flag erroneously copied up, consider updating to kernel 4.8 or later to fix")
 	}
 
 	// rename "d1" to "d2"
@@ -97,7 +112,7 @@ func doesSupportNativeDiff(d string) error {
 		return errors.Wrap(err, "failed to rename dir in merged directory")
 	}
 	// get the xattr of "d2"
-	xattrRedirect, err := system.Lgetxattr(filepath.Join(td, "l3", "d2"), "trusted.overlay.redirect")
+	xattrRedirect, err := system.Lgetxattr(filepath.Join(td, "l3", "d2"), overlayutils.GetOverlayXattr("redirect"))
 	if err != nil {
 		return errors.Wrap(err, "failed to read redirect flag on upper layer")
 	}
diff --git a/pkg/archive/archive_linux.go b/pkg/archive/archive_linux.go
index 2c3786cd50c8a..1cecfb65acfc2 100644
--- a/pkg/archive/archive_linux.go
+++ b/pkg/archive/archive_linux.go
@@ -6,6 +6,7 @@ import (
 	"path/filepath"
 	"strings"
 
+	"github.com/containerd/containerd/pkg/userns"
 	"github.com/docker/docker/pkg/system"
 	"github.com/pkg/errors"
 	"golang.org/x/sys/unix"
@@ -35,13 +36,18 @@ func (overlayWhiteoutConverter) ConvertWrite(hdr *tar.Header, path string, fi os
 	}
 
 	if fi.Mode()&os.ModeDir != 0 {
+		opaqueXattrName := "trusted.overlay.opaque"
+		if userns.RunningInUserNS() {
+			opaqueXattrName = "user.overlay.opaque"
+		}
+
 		// convert opaque dirs to AUFS format by writing an empty file with the prefix
-		opaque, err := system.Lgetxattr(path, "trusted.overlay.opaque")
+		opaque, err := system.Lgetxattr(path, opaqueXattrName)
 		if err != nil {
 			return nil, err
 		}
 		if len(opaque) == 1 && opaque[0] == 'y' {
-			delete(hdr.PAXRecords, paxSchilyXattr+"trusted.overlay.opaque")
+			delete(hdr.PAXRecords, paxSchilyXattr+opaqueXattrName)
 
 			// create a header for the whiteout file
 			// it should inherit some properties from the parent, but be a regular file
@@ -69,9 +75,14 @@ func (c overlayWhiteoutConverter) ConvertRead(hdr *tar.Header, path string) (boo
 
 	// if a directory is marked as opaque by the AUFS special file, we need to translate that to overlay
 	if base == WhiteoutOpaqueDir {
-		err := unix.Setxattr(dir, "trusted.overlay.opaque", []byte{'y'}, 0)
+		opaqueXattrName := "trusted.overlay.opaque"
+		if userns.RunningInUserNS() {
+			opaqueXattrName = "user.overlay.opaque"
+		}
+
+		err := unix.Setxattr(dir, opaqueXattrName, []byte{'y'}, 0)
 		if err != nil {
-			return false, errors.Wrapf(err, "setxattr(%q, trusted.overlay.opaque=y)", dir)
+			return false, errors.Wrapf(err, "setxattr(%q, %s=y)", dir, opaqueXattrName)
 		}
 		// don't write the file itself
 		return false, err


Test patch: