Repository: systemd/mkosi

Issue:
Unable to mount build-overlay with incremental builds
I am having issues getting incremental builds to run as the build overlay fails to mount. This is on Arch Linux with a btrfs filesystem, using mkosi commit 7030d76 (the latest as of the time of writing).

Using the following mkosi configuration:

```ini
[Distribution]
Distribution=debian
Release=bookworm

[Output]
CacheDirectory=cache
ImageId=test

[Content]
BuildPackages=
  make
BuildScript=build.sh
```

And a simple build script:

```bash
#!/usr/bin/bash
echo "building"
make --version
```

On the first run of `mkosi -i -f` everything runs successfully and I get an output image. Of potential interest is the dmesg output:

```console
[ 1907.326051] overlayfs: fs on '<path>/.mkosi-tmp7x3kcw9q/root' does not support file handles, falling back to index=off,nfs_export=off.
[ 1907.326054] overlayfs: fs on '<path>/.mkosi-tmp7x3kcw9q/root' does not support file handles, falling back to xino=off.
```

However, on a second run of `mkosi -i -f` I get the following error:

```console
‣ Removing output files…
‣ Building default image
Create subvolume '<path>/.mkosi-tmp38ur8lqj/root'
‣  Copying cached trees
‣  Running build script…
mount: <path>/.mkosi-tmp38ur8lqj/root: mount(2) system call failed: Stale file handle.
       dmesg(1) may have more information after failed mount system call.
‣ "mount --no-mtab overlay <path>/.mkosi-tmp38ur8lqj/root --types overlay --options lowerdir=<path>/.mkosi-tmp38ur8lqj/root,upperdir=<path>/.mkosi-tmp38ur8lqj/build-overlay,workdir=<path>/.mkosi-tmp38ur8lqj/build-overlay-workdirpawin893,userxattr" returned non-zero exit code 32.
umount: <path>/.mkosi-tmp38ur8lqj/root: not mounted.
‣ "umount --no-mtab <path>/.mkosi-tmp38ur8lqj/root" returned non-zero exit code 32.
‣   (Cleaning up overlayfs)
‣    (Removing overlay whiteout files…)
```

The  corresponding dmesg output is:
```console
[ 2126.464825] overlayfs: failed to verify origin (.mkosi-tmp38ur8lqj/root, ino=256, err=-116)
[ 2126.464829] overlayfs: failed to verify upper root origin
```

From [the overlayfs docs](https://www.kernel.org/doc/html/latest/filesystems/overlayfs.html#sharing-and-copying-layers)
> Mounting an overlay using an upper layer path, where the upper layer path was previously used by another mounted overlay in combination with a different lower layer path, is allowed, unless the "inodes index" feature or "metadata only copy up" feature is enabled.

It seems that Arch [has these features enabled by default](https://gitlab.archlinux.org/archlinux/packaging/packages/linux/-/blob/754a4cb9f87cb074eedbb6bebf1f926543a5b64f/config#L10193), and I am guessing that the use of a new temporary directory for the lower /root on each run causes a mismatch with the one used when the build overlay was created.

If I modify line 84 of `mkosi/mounts.py`
https://github.com/systemd/mkosi/blob/7030d76ae957d35d5b49e72c54c1d7f771cc4cac/mkosi/mounts.py#L84

to

```python
        options = [f"lowerdir={lower}" for lower in lowerdirs] + [f"upperdir={upperdir}", f"workdir={workdir}", "index=off"]
```

then the second incremental run works (it seems not to be neccessary to clean the cached images with `-ff`).  The dmesg warning from the first run about falling back to index=off seems to not apply to the build overlay.

Note that it wasn't neccessary (in my case, at least) to add `"metacopy=off"` to the options, although I guess this might be safer in general.

I am not sure if this is the proper fix; if so, I am happy to submit a pull request (or for a maintainer to just make the trivial change directly).


Discussion:
PR would be appreciated, but please disable both "index" and "metacopy" and link to https://www.kernel.org/doc/html/latest/filesystems/overlayfs.html#sharing-and-copying-layers as the explanation in a comment to explain why we disable these features.

Resolving patch:
diff --git a/mkosi/mounts.py b/mkosi/mounts.py
--- a/mkosi/mounts.py
+++ b/mkosi/mounts.py
@@ -81,7 +81,19 @@ def mount_overlay(
     read_only: bool = True,
 ) -> Iterator[Path]:
     with tempfile.TemporaryDirectory(dir=upperdir.parent, prefix=f"{upperdir.name}-workdir") as workdir:
-        options = [f"lowerdir={lower}" for lower in lowerdirs] + [f"upperdir={upperdir}", f"workdir={workdir}"]
+        options = [f"lowerdir={lower}" for lower in lowerdirs] + [
+            f"upperdir={upperdir}",
+            f"workdir={workdir}",
+            # Disable the inodes index and metacopy (only copy metadata upwards if possible)
+            # options. If these are enabled (e.g., if the kernel enables them by default),
+            # the mount will fail if the upper directory has been earlier used with a different
+            # lower directory, such as with a build overlay that was generated on top of a
+            # different temporary root.
+            # See https://www.kernel.org/doc/html/latest/filesystems/overlayfs.html#sharing-and-copying-layers
+            # and https://github.com/systemd/mkosi/issues/1841.
+            "index=off",
+            "metacopy=off"
+        ]
 
         # userxattr is only supported on overlayfs since kernel 5.11
         if GenericVersion(platform.release()) >= GenericVersion("5.11"):


Test patch: