Repository: systemd/mkosi

Issue:
`--force` is incompatible with `ToolsTree=default` and building multiple images
When `ToolsTree=default` is provided, an extra build step is added to each image.

This step checks for if it needs to be rebuilt in https://github.com/systemd/mkosi/blob/4a4b8bd7853ba25b1f39506d5c0760f97f5720f0/mkosi/__init__.py#L3440

This works fine the first time you build the images, because the extra step for each image checks whether the tools tree needs to be built.

Unfortunately, when `--force` is passed because you need to rebuild an image, it fails, because the tools tree rebuild step has been added to every image, and when `--force` is passed it bypasses the check for whether the image already exists, and the tools tree build step isn't idempotent, so it fails when it tries to rename the built tools tree into the output directory.

This produces an error that looks like:

```
richardmaw@skullthought:~/workspace/sy002/systemd-wt$ meson test -C build/ --suite integration-tests
ninja: no work to do.
ninja: Entering directory `/home/richardmaw/workspace/sy002/systemd-wt/build'
[0/1] Generating system_mkosi with a custom command
+ output=system
+ mkosi -C .. --image=system --without-tests -f build
‣ Removing output files…
‣ Removing output files…
‣ Removing output files…
‣ Removing output files…
‣ Removing output files…
‣ Removing output files…
‣ Removing output files…
‣ Removing output files…
‣ Removing output files…
‣ Removing output files…
‣ Removing output files…
‣ Removing output files…
‣ Removing output files…
‣ Removing output files…
‣ Removing output files…
‣ Removing output files…
‣ Building fedora-tools image
‣  Copying cached trees
‣  Generating system users
‣  Applying presets…
Removed "/home/richardmaw/.cache/mkosi-workspace951fiuri/root/etc/systemd/system/systemd-homed.service.wants/systemd-homed-activate.service".
Created symlink /home/richardmaw/.cache/mkosi-workspace951fiuri/root/etc/systemd/system/timers.target.wants/fstrim.timer → /lib/systemd/system/fstrim.timer.
Created symlink /home/richardmaw/.cache/mkosi-workspace951fiuri/root/etc/systemd/system/multi-user.target.wants/machines.target → /lib/systemd/system/machines.target.
Created symlink /home/richardmaw/.cache/mkosi-workspace951fiuri/root/etc/systemd/system/systemd-homed.service.wants/systemd-homed-activate.service → /lib/systemd/system/systemd-homed-activate.service.
‣  Generating hardware database
‣  /home/richardmaw/workspace/sy002/systemd-wt/mkosi.output/fedora-tools size is 855.4M.
‣ Building base image
‣  Copying cached trees
‣  Running build script /home/richardmaw/workspace/sy002/systemd-wt/mkosi.images/base/mkosi.build.chroot…
+ ninja -C /work/build
ninja: Entering directory `/work/build'
[1/688] Generating version.h with a custom command
+ meson install -C /work/build --quiet --no-rebuild --only-changed
‣  Copying in extra file trees…
‣  Generating system users
‣  Applying presets…
Created symlink /home/richardmaw/.cache/mkosi-workspacek8ue3hey/root/etc/systemd/system/timers.target.wants/fstrim.timer → /usr/lib/systemd/system/fstrim.timer.
‣  Generating hardware database
No hwdb files found, skipping.
‣  /home/richardmaw/workspace/sy002/systemd-wt/mkosi.output/base size is 142.6M.
‣ Building fedora-tools image
‣  Copying cached trees
‣  Generating system users
‣  Applying presets…
Removed "/home/richardmaw/.cache/mkosi-workspacepoa5wnqc/root/etc/systemd/system/systemd-homed.service.wants/systemd-homed-activate.service".
Created symlink /home/richardmaw/.cache/mkosi-workspacepoa5wnqc/root/etc/systemd/system/timers.target.wants/fstrim.timer → /lib/systemd/system/fstrim.timer.
Created symlink /home/richardmaw/.cache/mkosi-workspacepoa5wnqc/root/etc/systemd/system/multi-user.target.wants/machines.target → /lib/systemd/system/machines.target.
Created symlink /home/richardmaw/.cache/mkosi-workspacepoa5wnqc/root/etc/systemd/system/systemd-homed.service.wants/systemd-homed-activate.service → /lib/systemd/system/systemd-homed-activate.service.
‣  Generating hardware database
Traceback (most recent call last):
  File "/home/richardmaw/workspace/sy002/mkosi/mkosi/run.py", line 162, in uncaught_exception_handler
    yield
  File "/home/richardmaw/workspace/sy002/mkosi/mkosi/run.py", line 203, in fork_and_wait
    target()
  File "/home/richardmaw/workspace/sy002/mkosi/mkosi/__init__.py", line 3469, in <lambda>
    fork_and_wait(lambda: run_build(args, tools)) # pyright: ignore
                          ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/richardmaw/workspace/sy002/mkosi/mkosi/__init__.py", line 3384, in run_build
    build_image(args, config)
  File "/home/richardmaw/workspace/sy002/mkosi/mkosi/__init__.py", line 2890, in build_image
    finalize_staging(context)
  File "/home/richardmaw/workspace/sy002/mkosi/mkosi/__init__.py", line 2730, in finalize_staging
    move_tree(
  File "/home/richardmaw/workspace/sy002/mkosi/mkosi/tree.py", line 153, in move_tree
    raise e
  File "/home/richardmaw/workspace/sy002/mkosi/mkosi/tree.py", line 150, in move_tree
    src.rename(dst)
  File "/usr/lib/python3.11/pathlib.py", line 1176, in rename
    os.rename(self, target)
FileExistsError: [Errno 17] File exists: '/home/richardmaw/.cache/mkosi-workspacepoa5wnqc/staging/fedora-tools' -> '/home/richardmaw/workspace/sy002/systemd-wt/mkosi.output/fedora-tools'
FAILED: system 
```


Discussion:

Resolving patch:
diff --git a/mkosi/__init__.py b/mkosi/__init__.py
--- a/mkosi/__init__.py
+++ b/mkosi/__init__.py
@@ -3195,20 +3195,6 @@ def expand_specifier(s: str) -> str:
     return s.replace("%u", INVOKING_USER.name())
 
 
-def needs_build(args: Args, config: Config) -> bool:
-    return (
-        args.verb.needs_build() and
-        (
-            args.force > 0 or
-            not (config.output_dir_or_cwd() / config.output_with_compression).exists() or
-            # When the output is a directory, its name is the same as the symlink we create that points to the actual
-            # output when not building a directory. So if the full output path exists, we have to check that it's not
-            # a symlink as well.
-            (config.output_dir_or_cwd() / config.output_with_compression).is_symlink()
-        )
-    )
-
-
 @contextlib.contextmanager
 def prepend_to_environ_path(config: Config) -> Iterator[None]:
     if config.tools_tree or not config.extra_search_paths:
@@ -3285,7 +3271,21 @@ def check_workspace_directory(config: Config) -> None:
                 hint="Use WorkspaceDirectory= to configure a different workspace directory")
 
 
+def needs_clean(args: Args, config: Config) -> bool:
+    return (
+        args.force > 0 or
+        not (config.output_dir_or_cwd() / config.output_with_compression).exists() or
+        # When the output is a directory, its name is the same as the symlink we create that points to the actual
+        # output when not building a directory. So if the full output path exists, we have to check that it's not
+        # a symlink as well.
+        (config.output_dir_or_cwd() / config.output_with_compression).is_symlink()
+    )
+
+
 def run_clean(args: Args, config: Config) -> None:
+    if not needs_clean(args, config):
+        return
+
     become_root()
 
     # We remove any cached images if either the user used --force twice, or he/she called "clean" with it
@@ -3299,32 +3299,33 @@ def run_clean(args: Args, config: Config) -> None:
         remove_build_cache = args.force > 1
         remove_package_cache = args.force > 2
 
-    with complete_step("Removing output files…"):
-        if config.output_dir_or_cwd().exists():
-            rmtree(*(p for p in config.output_dir_or_cwd().iterdir() if p.name.startswith(config.output)))
+    if (outputs := list(config.output_dir_or_cwd().glob(f"{config.output}*"))):
+        with complete_step(f"Removing output files of {config.name()} image…"):
+            rmtree(*outputs)
 
     if remove_build_cache:
         if config.cache_dir:
-            with complete_step("Removing cache entries…"):
+            with complete_step(f"Removing cache entries of {config.name()} image…"):
                 rmtree(*(p for p in cache_tree_paths(config) if p.exists()))
 
         if config.build_dir and config.build_dir.exists() and any(config.build_dir.iterdir()):
-            with complete_step("Clearing out build directory…"):
+            with complete_step(f"Clearing out build directory of {config.name()} image…"):
                 rmtree(*config.build_dir.iterdir())
 
-    if remove_package_cache:
-        if config.cache_dir and config.cache_dir.exists() and any(config.cache_dir.iterdir()):
-            with complete_step("Clearing out package cache…"):
-                rmtree(
-                    *(
-                        config.cache_dir / p / d
-                        for p in ("cache", "lib")
-                        for d in ("apt", "dnf", "libdnf5", "pacman", "zypp")
-                    ),
-                )
+    if remove_package_cache and config.cache_dir and config.cache_dir.exists() and any(config.cache_dir.iterdir()):
+        with complete_step(f"Clearing out package cache of {config.name()} image…"):
+            rmtree(
+                *(
+                    config.cache_dir / p / d
+                    for p in ("cache", "lib")
+                    for d in ("apt", "dnf", "libdnf5", "pacman", "zypp")
+                ),
+            )
 
 
 def run_build(args: Args, config: Config) -> None:
+    check_inputs(config)
+
     if (uid := os.getuid()) != 0:
         become_root()
     unshare(CLONE_NEWNS)
@@ -3411,7 +3412,7 @@ def run_verb(args: Args, images: Sequence[Config]) -> None:
     # image build could end up deleting the output generated by an earlier image build.
 
     for config in images:
-        if not needs_build(args, config) and args.verb != Verb.clean:
+        if not args.verb.needs_build() and args.verb != Verb.clean:
             continue
 
         if config.tools_tree and config.tools_tree.name == "default":
@@ -3426,6 +3427,9 @@ def run_verb(args: Args, images: Sequence[Config]) -> None:
     build = False
 
     for i, config in enumerate(images):
+        if not args.verb.needs_build():
+            continue
+
         with (
             finalize_default_tools(args, config)
             if config.tools_tree and config.tools_tree.name == "default"
@@ -3437,14 +3441,12 @@ def run_verb(args: Args, images: Sequence[Config]) -> None:
                 tools_tree=tools.output_dir_or_cwd() / tools.output if tools else config.tools_tree,
             )
 
-            if tools and needs_build(args, tools):
-                check_inputs(tools)
+            if tools and not (tools.output_dir_or_cwd() / tools.output_with_compression).exists():
                 fork_and_wait(lambda: run_build(args, tools)) # pyright: ignore
 
-        if not needs_build(args, config):
+        if (config.output_dir_or_cwd() / config.output_with_compression).exists():
             continue
 
-        check_inputs(config)
         fork_and_wait(lambda: run_build(args, config))
 
         build = True


Test patch: