internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: systemd/mkosi

Issue:
Secureboot settings in images not meant to be built shouldn't matter
I get the following error, when I try to build an image without secureboot keys (without secureboot enabled), if another config in `mkosi.images` needs secureboot`:
```
‣ UEFI SecureBoot enabled, but couldn't find the certificate and private key.
‣ (Consider generating them with 'mkosi genkey'.)
```

To reproduce:
```
$ mkdir -p mkosi.images/{test1,test2}
$ echo -e '[Validation]\nSecureBoot=no' >mkosi.images/test1/mkosi.conf
$ echo -e '[Validation]\nSecureBoot=yes' >mkosi.images/test2/mkosi.conf
$ mkosi --image=test1 summary
‣ UEFI SecureBoot enabled, but couldn't find the certificate and private key.
‣ (Consider generating them with 'mkosi genkey'.)
```

This is not dramatic, but is a bit unexpected, it would be nice, if the image setting would ignore missing files in other image configs that the wanted image doesn't depend on.


Discussion:

Resolving patch:
diff --git a/mkosi/config.py b/mkosi/config.py
--- a/mkosi/config.py
+++ b/mkosi/config.py
@@ -2784,7 +2784,7 @@ def create_argument_parser(action: type[argparse.Action]) -> argparse.ArgumentPa
     return parser
 
 
-def resolve_deps(images: Sequence[Config], include: Sequence[str]) -> list[Config]:
+def resolve_deps(images: Sequence[argparse.Namespace], include: Sequence[str]) -> list[argparse.Namespace]:
     graph = {config.image: config.dependencies for config in images}
 
     if include:
@@ -3197,8 +3197,8 @@ def finalize_defaults(namespace: argparse.Namespace, defaults: argparse.Namespac
     if not images:
         die("No images defined in mkosi.images/")
 
-    images = [load_config(args, ns) for ns in images]
     images = resolve_deps(images, include)
+    images = [load_config(args, ns) for ns in images]
 
     return args, tuple(images)
 


Test patch:
Repository: systemd/mkosi

Issue:
Sysext example doesn't build with `Format=tar`
When attempting to build the sysext example, but changing to `Format=tar` (and `CompressOutput=false` and adjusting `BaseTrees` of `btrfs`) mkosi exits with the following error:

```
‣  Creating tar archive /var/tmp/mkosi-workspaceo_jtq2be/staging/base.tar…
‣  /home/nekko/Overlay/mkosi.output/base.tar size is 136.3M, consumes 136.3M.
‣ Building btrfs image
‣  Mounting base trees…
‣   Extracting tar archive /home/nekko/Overlay/mkosi.output/base.tar…
bwrap: Can't find source path /var/tmp/mkosi-workspacei7ho9ng6/bases/base.tar-e64a5ae4b49c47d4b299acd1ba3f3380: No such file or directory
bwrap: Can't find source path /var/tmp/mkosi-workspacei7ho9ng6/bases/base.tar-e64a5ae4b49c47d4b299acd1ba3f3380: No such file or directory
‣ "tar --extract --file - --directory /var/tmp/mkosi-workspacei7ho9ng6/bases/base.tar-e64a5ae4b49c47d4b299acd1ba3f3380 --keep-directory-symlink --no-overwrite-dir --same-permissions --numeric-owner --same-order --acls --selinux --xattrs --force-local --exclude './dev/*' --exclude './proc/*' --exclude './sys/*' --exclude './tmp/*' --exclude './run/*' --exclude './var/tmp/*'" returned non-zero exit code 1.
```

This does not happen when removing `Overlay=true`


Discussion:

Resolving patch:
diff --git a/mkosi/__init__.py b/mkosi/__init__.py
--- a/mkosi/__init__.py
+++ b/mkosi/__init__.py
@@ -108,7 +108,7 @@ def mount_base_trees(context: Context) -> Iterator[None]:
                 extract_tar(
                     path, d,
                     tools=context.config.tools(),
-                    sandbox=context.sandbox(options=["--bind", d, d]),
+                    sandbox=context.sandbox(options=["--ro-bind", path, path, "--bind", d.parent, d.parent]),
                 )
                 bases += [d]
             elif path.suffix == ".raw":


Test patch:
Repository: systemd/mkosi

Issue:
output files invisible to match section
I'm building a kernel (multiple kernel builds for multiple architectures) using mkosi in Format=directory.
I then want to use the data out of this in other build targets, but also use that kernel with `QemuKernel=`.
To this end, I have a drop-in that looks like so:
```ini
[Match]
PathExists=%O/kernel/boot/linux-%a

[Host]
QemuKernel=%O/kernel/boot/linux-%a
```
This never triggers.
If I comment out `PathExists=`, then qemu will boot as expected, but obviously mkosi will refuse to run without an mkosi-built image already being built.

Here's an example minimal setup to reproduce:
```ini
# mkosi.conf
[Distribution]
Distribution=fedora
Release=39

[Content]
Bootable=no
```

```ini
# mkosi.conf.d/qemu.conf
[Match]
PathExists=%O/a/akernel

[Host]
QemuKernel=%O/a/akernel
```

```sh
# ensure output directory exists
mkdir mkosi.output

# build the image
mkosi

# booting via qemu fails for unrelated reasons, this is expected
mkosi qemu

# let's trigger the condition
mkdir mkosi.output/a
touch mkosi.output/a/akernel

# booting via qemu fails for the same reason as before
mkosi qemu

# edit mkosi.conf.d/qemu.conf to comment out the PathExists condition
# we now get the expected error!
mkosi qemu
# => File is smaller than DOS magic (got 0 of 2 bytes)
# => ‣ "bootctl kernel-identify /tmp/t/mkosi.output/a/akernel" returned non-zero exit code 1.
```

I'm running this on fedora 39 with mkosi 20.1, but based on the NEWS file it doesn't seem like this was recently changed.


Discussion:
This is because we only check matches at config parsing time. This means, that this can only match things, that exist at that point in time. The use case, though, is understandable and we should probably add a way to do this.
Thanks!

This is still a bit confusing to me though, since it will fail even after the initial build.
I.e I can test the existence of the file before calling `mkosi`, but it will continue to act as though it doesn't exist.

Resolving patch:
diff --git a/mkosi/config.py b/mkosi/config.py
--- a/mkosi/config.py
+++ b/mkosi/config.py
@@ -2968,6 +2968,8 @@ def match_config(path: Path, namespace: argparse.Namespace, defaults: argparse.N
             negate = v.startswith("!")
             v = v.removeprefix("!")
 
+            v = expand_specifiers(v, namespace, defaults)
+
             if not v:
                 die("Match value cannot be empty")
 
@@ -3590,6 +3592,7 @@ def bold(s: Any) -> str:
                     QEMU Use CD-ROM: {yes_no(config.qemu_cdrom)}
                       QEMU Firmware: {config.qemu_firmware}
             QEMU Firmware Variables: {none_to_none(config.qemu_firmware_variables)}
+                        QEMU Kernel: {none_to_none(config.qemu_kernel)}
                QEMU Extra Arguments: {line_join_list(config.qemu_args)}
 """
 


Test patch:
Repository: systemd/mkosi

Issue:
Firmware files not included in initrd
Even though the initrd seems to be generated correctly (i.e. the log does not mention that the firmware files have not been found) there are no firmware files in the generated initrd. When generating an image with [iwlwifi-mvm-firmware](https://packages.fedoraproject.org/pkgs/linux-firmware/iwlwifi-mvm-firmware/) I get several `iwlwifi 0000:00:14.3: Direct firmware load for iwlwifi-QuZ-a0-hr-b0-63.ucode failed with error -2` messages in dmesg, one for every firmware version. I manually booted into the initrd `rd.rescue` and to my surprise the whole `/usr/lib/firmware` directory was empty.

I have `iwlwifi-mvm-firmware` listed as part of my normal packages and otherwise no options set for initrd. Based on my understanding this should result in all kernel modules (and their required firmware files) being included in the initrd. I even hooked into the initrd generation code and saved all files passed to the function in a file. The firmware files are listed in the initrd generated in `build_kernel_modules_initrd()` but not the final one.


Discussion:
Isn't this a duplicate of #2334?

Resolving patch:
diff --git a/mkosi/installer/__init__.py b/mkosi/installer/__init__.py
--- a/mkosi/installer/__init__.py
+++ b/mkosi/installer/__init__.py
@@ -56,6 +56,7 @@ def mounts(cls, context: Context) -> list[PathString]:
                         Path("/var") / d / subdir / p.relative_to(src),
                     ]
                     for p in caches
+                    if (context.config.package_cache_dir_or_default() / d / subdir / p.relative_to(src)).exists()
                 )
 
         return mounts
diff --git a/mkosi/kmod.py b/mkosi/kmod.py
--- a/mkosi/kmod.py
+++ b/mkosi/kmod.py
@@ -164,6 +164,9 @@ def gen_required_kernel_modules(
     yield modulesd
     yield modulesd / "kernel"
 
+    if (root / "usr/lib/firmware").exists():
+        yield root / "usr/lib/firmware"
+
     for d in (modulesd, root / "usr/lib/firmware"):
         for p in (root / d).rglob("*"):
             if p.is_dir():
diff --git a/mkosi/run.py b/mkosi/run.py
--- a/mkosi/run.py
+++ b/mkosi/run.py
@@ -210,7 +210,7 @@ def preexec() -> None:
             log_process_failure(cmdline, e.returncode)
         if ARG_DEBUG_SHELL.get():
             subprocess.run(
-                [*sandbox, "sh"],
+                [*sandbox, "bash"],
                 check=False,
                 stdin=sys.stdin,
                 text=True,


Test patch:
Repository: systemd/mkosi

Issue:
Unable to set `Include=mkosi-initrd` in `mkosi.conf`
It says that the file `/mkosi-initrd` is not found. It seems like the resource path is not correctly passed through to the point where it is parsed. The CLI flag work, though.


Discussion:

Resolving patch:
diff --git a/mkosi/config.py b/mkosi/config.py
--- a/mkosi/config.py
+++ b/mkosi/config.py
@@ -2885,7 +2885,7 @@ def parse_new_includes(
                     continue
 
                 if any(p == Path(c) for c in BUILTIN_CONFIGS):
-                    _, [config] = parse_config(["--include", os.fspath(path)])
+                    _, [config] = parse_config(["--directory", "", "--include", os.fspath(path)])
                     make_executable(
                         *config.prepare_scripts,
                         *config.postinst_scripts,


Test patch:
Repository: systemd/mkosi

Issue:
s390x issue: bootctl doesn't know kernel-identify verb, assuming 'unknown' kernel type
## Description
when build `s390x` architecture image by command `mkosi -d fedora --architecture s390x -f qemu`, error as below:
```
Created symlink /var/tmp/mkosi-workspacemg93lixd/root/etc/systemd/system/systemd-homed.service.wants/systemd-homed-activate.service → /lib/systemd/system/systemd-homed-activate.service.
‣ bootctl doesn't know kernel-identify verb, assuming 'unknown' kernel type
‣ systemd-firstboot is not installed, not applying first boot settings
‣  Generating hardware database
‣  Removing files…
Traceback (most recent call last):
  File "/usr/local/bin/mkosi/mkosi/run.py", line 55, in uncaught_exception_handler
  File "/usr/local/bin/mkosi/mkosi/run.py", line 96, in fork_and_wait
  File "/usr/local/bin/mkosi/mkosi/__init__.py", line 3788, in run_build
  File "/usr/local/bin/mkosi/mkosi/__init__.py", line 3181, in build_image
  File "/usr/local/bin/mkosi/mkosi/__init__.py", line 2100, in copy_uki
  File "/usr/local/bin/mkosi/mkosi/__init__.py", line 838, in efi_boot_binary
AssertionError
```
That's because `bootctl`, which is provided by systemd-boot for UEFI, but this package is also not available on s390x. Just return kernel type "unknown"


Discussion:

Resolving patch:
diff --git a/mkosi/__init__.py b/mkosi/__init__.py
--- a/mkosi/__init__.py
+++ b/mkosi/__init__.py
@@ -2090,6 +2090,9 @@ def copy_uki(context: Context) -> None:
     if (context.staging / context.config.output_split_uki).exists():
         return
 
+    if not want_efi(context.config):
+        return
+
     ukis = sorted(
         (context.root / "boot/EFI/Linux").glob("*.efi"),
         key=lambda p: GenericVersion(p.name),


Test patch:
Repository: systemd/mkosi

Issue:
bootctl: not found in s390x host build image
when execute command `mkosi -d fedora --architecture s390x -f qemu --bootable no` on a `s390x` host.
encounter the error as 
```

>>> Running trigger-install scriptlet: systemd-udev-0:253.15-2.fc38.s390x
>>> Stop trigger-install scriptlet: systemd-udev-0:253.15-2.fc38.s390x
bootctl: line 1: exec: bootctl: not found
‣ "bootctl --version" returned non-zero exit code 127.
```
It's reported at https://github.com/systemd/mkosi/blob/main/mkosi/qemu.py#L144
(by run the https://github.com/systemd/mkosi/blob/main/mkosi/config.py#L3903)
It lack a condition judgement like this https://github.com/systemd/mkosi/blob/main/mkosi/__init__.py#L1957

Because on the `s390x` platform, there is no `bootctl` available, check the version of `bootctl` cause the failure



Discussion:

Resolving patch:
diff --git a/mkosi/qemu.py b/mkosi/qemu.py
--- a/mkosi/qemu.py
+++ b/mkosi/qemu.py
@@ -141,6 +141,10 @@ class KernelType(StrEnum):
 
     @classmethod
     def identify(cls, config: Config, path: Path) -> "KernelType":
+        if not find_binary("bootctl", root=config.tools()):
+            logging.warning("bootctl is not installed, assuming 'unknown' kernel type")
+            return KernelType.unknown
+
         if systemd_tool_version(config, "bootctl") < 253:
             logging.warning("bootctl doesn't know kernel-identify verb, assuming 'unknown' kernel type")
             return KernelType.unknown


Test patch:
Repository: systemd/mkosi

Issue:
Generating disk images is broken
Minimum example (on master)
===========================

```
mkosi.conf:
```

```
[Output]
Output=disk_file
Format=disk

[Distribution]
Distribution=ubuntu
Release=jammy
Repositories=main restricted universe multiverse

[Content]
CleanPackageMetadata=no
WithDocs=no
Packages=systemd
         systemd-sysv
         libpam-systemd
         udev
         dbus
```

```
$ mkosi
```

Result
======

```
bwrap: Can't find source path /home/gdonval/.cache/mkosi/mkosi-workspace50iodapf/staging/disk_file: No such file or directory
bwrap: Can't find source path /home/gdonval/.cache/mkosi/mkosi-workspace50iodapf/staging/disk_file: No such file or directory
‣ "cp --recursive --no-dereference --preserve=mode,links,timestamps,ownership,xattr --reflink=auto --copy-contents /home/gdonval/.cache/mkosi/mkosi-workspace50iodapf/staging/disk_file /home/gdonval/Documents/Work/RVMI/CustomOS/nwe/disk_file" returned non-zero exit code 1.
```

The code [there](https://github.com/systemd/mkosi/blob/main/mkosi/__init__.py#L3402) creates a `.../staging/disk_file -> .../staging/disk_file.raw` symlink.

Just below that, the call to `finalize_staging(ctx)` triggers the [moving](https://github.com/systemd/mkosi/blob/main/mkosi/__init__.py#L3192) of the files from `.../staging` to `%O/` (i.e. `mkosi.output` by default) from **within** the `bwrap` sandbox.

The move fails (as shown above). Hooking `print(run(["ls", src.parent]))` in different places of `move_tree` shows that the `disk_file` symlink exists, before and after the sandboxed copy happened.


Discussion:

Resolving patch:
diff --git a/mkosi/__init__.py b/mkosi/__init__.py
--- a/mkosi/__init__.py
+++ b/mkosi/__init__.py
@@ -3189,6 +3189,12 @@ def finalize_staging(context: Context) -> None:
         # Make sure all build outputs that are not directories are owned by the user running mkosi.
         if not f.is_dir():
             os.chown(f, INVOKING_USER.uid, INVOKING_USER.gid, follow_symlinks=False)
+
+        if f.is_symlink():
+            (context.config.output_dir_or_cwd() / f.name).symlink_to(f.readlink())
+            os.chown(f, INVOKING_USER.uid, INVOKING_USER.gid, follow_symlinks=False)
+            continue
+
         move_tree(
             f, context.config.output_dir_or_cwd(),
             use_subvolumes=context.config.use_subvolumes,


Test patch:
Repository: systemd/mkosi

Issue:
$MKOSI_CONFIG file is empty
Just noticed that after pulling new changes that `$MKOSI_CONFIG` is containing the same file but that file is empty.
Bisecting reveals that this was introduced with 9dd49ad22944bdb90a748d5b35d138ed1ce0ae2a

To reproduce:
```bash
cd $(mktemp -d)
cat <<EOF >mkosi.sync
#!/bin/bash

cat \$MKOSI_CONFIG

exit 1
EOF
chmod +x mkosi.sync
mkosi
```


Discussion:

Resolving patch:
diff --git a/mkosi/__init__.py b/mkosi/__init__.py
--- a/mkosi/__init__.py
+++ b/mkosi/__init__.py
@@ -428,6 +428,7 @@ def finalize_chroot_scripts(context: Context) -> contextlib.AbstractContextManag
 def finalize_config_json(config: Config) -> Iterator[Path]:
     with tempfile.NamedTemporaryFile(mode="w") as f:
         f.write(config.to_json())
+        f.flush()
         yield Path(f.name)
 
 


Test patch:
Repository: systemd/mkosi

Issue:
/var/lib/dpkg/available does not exist and causes errors when installing packages using apt in image
### mkosi commit the issue has been seen with

v22

### Used host distribution

Debian 12 (backports enabled)

### Used target distribution

Ubuntu 20.04

### Linux kernel version used

6.1.0-18-amd64

### CPU architectures issue was seen on

x86_64

### Unexpected behaviour you saw

mkosi does not create /var/lib/dpkg/available which is used by the old tool dselect. This caused errors when using piuparts to test Debian packages. The error occurs when piuparts installs the package to be tested into a chroot environment.

We have already found a workaround by running the following command in a prepare script:
```bash
dpkg --update-avail
```

I'm wondering if this command should be part of the mkosi build process.

### Used mkosi config

```ini
[Output]
Format=tar
CompressOutput=false

[Host]
QemuSwtpm=False
# setup qemu to exit on kernel panic
# https://unix.stackexchange.com/questions/443017/can-i-make-qemu-exit-with-failure-on-kernel-panic
QemuArgs=-no-reboot
KernelCommandLineExtra=panic=-1

[Content]
Bootable=false
WithDocs=false
PackageDirectories=../extra_packages/

[Packages]
WithNetwork=true

[Distribution]
Architecture=x86-64
Distribution=ubuntu
Release=focal
Repositories=main,universe

@Mirror=http://my-package-mirror.local/ubuntu

[Content]
Packages=apt
         apt-transport-https
         sudo
         ca-certificates
         ca-custom
         gpg
SkeletonTrees=../../../../data/rootfs
```


### mkosi output

```sh
piuparts output:

Preconfiguring packages ...
  dpkg: error: failed to open package info file '/var/lib/dpkg/available' for reading: No such file or directory
  Fetched 497 MB in 11s (44.2 MB/s)
  E: Sub-process dpkg --set-selections returned an error code (2)
  E: Couldn't record the approved state changes as dpkg selection states
```



Discussion:

Resolving patch:
diff --git a/mkosi/installer/apt.py b/mkosi/installer/apt.py
--- a/mkosi/installer/apt.py
+++ b/mkosi/installer/apt.py
@@ -82,6 +82,8 @@ def setup(cls, context: Context, repos: Iterable[Repository]) -> None:
             (context.root / "var/lib/dpkg").mkdir(parents=True, exist_ok=True)
             (context.root / "var/lib/dpkg/status").touch()
 
+            (context.root / "var/lib/dpkg/available").touch()
+
         # We have a special apt.conf outside of pkgmngr dir that only configures "Dir::Etc" that we pass to APT_CONFIG
         # to tell apt it should read config files from /etc/apt in case this is overridden by distributions. This is
         # required because apt parses CLI configuration options after parsing its configuration files and as such we


Test patch:
Repository: systemd/mkosi

Issue:
mkosi-initrd kernel-install script doesn't work when kernel-install is called from `/`
### mkosi commit the issue has been seen with

76687febf72d00693d94382f782f4fd960848a03

### Used host distribution

Fedora 40

### Used target distribution

Fedora 40

### Linux kernel version used

6.8.9-300.fc40.x86_64

### CPU architectures issue was seen on

x86_64

### Unexpected behaviour you saw

The kernel-install script for mkosi-initrd doesn't work when kernel-install is called from `/` (which happens to be where it is called from in the `%posttrans` script of `kernel-core` using `dnf` (and `dnf5`)).

If `kernel-install` is called from (for instance) `/root/` (or anywhere else that doesn't contain `/var/tmp/` it works perfectly fine.

This seems to come from https://github.com/systemd/mkosi/blob/55f17a8ae67ece9712324c3107804daad94ad0d9/kernel-install/50-mkosi.install#L133 which keep the current directory as mkosi's working directory. (and the fact that kernel-install doesn't normalize the working directory.)

### Used mkosi config

_No response_

### mkosi output

```sh
‣ Building initrd
‣ The workspace directory (/var/tmp) must be located outside the current working directory (/)
‣ (Use WorkspaceDirectory= to configure a different workspace directory)
‣ "/usr/local/bin/mkosi --directory '' --format cpio --output initrd --workspace-dir=/var/tmp --package-cache-dir=/var --cache-only=metadata --output-dir /tmp/kernel-install.staging.OLhobX --extra-tree /usr/lib/modules/6.8.9-300.fc40.x86_64:/usr/lib/modules/6.8.9-300.fc40.x86_64 --extra-tree=/usr/lib/firmware:/usr/lib/firmware '--kernel-modules-exclude=.*' --kernel-modules-include-host=yes --include=mkosi-initrd --include /etc/mkosi-initrd --package-manager-tree /tmp/tmpc7v4onop" returned non-zero exit code 1.
```



Discussion:
On my end I made it use `/etc/kernel` as it's working directory as that seemed to make sense.

Resolving patch:
diff --git a/mkosi/__init__.py b/mkosi/__init__.py
--- a/mkosi/__init__.py
+++ b/mkosi/__init__.py
@@ -4233,10 +4233,6 @@ def finalize_default_tools(args: Args, config: Config, *, resources: Path) -> Co
 def check_workspace_directory(config: Config) -> None:
     wd = config.workspace_dir_or_default()
 
-    if wd.is_relative_to(Path.cwd()):
-        die(f"The workspace directory ({wd}) must be located outside the current working directory ({Path.cwd()})",
-            hint="Use WorkspaceDirectory= to configure a different workspace directory")
-
     for tree in config.build_sources:
         if wd.is_relative_to(tree.source):
             die(f"The workspace directory ({wd}) cannot be a subdirectory of any source directory ({tree.source})",


Test patch: