internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: systemd/mkosi

Issue:
CI failures for Fedora and CentOS
I can reproduce this locally with Fedora 33:

```
SeaBIOS (version ArchLinux 1.14.0-1)
Booting from Hard Disk...
...
```

When starting QEMU, it gets stuck in a boot loop and prints this message every time. I'm super confused as to why it's using version ArchLinux on Fedora. Maybe that has something to do with the failure.




Discussion:
Narrowed it down to 0d489464ca00f911159405e49d90c280c12690d6. I guess there's a missing device that causes boot to fail but I wonder which one and why it is Fedora specific.

Resolving patch:
diff --git a/mkosi/__init__.py b/mkosi/__init__.py
--- a/mkosi/__init__.py
+++ b/mkosi/__init__.py
@@ -5483,6 +5483,10 @@ def run_qemu(args: CommandLineArguments) -> None:
         # -nodefaults removes the default CDROM device which avoids an error message during boot
         # -serial mon:stdio adds back the serial device removed by -nodefaults.
         cmdline += ["-nographic", "-nodefaults", "-serial", "mon:stdio"]
+        # Fix for https://github.com/systemd/mkosi/issues/559. QEMU gets stuck in a boot loop when using BIOS
+        # if there's no vga device.
+        if "bios" in args.boot_protocols:
+            cmdline += ["-vga", "virtio"]
 
     if args.network_veth:
         # On Linux, interface names are limited to 16 characters so we do the same.


Test patch:
Repository: systemd/mkosi

Issue:
Output dir variable in the finalize script
In my case i need to copy some artifacts, like ssh key, from image to output dir. I did not find any way to do this expect the hardcode mkosi.output in the finalize script. Environment variable like $OUTPUTDIR will be very useful to this case


Discussion:

Resolving patch:
diff --git a/mkosi/__init__.py b/mkosi/__init__.py
--- a/mkosi/__init__.py
+++ b/mkosi/__init__.py
@@ -2896,6 +2896,10 @@ def run_postinst_script(args: CommandLineArguments, root: str, loopdev: Optional
         os.unlink(os.path.join(root, "root/postinst"))
 
 
+def output_dir(args: CommandLineArguments) -> str:
+    return args.output_dir or os.getcwd()
+
+
 def run_finalize_script(args: CommandLineArguments, root: str, do_run_build_script: bool, for_cache: bool) -> None:
     if args.finalize_script is None:
         return
@@ -2905,7 +2909,7 @@ def run_finalize_script(args: CommandLineArguments, root: str, do_run_build_scri
     verb = "build" if do_run_build_script else "final"
 
     with complete_step('Running finalize script'):
-        env = collections.ChainMap({'BUILDROOT': root}, os.environ)
+        env = collections.ChainMap({'BUILDROOT': root, 'OUTPUTDIR': output_dir(args)}, os.environ)
         run([args.finalize_script, verb], env=env)
 
 


Test patch:
Repository: systemd/mkosi

Issue:
Debian unstable with HostonlyInitrd enabled fails to boot in QEMU
Error: `[FAILED] Failed to start Switch Root.`

Suspicious logs:

```
Dec 10 18:44:52 localhost systemd-gpt-auto-generator[213]: EFI loader partition unknown, exiting.
Dec 10 18:44:52 localhost systemd-gpt-auto-generator[213]: (The boot loader did not set EFI variable LoaderDevicePartUUID.)
```

Config to reproduce: 

```
[Distribution]
Distribution=debian
Release=unstable

[Output]
Bootable=yes
HostonlyInitrd=yes

[Host]
QemuHeadless=yes
```


Discussion:

Resolving patch:
diff --git a/mkosi/__init__.py b/mkosi/__init__.py
--- a/mkosi/__init__.py
+++ b/mkosi/__init__.py
@@ -1416,6 +1416,10 @@ def configure_dracut(args: CommandLineArguments, root: str) -> None:
     ):
         with open(os.path.join(dracut_dir, "30-mkosi-uefi-stub.conf"), "w") as f:
             f.write("uefi_stub=/usr/lib/systemd/boot/efi/linuxx64.efi.stub\n")
+
+    # efivarfs must be present in order to GPT root discovery work
+    if args.esp_partno is not None:
+        with open(os.path.join(dracut_dir, "30-mkosi-efivarfs.conf"), "w") as f:
             f.write('add_drivers+=" efivarfs "\n')
 
 


Test patch:
Repository: systemd/mkosi

Issue:
RFC: Switch order of mkosi.extra/ and copying build results?
In the case of systemd, copying the build results into the final image also copies a bunch of configuration files into the final image. We currently can't override these by using mkosi.extra/. I'm unsure if there's any big advantage to the current order so I propose we switch the order so we can override installed files from the build if needed.


Discussion:

Resolving patch:
diff --git a/mkosi/__init__.py b/mkosi/__init__.py
--- a/mkosi/__init__.py
+++ b/mkosi/__init__.py
@@ -6543,9 +6543,9 @@ def build_image(
                 install_etc_hostname(args, root, cached_tree)
                 install_boot_loader(args, root, loopdev, do_run_build_script, cached_tree)
                 run_prepare_script(args, root, do_run_build_script, cached_tree)
-                install_extra_trees(args, root, for_cache)
                 install_build_src(args, root, do_run_build_script, for_cache)
                 install_build_dest(args, root, do_run_build_script, for_cache)
+                install_extra_trees(args, root, for_cache)
                 set_root_password(args, root, do_run_build_script, cached_tree)
                 set_serial_terminal(args, root, do_run_build_script, cached_tree)
                 set_autologin(args, root, do_run_build_script, cached_tree)


Test patch:
Repository: systemd/mkosi

Issue:
--debug flag doesn't accept multiple values
```
mkosi --debug=disk,run build
...
mkosi: error: argument --debug: invalid choice: 'disk,run' (choose from 'run', 'build-script', 'workspace-command', 'disk')
```

I tried quoting and using square brackets but I always got the same error. Am I missing something obvious?


Discussion:
That seems to be fallout from #671. 

When not passing `choices` to `super().__init__` in `ListAction.__init__` it works as expected, but when we pass it this is caught somewhere in argpases code before reaching our `call` implementation. I assume the string `disk,run` is checked against the valid choices before anything else is done. 

Looks like we need to manage the list of available choices ourselves for now.

Resolving patch:
diff --git a/mkosi/__init__.py b/mkosi/__init__.py
--- a/mkosi/__init__.py
+++ b/mkosi/__init__.py
@@ -5973,7 +5973,6 @@ def create_parser() -> ArgumentParserMkosi:
     group.add_argument(
         "--debug",
         action=CommaDelimitedListAction,
-        choices=("run", "build-script", "workspace-command", "disk"),
         default=[],
         help="Turn on debugging output",
     )


Test patch:
Repository: systemd/mkosi

Issue:
systemd-veritysetup@root.service not started
I've created an Arch image with these options: `mkosi --distribution=arch --bootable --verity --split-artifacts --qemu-headless --autologin qemu`.
However /dev/mapper/root isn't created: `[   91.134659] archlinux systemd[1]: Timed out waiting for device /dev/mapper/root.` and the emergency shell starts. If I run `# systemctl start systemd-veritysetup@root.service` the root mapper is created and I can start `multi-user.target` afterward.

Should this be fixed in mkosi? What is the right way to make sure `systemd-veritysetup@root.service` is started? 

[Log output](https://gist.github.com/pyfisch/a358037a1dbe072f5039610199e88e39). There is a `blockdev@dev-mapper-root.target` which isn't found.

Edit: I am running mkosi from the main branch, because #1003 is fixed there.


Discussion:

Resolving patch:
diff --git a/mkosi/__init__.py b/mkosi/__init__.py
--- a/mkosi/__init__.py
+++ b/mkosi/__init__.py
@@ -130,8 +130,10 @@
     "/usr/lib/systemd/system-generators/systemd-veritysetup-generator",
     "/usr/lib/systemd/system/initrd-root-fs.target.wants/systemd-repart.service",
     "/usr/lib/systemd/system/initrd-usr-fs.target",
+    "/usr/lib/systemd/system/sysinit.target.wants/veritysetup.target",
     "/usr/lib/systemd/system/systemd-repart.service",
     "/usr/lib/systemd/system/systemd-volatile-root.service",
+    "/usr/lib/systemd/system/veritysetup.target",
     "/usr/lib/systemd/systemd-veritysetup",
     "/usr/lib/systemd/systemd-volatile-root",
     "/usr/bin/systemd-ask-password",


Test patch:
Repository: systemd/mkosi

Issue:
Inserted partition is too small
When a partition is inserted into a GPT image, the inserted partition **contents** does not match the contents of the inserted partition. The partition size is big enough, but the data copied in is too small. This can be triggered when either:

  * The `UsrOnly` option is set
  * The `Minimize` option is set
  * SquashFS is used

I've tracked this down to the use of `os.sendfile()` which is only transferring 2147479552 bytes (2GB - 1020KB). I then found [this bug](https://github.com/python/cpython/issues/79360), especially [this](https://github.com/python/cpython/issues/79360#issuecomment-1093804149) reply.

Test code below.

```python
from os import sendfile

f_in = open('testfile', 'r+b')
f_out = open('/dev/null', 'wb')
for size_gb in range(8):
    _ = f_in.seek(0)
    size_bytes = (size_gb+1) * 2 ** 30
    print(f'Truncated to {f_in.truncate(size_bytes)} bytes')
    size_bytes = sendfile(f_out.fileno(), f_in.fileno(), offset=0, count=size_bytes)
    print(f'Transferred  {size_bytes} bytes')
```

Test output.
```
Truncated to 1073741824 bytes
Transferred  1073741824 bytes
Truncated to 2147483648 bytes
Transferred  2147479552 bytes
Truncated to 3221225472 bytes
Transferred  2147479552 bytes
Truncated to 4294967296 bytes
Transferred  2147479552 bytes
Truncated to 5368709120 bytes
Transferred  2147479552 bytes
Truncated to 6442450944 bytes
Transferred  2147479552 bytes
Truncated to 7516192768 bytes
Transferred  2147479552 bytes
Truncated to 8589934592 bytes
Transferred  2147479552 bytes
```

This explains why I was getting issues with the verity partition data.


Discussion:

Resolving patch:
diff --git a/mkosi/__init__.py b/mkosi/__init__.py
--- a/mkosi/__init__.py
+++ b/mkosi/__init__.py
@@ -3665,7 +3665,15 @@ def insert_partition(
 
             # Without this the entire blob will be read into memory which could exceed system memory
             with open(path, mode='wb') as path_fp:
-                os.sendfile(path_fp.fileno(), blob.fileno(), offset=0, count=blob_size)
+                # Chunk size for 32/64-bit systems
+                # Chunking required because sendfile under Linux has a maximum copy size
+                chunksize = 2 ** 30 if sys.maxsize < 2 ** 32 else 0x7ffff000
+                offset = 0
+                while True:
+                    sent = os.sendfile(path_fp.fileno(), blob.fileno(), offset=offset, count=chunksize)
+                    if not sent:
+                        break
+                    offset += sent
 
     return part
 


Test patch:
Repository: systemd/mkosi

Issue:
Can't buid images with --bootable and --format directory or cpio
Before #1090 the combination of --bootable and --format=directory was rejected. Now an error occurs in `def install_boot_loader`. I'm not sure what the expected behavior is: the function calls `bootctl install` which sets ESP and EFI variables but these don't exist without a GPT image. Should the function be skipped?

cc @DaanDeMeyer 

```
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/foo/mkosi/mkosi/__main__.py", line 36, in <module>
    main()
  File "/home/foo/mkosi/mkosi/__main__.py", line 27, in main
    run_verb(a)
  File "/home/foo/mkosi/mkosi/__init__.py", line 8166, in run_verb
    manifest = build_stuff(config)
  File "/home/foo/mkosi/mkosi/__init__.py", line 7471, in build_stuff
    image = build_image(state, manifest=manifest)
  File "/home/foo/mkosi/mkosi/__init__.py", line 7193, in build_image
    install_boot_loader(state, loopdev, cached_tree)
  File "/home/foo/mkosi/mkosi/__init__.py", line 3223, in install_boot_loader
    assert loopdev is not None
AssertionError
```


Discussion:
Yeah it probably should, I think this is remnant from grub support. bootctl should be fine with directory output, so I think we can drop the loopdev assertion and just install the bootloader, no need to skip the function. Happy to accept a patch.

Resolving patch:
diff --git a/mkosi/__init__.py b/mkosi/__init__.py
--- a/mkosi/__init__.py
+++ b/mkosi/__init__.py
@@ -3217,13 +3217,9 @@ def run_finalize_script(state: MkosiState) -> None:
             env={**state.environment, "BUILDROOT": str(state.root), "OUTPUTDIR": str(output_dir(state.config))})
 
 
-def install_boot_loader(
-    state: MkosiState, loopdev: Optional[Path], cached: bool
-) -> None:
+def install_boot_loader(state: MkosiState, cached: bool) -> None:
     if not state.config.bootable or state.do_run_build_script:
         return
-    assert loopdev is not None
-
     if cached:
         return
 
@@ -7192,7 +7188,7 @@ def build_image(
                 install_extra_trees(state)
                 configure_dracut(state, cached_tree)
                 run_kernel_install(state, cached_tree)
-                install_boot_loader(state, loopdev, cached_tree)
+                install_boot_loader(state, cached_tree)
                 set_root_password(state, cached_tree)
                 set_serial_terminal(state, cached_tree)
                 set_autologin(state, cached_tree)


Test patch:
Repository: systemd/mkosi

Issue:
QEMU sometimes sees incomplete image when --ephemeral is used
When using `--ephemeral` with the `qemu` verb, I've sometimes seen images not boot correctly: the symptom is UEFI dropping me out to a shell and not finding a valid loader to execute.

Digging into this, it looks like the problem is that, when `copy_file_object` takes the code path where it calls `shutil.copyfileobj`, there's a risk that the writes to the destination aren't flushed and visible to external processes (namely, `qemu`). Adding in a call to `f.flush()` resolves this problem.

I'll post a PR shortly with a proposed fix.


Discussion:

Resolving patch:
diff --git a/mkosi/__init__.py b/mkosi/__init__.py
--- a/mkosi/__init__.py
+++ b/mkosi/__init__.py
@@ -616,6 +616,7 @@ def copy_file_object(oldobject: BinaryIO, newobject: BinaryIO) -> None:
         if e.errno not in {errno.EXDEV, errno.EOPNOTSUPP, errno.ENOTTY}:
             raise
         shutil.copyfileobj(oldobject, newobject)
+        newobject.flush()
 
 
 def copy_file(oldpath: PathString, newpath: PathString) -> None:


Test patch:
Repository: systemd/mkosi

Issue:
Only parse configuration files suffixed with ".conf" in dropin directories
I've encountered crash when run `mkosi` command. The problem was it tried to parse `.*.swp` file in `conf.d` directory. I opened config on server, vim created its file, I got python traceback.

Maybe filter out hidden files here?
https://github.com/systemd/mkosi/blob/beee070079e057b83cc8cae791e7b140f6f4b005/mkosi/__init__.py#L2108


Discussion:
We should only parse config files suffixed with .conf

Resolving patch:
diff --git a/mkosi/__init__.py b/mkosi/__init__.py
--- a/mkosi/__init__.py
+++ b/mkosi/__init__.py
@@ -2105,7 +2105,7 @@ def parse_args_file_group(
     for dropin_dir in dirs:
         if dropin_dir.is_dir():
             for entry in sorted(dropin_dir.iterdir()):
-                if entry.is_file():
+                if entry.is_file() and entry.match("*.conf"):
                     config_files += [f"@{entry}"]
 
     # Parse all parameters handled by mkosi.


Test patch:
diff --git a/tests/test_parse_load_args.py b/tests/test_parse_load_args.py
--- a/tests/test_parse_load_args.py
+++ b/tests/test_parse_load_args.py
@@ -59,6 +59,15 @@ def test_os_distribution() -> None:
             config.write_text(f"[Distribution]\nDistribution={dist}")
             assert parse([]).distribution == dist
 
+def test_parse_config_files_filter() -> None:
+    with cd_temp_dir():
+        confd = Path("mkosi.conf.d")
+        confd.mkdir(0o755)
+
+        (confd / "10-file.conf").write_text("[Content]\nPackages=yes")
+        (confd / "20-file.noconf").write_text("[Content]\nPackages=nope")
+
+        assert parse([]).packages == ["yes"]
 
 def test_hostname() -> None:
     assert parse(["--hostname", "name"]).hostname == "name"
Repository: systemd/mkosi

Issue:
Support multiple kernel versions with image version
When setting an image version, the same name is used for the UKI filename for multiple kernel versions.

See here: https://github.com/systemd/mkosi/blob/5616f28505a00dfe58793fadfa81919446024596/mkosi/__init__.py#L880-L881


Discussion:
Yeah that's a bug, can you file a PR to put the kernel version in the name?

Resolving patch:
diff --git a/mkosi/__init__.py b/mkosi/__init__.py
--- a/mkosi/__init__.py
+++ b/mkosi/__init__.py
@@ -878,7 +878,7 @@ def install_unified_kernel(state: MkosiState, roothash: Optional[str]) -> None:
                 boot_count = f'+{state.root.joinpath("etc/kernel/tries").read_text().strip()}'
 
             if state.config.image_version:
-                boot_binary = state.root / f"efi/EFI/Linux/{image_id}_{state.config.image_version}{boot_count}.efi"
+                boot_binary = state.root / f"efi/EFI/Linux/{image_id}_{state.config.image_version}-{kver}{boot_count}.efi"
             elif roothash:
                 _, _, h = roothash.partition("=")
                 boot_binary = state.root / f"efi/EFI/Linux/{image_id}-{kver}-{h}{boot_count}.efi"


Test patch:
Repository: systemd/mkosi

Issue:
mkosi should probably touch `/usr/lib/clock-epoch` during build
It would probably make sense of mkosi would touch /usr/lib/clock-epoch late during image building.

It's mtime is used by PID1 during boot to bump the clock if it is before that timestamp. It is helpful on systems lacking a working clock to at least get into territory where time makes sense at all. 

PID 1 will bump the clock via a built-in timestamp selected at RPM/DEB built-time anyway. But touching this file from mkosi would be a good idea, since of course image builders just take the distro built packages and then package them up anew, and thus at a later point in time.

We should probably do this always, unconditionally. And if the SOURCE_DATE_EPOCH env var is set during mkosi invocation use it for setting the mtime of that file. (as per https://reproducible-builds.org/docs/source-date-epoch/)


Discussion:
Should we put this in systemd-firstboot instead?

Or maybe firstboot is not the right tool, but it'd be nice if something in systemd could handle those few files that need to be configured in an image automatically, basically the stuff from https://github.com/systemd/systemd/blob/main/docs/BUILDING_IMAGES.md. I'm not sure if it really fits into any existing tool though.
> Should we put this in systemd-firstboot instead?

I like this idea. `systemd-firstboot --reset --root=…`. I guess the option should take a list of opt-in/opt-out steps to do: `--reset=hostname` or `--reset=no-hostname,no-epoch`. 

Resolving patch:
diff --git a/mkosi/__init__.py b/mkosi/__init__.py
--- a/mkosi/__init__.py
+++ b/mkosi/__init__.py
@@ -1063,6 +1063,11 @@ def configure_initrd(state: MkosiState) -> None:
         (state.root / "etc/initrd-release").symlink_to("/etc/os-release")
 
 
+def configure_clock(state: MkosiState) -> None:
+    with umask(~0o644):
+        (state.root / "usr/lib/clock-epoch").touch()
+
+
 def run_depmod(state: MkosiState) -> None:
     if state.config.bootable == ConfigFeature.disabled:
         return
@@ -1382,6 +1387,7 @@ def build_image(args: MkosiArgs, config: MkosiConfig) -> None:
             configure_autologin(state)
             configure_initrd(state)
             configure_ssh(state)
+            configure_clock(state)
 
             install_boot_loader(state)
             run_sysusers(state)


Test patch:
Repository: systemd/mkosi

Issue:
KernelCommandLine is reversed
When specifying multiple arguments in one `KernelCommandLine` config option, the kernel cmdline is reversed.

I guess this is due to the list parsing insertion here: https://github.com/systemd/mkosi/blob/main/mkosi/config.py#L384

Ex.: `KernelCommandLine=quiet loglevel=3` results in `Kernel Command Line: loglevel=3 quiet`.
Sometimes the ordering is important though.


Discussion:
This issue has a wider impact. All options are concerned, but particularly the lists where the order is important (such as environment variables).

The configuration is parsed from the highest priority to the lowest, starting with the command line arguments and continuing with the configuration files.

Options with a single value should use the one with the highest priority and lists should be ordered from lowest priority to highest.

However, there are a few cases where the priority is reversed:

- As noticed, a list of values passed as a single string joined by a delimiter are reversed.
- Also, an option set multiple times on the command line gives a higher priority to its earlier values than the later ones. The inverse is usually expected so that a command line option can be overridden by a later one.

Resolving patch:
diff --git a/mkosi/config.py b/mkosi/config.py
--- a/mkosi/config.py
+++ b/mkosi/config.py
@@ -361,6 +361,8 @@ def config_parse_list(dest: str, value: Optional[str], namespace: argparse.Names
         else:
             values = value.replace(delimiter, "\n").split("\n")
 
+        new = []
+
         for v in values:
             if not v:
                 continue
@@ -373,9 +375,9 @@ def config_parse_list(dest: str, value: Optional[str], namespace: argparse.Names
                 if fnmatch.fnmatchcase(v, i):
                     break
             else:
-                l.insert(0, parse(v))
+                new.append(parse(v))
 
-        return l
+        return new + l
 
     return config_parse_list
 


Test patch:
Repository: systemd/mkosi

Issue:
Uki and `.vmlinuz` signatures differ
Following https://github.com/systemd/mkosi/commit/85d8342e974e5537529ea959b4e405fa330467b9, the signatures between the `.vmlinuz` output and the kernel image inside the UKI differ.
The kernel inside the UKI is signed by the mkosi secureboot keys, the `.vmlinuz` output is not.

Since only UEFI hosts are supported with systemd-boot, I don't see why the split out kernel is needed after all, but I guess it should then work the same way as the UKI.


Discussion:
I guess it would make sense to drop the split out stuff, since it can be easily obtained with an objdump from the UKI in a finalize script
This is intended to be there, It's useful to allow booting it's there to allow booting cpios in qemu directly without needing a UKI or rootfs.
Should be trivial to sign the vmlinuz as well if needed.

Resolving patch:
diff --git a/mkosi/__init__.py b/mkosi/__init__.py
--- a/mkosi/__init__.py
+++ b/mkosi/__init__.py
@@ -773,6 +773,20 @@ def install_unified_kernel(state: MkosiState, roothash: Optional[str]) -> None:
             if not (state.staging / state.config.output_split_uki).exists():
                 shutil.copy(boot_binary, state.staging / state.config.output_split_uki)
 
+                # ukify will have signed the kernel image as well. Let's make sure we put the signed kernel
+                # image in the output directory instead of the unsigned one by reading it from the UKI.
+
+                import pefile  # type: ignore
+                pe = pefile.PE(boot_binary, fast_load=True)
+                linux = {s.Name.decode().strip("\0"): s for s in pe.sections}[".linux"]
+                run(["dd",
+                     f"if={boot_binary}",
+                     f"of={state.staging / state.config.output_split_kernel}",
+                     f"skip={linux.PointerToRawData}",
+                     # Get the actual size using Misc_VirtualSize instead of the aligned size from SizeOfRawData.
+                     f"count={linux.Misc_VirtualSize}",
+                     "iflag=skip_bytes,count_bytes"])
+
             print_output_size(boot_binary)
 
     if state.config.bootable == ConfigFeature.enabled and not (state.staging / state.config.output_split_uki).exists():


Test patch: