Repository: systemd/mkosi

Issue:
Missing security updates in images created with debootstrap
Unfortunately, debootstrap doesn't include security updates (or updates from point releases) in /etc/apt/sources.list.

In mkosi, this can get worked around in a mkosi.prepare script, but it's a time-consuming process and requires parsing /etc/apt/sources.list and /etc/os-release to pick the correct mirrors and componentes (in mkosi: repositories).

It would be nice if mkosi could handle this.

In the past I used [mmdebstrap](https://gitlab.mister-muffin.de/josch/mmdebstrap) to create Debian-based root filesystems. It's a drop-in replacement for debootstrap, which solves this problem by using apt to set up the whole system. You might consider supporting it, at least as an alternative, if present.


Discussion:
Sure, we can put this behind an option If it's not too much of a much of a maintenance burden. It doesn't seem like adding this would require too many changes though. PRs are welcome.
I'm not even sure if it needs to be an option. I think we could do it unconditionally for Debian (and Ubuntu?).
I gave it a first try yesterday: https://github.com/systemd/mkosi/compare/master...mtdcr:345a32c5b678fd34bb16b455fd8a9de8708d7681

Issues noticed so far:
- Versions below 0.7.0 can't install into a non-empty root. Version 0.7.1 is available in Debian bullseye, but not in any version of Ubuntu.
- It deletes all installed .deb files unconditionally, so the cache doesn't work as intended (cf. [upstream issue](https://gitlab.mister-muffin.de/josch/mmdebstrap/issues/19)).
- It uses system keyrings, which means creating an Ubuntu image from Debian requires prior installation of `ubuntu-keyring`, for example.
- It only works on systems shipping apt-get, so it's not going to replace debootstrap on non-Debian systems.

- >Versions below 0.7.0 can't install into a non-empty root. Version 0.7.1 is available in Debian bullseye, but not in any version of Ubuntu.

mkosi uses empty roots AFAIK but I might be wrong here.

- >It deletes all installed .deb files unconditionally, so the cache doesn't work as intended (cf. upstream issue).

Without caching support I wouldn't enable this unconditionally. The caching is pretty important as it saves a large amount of time on rebuilds.

- > It uses system keyrings, which means creating an Ubuntu image from Debian requires prior installation of ubuntu-keyring, for example.

The system keyring thing is expected, on Arch I have to install the debian and ubuntu keyrings as well to generate debian/ubuntu images.

- > It only works on systems shipping apt-get, so it's not going to replace debootstrap on non-Debian systems.

There's actually an AUR package for apt but I agree that having to install that would be a degradation in user experience.

I'm kinda torn on using this unconditionally. I'm fine with making it opt-in though so users explicitly opt-in to using this. It's probably a good idea to have a testing period regardless before we switch the default to something else than debootstrap.
> In mkosi, this can get worked around in a mkosi.prepare script, but it's a time-consuming process and requires parsing /etc/apt/sources.list and /etc/os-release to pick the correct mirrors and componentes (in mkosi: repositories).

I'm not quite sure I follow. It's definitely sensible to add an `/etc/apt/sources.list`, but we know a mirror (the one we are bootstrapping from) and we know the suite we are installing and the names for repositories are well-known. Shouldn't it be as easy as dropping a file containing
```
deb {mirror} {args.release} main contrib non-free
```
and not require parsing anything?

Possibly, if `args.release` is not sid, a few more lines for `{args.release}-updates`, `{args.release}/updates` and possibly `{args.release}-backports`?

> mkosi uses empty roots AFAIK but I might be wrong here.

Since 0.7.0 it's possible to pass `--skip check/empty` to it.
> I'm not quite sure I follow. It's definitely sensible to add an `/etc/apt/sources.list`, but we know a mirror (the one we are bootstrapping from) and we know the suite we are installing and the names for repositories are well-known. Shouldn't it be as easy as dropping a file containing
> 
> ```
> deb {mirror} {args.release} main contrib non-free
> ```
> 
> and not require parsing anything?
> 
> Possibly, if `args.release` is not sid, a few more lines for `{args.release}-updates`, `{args.release}/updates` and possibly `{args.release}-backports`?

At which point in mkosi's execution flow would you recommend injecting a sources.list? Do you propose letting mkosi handle it?

What I'm looking for is a solution that works for many different configurations without having to provide redundant information (all the parameters are already known from mkosi.default or the command-line, so it shouldn't be needed to craft a custom sources.list manually), which is why I preferred parsing in a mkosi.prepare script shared by many configurations as a workaround.

Anyway, the most important thing is that whatever solution is preferred, **it should not require an invocation of `apt-get update && apt-get upgrade`**, which is the time consuming part of the problem, and there's no usable hook between the invocation of `debootstrap` (which generates the sources.list) and `apt-get install`.

If I'm missing the obvious, I'd appreciate a kind pointer.
I guess the right point to set a sources.list (and call `apt-get update` and `apt-get upgrade`) is between the call to `debootstrap` and our first call to `apt-get`. Somewhere around like 2150. One could either clobber the debootstrap supplied sources.list or move it out of the way temporarily.

Running `apt-get update` and `apt-get upgrade` also shouldn't take all too long, since the bootstrap is minbase anyway. 

I'd prefer to stay within upstream debootstrap land instead of using a tool that is quite dated in current stable (without a newer version in backports) and that would introduce a second code path for bootstrapping Debian-based systems, if your problem can be solved within those constraints. Maybe I'm misunderstanding your problem, though.
I just bootstrapped a system via
```bash
debootstrap --variant=minbase buster ./buster-chroot/ http://deb.debian.org/debian
```
and added 
```
deb http://deb.debian.org/debian buster-updates main
```
in addition to the generated line 
```
deb http://deb.debian.org/debian buster main
```

The following `apt upgrade` was a noop:
```
root@hostname:/# apt update
Hit:1 http://deb.debian.org/debian buster InRelease
Get:2 http://deb.debian.org/debian buster-updates InRelease [51.9 kB]
Get:3 http://deb.debian.org/debian buster/main Translation-en [5968 kB]
Get:4 http://deb.debian.org/debian buster-updates/main amd64 Packages [7868 B]
Get:5 http://deb.debian.org/debian buster-updates/main Translation-en [5672 B]
Fetched 6033 kB in 2s (3556 kB/s)                                     
Reading package lists... Done
Building dependency tree... Done
All packages are up to date.
root@hostname:/# apt upgrade
Reading package lists... Done
Building dependency tree... Done
Calculating upgrade... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
```

Am I missing something? I fail to see where the packages are not current. Sorry if I'm being dense.
> I guess the right point to set a sources.list (and call apt-get update and apt-get upgrade) is between the call to debootstrap and our first call to apt-get.

Here's an excerpt from mmdebstrap, where it determines the URL of a security mirror.
```perl
    my $mirror    = 'http://deb.debian.org/debian';
    my $secmirror = 'http://security.debian.org/debian-security';
    if (any { $_ eq $suite } @ubuntustable) {
        if (any { $_ eq $arch } ('amd64', 'i386')) {
            $mirror    = 'http://archive.ubuntu.com/ubuntu';
            $secmirror = 'http://security.ubuntu.com/ubuntu';
        } else {
            $mirror    = 'http://ports.ubuntu.com/ubuntu-ports';
            $secmirror = 'http://ports.ubuntu.com/ubuntu-ports';
        }
    ...
```
Would you like to include such information in mkosi?
> Am I missing something? I fail to see where the packages are not current. Sorry if I'm being dense.

I was wrong assuming that $suite-updates would be used for point releases. It's the replacement for the former volatile repository: https://www.debian.org/News/2011/20110215

However, you forgot to add the security repository. There may or may not be security updates for packages in minbase, but we can't know in advance. Security updates for minbase will generally be rare and small. I'm more concerned about missing security updates in other packages.
It would probably make sense to add the security repository, but it's not extremely crucial. If my understanding is correct, it is meant to push security fixes in between the regular sync intervals of regular mirrors, which is at most four hours (I think most sync via some push notification). The contents of this repository change irregularly, but when they come, they're usually gone just as quickly. 

So, yes, it would probably be an improvement to patch these repositories into sources.list, but generally the regular mirrors should be current enough for bootstrapping and afterwards the update cycles of the deployed machine will probably be longer than the sync cycle of the mirror, if the machines are updated at all (instead of redeployed).
> If my understanding is correct, it is meant to push security fixes in between the regular sync intervals of regular mirrors, which is at most four hours (I think most sync via some push notification). The contents of this repository change irregularly, but when they come, they're usually gone just as quickly.

No, that's wrong. Even though security tarballs might get synced into the regular archive early, it doesn't mean they're going to end up in its package index before a new point release every few months.

Here's an overview of recent security advisories for the buster release, starting September 18th: https://www.debian.org/security/#DSAS

The latest [point release 10.6](https://www.debian.org/News/2020/20200926) appeared on September 26th, but does not include any of the fixes listed on that page.

Let's take mediawiki as an example:
```sh
$ apt-cache policy mediawiki
mediawiki:
  Installed: (none)
  Candidate: 1:1.31.10-1~deb10u1
  Version table:
     1:1.35.0-1~bpo10+1 100
        100 http://deb.debian.org/debian buster-backports/main amd64 Packages
        100 http://deb.debian.org/debian buster-backports/main i386 Packages
     1:1.31.10-1~deb10u1 500
        500 http://deb.debian.org/debian-security buster/updates/main amd64 Packages
        500 http://deb.debian.org/debian-security buster/updates/main i386 Packages
     1:1.31.7-1~deb10u1 500
        500 http://deb.debian.org/debian buster/main amd64 Packages
        500 http://deb.debian.org/debian buster/main i386 Packages
```
You can find version 1:1.31.10-1~deb10u1 on both http://deb.debian.org/debian/pool/main/m/mediawiki/ and http://security.debian.org/debian-security/pool/main/m/mediawiki/, but it gets referenced from the package lists of debian-security only.
Thank you for the clarification. The point remains, though, that dropping a single line for the security repositories into `sources.list` and running `apt update` and `apt upgrade` should be sufficient.
mmdebstrap is amazing. Please add an option for it.

Caching can be done using apt-cacher-ng.
(apt-cacher-ng works with both, debootstrap and mmdebstrap.)
The speed gains of mmdebstrap over debootstrap are amazing.
Also security of mmdebstrap is better since it supports valid-until (since it uses APT) and since it can bootstrap from multiple repositories (regular and security) at the same time.

> Thank you for the clarification. The point remains, though, that dropping a single line for the security repositories into `sources.list` and running `apt update` and `apt upgrade` should be sufficient.

No, sometimes APT itself is vulnerable. In these cases, mmdebstrap is the only secure way to bootstrap the patched APT from the security repository. Thereby skipping to ever install the vulnerable APT from the non-security (regular) repository. By installing a vulnerable APT from the regular repository first, then running apt update and apt upgrade, one is exposed during times where APT didn't yet migrate from security repository to regular repository.

//cc @josch (creator of mmdebstrap)

(Words are my own. Not affiliated with mmdebstrap.)

> Thank you for the clarification. The point remains, though, that dropping a single line for the security repositories into `sources.list` and running `apt update` and `apt upgrade` should be sufficient.

I agree. Somebody who uses Debian or Ubuntu should add this… It should be a few lines of code at most.
(We might or might not add support for mmdebstrap later on, but since it seems unlikely we'll get rid of support for debootstrap, we need to fix this issue here too.)

Resolving patch:
diff --git a/mkosi/__init__.py b/mkosi/__init__.py
--- a/mkosi/__init__.py
+++ b/mkosi/__init__.py
@@ -131,6 +131,8 @@
     "/usr/lib/systemd/system/systemd-volatile-root.service",
     "/usr/lib/systemd/systemd-veritysetup",
     "/usr/lib/systemd/systemd-volatile-root",
+    "/usr/bin/systemd-ask-password",
+    "/usr/bin/systemd-tty-ask-password-agent"
 ]
 
 
@@ -2684,10 +2686,10 @@ def invoke_apt(
     do_run_build_script: bool,
     root: Path,
     command: str,
-    packages: Iterable[str],
+    extra: Iterable[str],
 ) -> None:
 
-    cmdline = ["/usr/bin/apt-get", "--assume-yes", "--no-install-recommends", "--auto-remove", command, *packages]
+    cmdline = ["/usr/bin/apt-get", "--assume-yes", command, *extra]
     env = {
         "DEBIAN_FRONTEND": "noninteractive",
         "DEBCONF_NONINTERACTIVE_SEEN": "true",
@@ -2703,13 +2705,13 @@ def install_debian_or_ubuntu(args: MkosiArgs, root: Path, *, do_run_build_script
     os.makedirs(dpkg_io_conf.parent, mode=0o755, exist_ok=True)
     dpkg_io_conf.write_text("force-unsafe-io\n")
 
+    repos = set(args.repositories) or {"main"}
+    # Ubuntu needs the 'universe' repo to install 'dracut'
+    if args.distribution == Distribution.ubuntu and args.bootable:
+        repos.add("universe")
+
     # debootstrap fails if a base image is used with an already populated root, so skip it.
     if args.base_image is None:
-        repos = set(args.repositories) or {"main"}
-        # Ubuntu needs the 'universe' repo to install 'dracut'
-        if args.distribution == Distribution.ubuntu and args.bootable:
-            repos.add("universe")
-
         cmdline: List[PathString] = [
             "debootstrap",
             "--variant=minbase",
@@ -2792,7 +2794,25 @@ def install_debian_or_ubuntu(args: MkosiArgs, root: Path, *, do_run_build_script
             if "VERSION_ID" not in os_release and "BUILD_ID" not in os_release:
                 f.write(f"BUILD_ID=mkosi-{args.release}\n")
 
-    invoke_apt(args, do_run_build_script, root, "install", extra_packages)
+    if args.release not in ("testing", "unstable"):
+        if args.distribution == Distribution.ubuntu:
+            updates = f"deb http://archive.ubuntu.com/ubuntu {args.release}-updates {' '.join(repos)}"
+        else:
+            updates = f"deb http://deb.debian.org/debian {args.release}-updates {' '.join(repos)}"
+
+        root.joinpath(f"etc/apt/sources.list.d/{args.release}-updates.list").write_text(f"{updates}\n")
+
+        if args.distribution == Distribution.ubuntu:
+            security = f"deb http://archive.ubuntu.com/ubuntu {args.release}-security {' '.join(repos)}"
+        elif args.release in ("stretch", "buster"):
+            security = f"deb http://security.debian.org/debian-security/ {args.release}/updates main"
+        else:
+            security = f"deb https://security.debian.org/debian-security {args.release}-security main"
+
+        root.joinpath(f"etc/apt/sources.list.d/{args.release}-security.list").write_text(f"{security}\n")
+
+    invoke_apt(args, do_run_build_script, root, "update", [])
+    invoke_apt(args, do_run_build_script, root, "install", ["--no-install-recommends", *extra_packages])
 
     policyrcd.unlink()
     dpkg_io_conf.unlink()
@@ -2815,6 +2835,9 @@ def install_debian_or_ubuntu(args: MkosiArgs, root: Path, *, do_run_build_script
         for kver, kimg in gen_kernel_images(args, root):
             run_workspace_command(args, root, ["kernel-install", "add", kver, Path("/") / kimg])
 
+    root.joinpath("etc/locale.conf").write_text("LANG=C.UTF-8\n")
+    root.joinpath("etc/default/locale").symlink_to("/etc/locale.conf")
+
 
 @complete_step("Installing Debian…")
 def install_debian(args: MkosiArgs, root: Path, do_run_build_script: bool) -> None:
@@ -2842,14 +2865,14 @@ def patch_locale_gen(args: MkosiArgs, root: Path) -> None:
     try:
 
         def _patch_line(line: str) -> str:
-            if line.startswith("#en_US.UTF-8"):
+            if line.startswith("#C.UTF-8"):
                 return line[1:]
             return line
 
         patch_file(root / "etc/locale.gen", _patch_line)
 
     except FileNotFoundError:
-        root.joinpath("etc/locale.gen").write_text("en_US.UTF-8 UTF-8\n")
+        root.joinpath("etc/locale.gen").write_text("C.UTF-8 UTF-8\n")
 
 
 @complete_step("Installing Arch Linux…")
@@ -3021,7 +3044,7 @@ def install_arch(args: MkosiArgs, root: Path, do_run_build_script: bool) -> None
     patch_locale_gen(args, root)
     run_workspace_command(args, root, ["/usr/bin/locale-gen"])
 
-    root.joinpath("etc/locale.conf").write_text("LANG=en_US.UTF-8\n")
+    root.joinpath("etc/locale.conf").write_text("LANG=C.UTF-8\n")
 
     # Arch still uses pam_securetty which prevents root login into
     # systemd-nspawn containers. See https://bugs.archlinux.org/task/45903.
@@ -3190,7 +3213,7 @@ def remove_packages(args: MkosiArgs, root: Path) -> None:
         args.distribution != Distribution.photon):
         remove = lambda p: invoke_dnf(args, root, 'remove', p)
     elif args.distribution.package_type == PackageType.deb:
-        remove = lambda p: invoke_apt(args, False, root, "purge", p)
+        remove = lambda p: invoke_apt(args, False, root, "purge", ["--auto-remove", *p])
     else:
         # FIXME: implement removal for other package managers: tdnf, swupd, pacman
         return
@@ -5622,6 +5645,8 @@ def create_parser() -> ArgumentParserMkosi:
     group.add_argument("--qemu-headless", action=BooleanAction, help="Configure image for qemu's -nographic mode")
     group.add_argument("--qemu-smp", help="Configure guest's SMP settings", metavar="SMP", default="2")
     group.add_argument("--qemu-mem", help="Configure guest's RAM size", metavar="MEM", default="1G")
+    group.add_argument("--qemu-kvm", action=BooleanAction, help="Configure whether to use KVM or not",
+                       default=qemu_check_kvm_support())
     group.add_argument(
         "--nspawn-keep-unit",
         action=BooleanAction,
@@ -6636,6 +6661,9 @@ def load_args(args: argparse.Namespace) -> MkosiArgs:
     if args.base_image is not None:
         args.base_packages = False
 
+    if args.qemu_kvm and not qemu_check_kvm_support():
+        die("Sorry, the host machine does not support KVM acceleration.")
+
     return MkosiArgs(**vars(args))
 
 
@@ -6925,6 +6953,12 @@ def setup_ssh(
                               ListenStream=
                               ListenStream={args.ssh_port}
                               """)
+
+        add_dropin_config(root, "ssh@.service", "runtime-directory-preserve",
+                          """\
+                          [Service]
+                          RuntimeDirectoryPreserve=yes
+                          """)
     else:
         unit = "sshd"
 
@@ -7425,12 +7459,14 @@ def suppress_stacktrace() -> Iterator[None]:
         raise MkosiException() from e
 
 
-def virt_name(args: MkosiArgs) -> str:
+def machine_name(args: MkosiArgs) -> str:
+    return args.hostname or args.image_id or args.output.with_suffix("").name.partition("_")[0]
+
 
-    name = args.hostname or args.image_id or args.output.with_suffix("").name.partition("_")[0]
-    # Shorten to 13 characters so we can prefix with ve- or vt- for the netdev ifname which is limited
-    # to 16 characters.
-    return name[:13]
+def interface_name(args: MkosiArgs) -> str:
+    # Shorten to 12 characters so we can prefix with ve- or vt- for the netdev ifname which is limited
+    # to 15 characters.
+    return machine_name(args)[:12]
 
 
 def has_networkd_vm_vt() -> bool:
@@ -7517,7 +7553,7 @@ def run_shell_cmdline(args: MkosiArgs, pipe: bool = False, commands: Optional[Se
     if args.ephemeral:
         cmdline += ["--ephemeral"]
 
-    cmdline += ["--machine", virt_name(args)]
+    cmdline += ["--machine", machine_name(args)]
 
     if args.nspawn_keep_unit:
         cmdline += ["--keep-unit"]
@@ -7632,8 +7668,7 @@ def qemu_check_kvm_support() -> bool:
 
 @contextlib.contextmanager
 def run_qemu_cmdline(args: MkosiArgs) -> Iterator[List[str]]:
-    has_kvm = qemu_check_kvm_support()
-    accel = "kvm" if has_kvm else "tcg"
+    accel = "kvm" if args.qemu_kvm else "tcg"
 
     firmware, fw_supports_sb = find_qemu_firmware()
 
@@ -7651,7 +7686,7 @@ def run_qemu_cmdline(args: MkosiArgs) -> Iterator[List[str]]:
         "virtio-rng-pci,rng=rng0,id=rng-device0",
     ]
 
-    if has_kvm:
+    if args.qemu_kvm:
         cmdline += ["-cpu", "host"]
 
     if args.qemu_headless:
@@ -7671,7 +7706,7 @@ def run_qemu_cmdline(args: MkosiArgs) -> Iterator[List[str]]:
             cmdline += ["-nic", f"user,model=virtio-net-pci{fwd}"]
         else:
             # Use vt- prefix so we can take advantage of systemd-networkd's builtin network file for VMs.
-            ifname = f"vt-{virt_name(args)}"
+            ifname = f"vt-{interface_name(args)}"
             # vt-<image-name> is the ifname on the host and is automatically picked up by systemd-networkd which
             # starts a DHCP server on that interface. This gives IP connectivity to the VM. By default, QEMU
             # itself tries to bring up the vt network interface which conflicts with systemd-networkd which is
@@ -7735,7 +7770,7 @@ def find_address(args: MkosiArgs) -> Tuple[str, str]:
     if not ensure_networkd(args) and args.ssh_port != 22:
         return "", "127.0.0.1"
 
-    name = virt_name(args)
+    name = interface_name(args)
     timeout = float(args.ssh_timeout)
 
     while timeout >= 0:
@@ -7746,7 +7781,7 @@ def find_address(args: MkosiArgs) -> Tuple[str, str]:
             elif interface_exists(f"vt-{name}"):
                 dev = f"vt-{name}"
             else:
-                raise MkosiException("Container/VM interface not found")
+                die(f"Container/VM interface ve-{name}/vt-{name} not found")
 
             link = json.loads(run(["ip", "-j", "link", "show", "dev", dev], stdout=PIPE, text=True).stdout)[0]
             if link["operstate"] == "DOWN":
@@ -7783,7 +7818,7 @@ def run_command_image(args: MkosiArgs, commands: Sequence[str], timeout: int, ch
     if args.verb == Verb.qemu:
         return run_ssh(args, commands, check, stdout, stderr, timeout)
     elif args.verb == Verb.boot:
-        cmdline = ["systemd-run", "--quiet", "--wait", "--pipe", "-M", virt_name(args), "/usr/bin/env", *commands]
+        cmdline = ["systemd-run", "--quiet", "--wait", "--pipe", "-M", machine_name(args), "/usr/bin/env", *commands]
         return run(cmdline, check=check, stdout=stdout, stderr=stderr, text=True, timeout=timeout)
     else:
         return run(run_shell_cmdline(args, pipe=True, commands=commands), check=check, stdout=stdout, stderr=stderr, text=True, timeout=timeout)
diff --git a/mkosi/backend.py b/mkosi/backend.py
--- a/mkosi/backend.py
+++ b/mkosi/backend.py
@@ -516,6 +516,7 @@ class MkosiArgs:
     qemu_headless: bool
     qemu_smp: str
     qemu_mem: str
+    qemu_kvm: bool
 
     # systemd-nspawn specific options
     nspawn_keep_unit: bool
diff --git a/mkosi/machine.py b/mkosi/machine.py
--- a/mkosi/machine.py
+++ b/mkosi/machine.py
@@ -6,9 +6,10 @@
 import os
 import signal
 import subprocess
+import sys
 import unittest
 from textwrap import dedent
-from typing import Any, Iterator, Optional, Sequence
+from typing import Any, Iterator, Optional, Sequence, TextIO, Union
 
 import pexpect  # type: ignore
 
@@ -34,11 +35,10 @@
 
 
 class Machine:
-    def __init__(self, args: Sequence[str] = [], debug: bool = False) -> None:
+    def __init__(self, args: Sequence[str] = []) -> None:
         # Remains None until image is built and booted, then receives pexpect process.
         self._serial: Optional[pexpect.spawn] = None
         self.exit_code: int = -1
-        self.debug = debug
         self.stack = contextlib.ExitStack()
         self.args: MkosiArgs
 
@@ -63,12 +63,12 @@ def __init__(self, args: Sequence[str] = [], debug: bool = False) -> None:
         tmp.force = 1
         tmp.autologin = True
         tmp.ephemeral = True
-        tmp.bootable = True
         if tmp.verb == Verb.qemu:
+            tmp.bootable = True
             tmp.qemu_headless = True
-            tmp.hostonly_initrd = True
             tmp.netdev = True
             tmp.ssh = True
+            tmp.ssh_timeout = 240
         elif tmp.verb not in (Verb.shell, Verb.boot):
             die("No valid verb was entered.")
 
@@ -134,19 +134,23 @@ def boot(self) -> None:
             # Then we tell the process to look for the # sign, which indicates the CLI for that image is active.
             # Once we've build/boot an image the CLI will prompt "root@image ~]# ".
             # Then, when pexpects finds the '#' it means we're ready to interact with the process.
-            self._serial = pexpect.spawnu(cmd, logfile=None, timeout=240)
+            self._serial = pexpect.spawnu(cmd, logfile=sys.stdout, timeout=240)
             self._serial.expect("#")
             self.stack = stack.pop_all()
 
-    def run(self, commands: Sequence[str], timeout: int = 900, check: bool = True) -> CompletedProcess:
+    def run(
+        self,
+        commands: Sequence[str],
+        timeout: int = 900,
+        check: bool = True,
+        capture_output: bool = False,
+    ) -> CompletedProcess:
         self._ensure_booted()
 
-        process = run_command_image(self.args, commands, timeout, check, subprocess.PIPE, subprocess.PIPE)
-        if self.debug:
-            print(f"Stdout:\n {process.stdout}")
-            print(f"Stderr:\n {process.stderr}")
+        stdout: Union[int, TextIO] = subprocess.PIPE if capture_output else sys.stdout
+        stderr: Union[int, TextIO] = subprocess.PIPE if capture_output else sys.stderr
 
-        return process
+        return run_command_image(self.args, commands, timeout, check, stdout, stderr)
 
     def kill(self) -> None:
         self.__exit__(None, None, None)
@@ -160,7 +164,7 @@ def __exit__(self, *args: Any, **kwargs: Any) -> None:
 
 
 @contextlib.contextmanager
-def test_skip_not_supported() -> Iterator[None]:
+def skip_not_supported() -> Iterator[None]:
     """See if load_args() raises exception about args and added configurations on __init__()."""
     try:
         yield
@@ -177,7 +181,7 @@ def __init_subclass__(cls, args: Sequence[str] = []) -> None:
 
     @classmethod
     def setUpClass(cls) -> None:
-        with test_skip_not_supported():
+        with skip_not_supported():
             cls.machine = Machine(cls.args)
 
         verb = cls.machine.args.verb


Test patch:
diff --git a/tests/conftest.py b/tests/conftest.py
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -30,7 +30,9 @@ def valid(self) -> Set[str]:
         return {o for o in self.intersect if self.expected_dict[o] == self.current_dict[o]}
 
 
-def pytest_assertrepr_compare(op: str, left: MkosiConfig, right: Dict[str, Any]) -> List[str]:
+def pytest_assertrepr_compare(op: str, left: MkosiConfig, right: Any) -> List[str]:
+    if not isinstance(right, dict):
+        return []
     for r in right.values():
         if not isinstance(vars(r), dict):
             return ["Invalid datatype"]
diff --git a/tests/test_config_parser.py b/tests/test_config_parser.py
--- a/tests/test_config_parser.py
+++ b/tests/test_config_parser.py
@@ -132,6 +132,7 @@ def add_reference_config(self, job_name: str = DEFAULT_JOB_NAME) -> None:
             "qemu_headless": False,
             "qemu_smp": "2",
             "qemu_mem": "1G",
+            "qemu_kvm": mkosi.qemu_check_kvm_support(),
             "nspawn_keep_unit": False,
             "netdev": False,
             "ephemeral": False,
diff --git a/tests/test_machine.py b/tests/test_machine.py
--- a/tests/test_machine.py
+++ b/tests/test_machine.py
@@ -5,8 +5,8 @@
 
 import pytest
 
-from mkosi.backend import Verb
-from mkosi.machine import Machine, MkosiMachineTest, test_skip_not_supported
+from mkosi.backend import Distribution, Verb
+from mkosi.machine import Machine, MkosiMachineTest, skip_not_supported
 
 pytestmark = [
     pytest.mark.integration,
@@ -14,8 +14,15 @@
 ]
 
 class MkosiMachineTestCase(MkosiMachineTest):
+    @classmethod
+    def setUpClass(cls) -> None:
+        super().setUpClass()
+
+        if cls.machine.args.distribution == Distribution.centos_epel and cls.machine.args.verb == Verb.qemu and not cls.machine.args.qemu_kvm:
+            pytest.xfail("QEMU's CPU does not support the CentOS EPEL image arch when running without KVM")
+
     def test_simple_run(self) -> None:
-        process = self.machine.run(["echo", "This is a test."])
+        process = self.machine.run(["echo", "This is a test."], capture_output=True)
         assert process.stdout.strip("\n") == "This is a test."
 
     def test_wrong_command(self) -> None:
@@ -29,7 +36,7 @@ def test_wrong_command(self) -> None:
         result = self.machine.run(["NonExisting", "Command"], check=False)
         assert result.returncode in (1, 127, 203)
 
-        result = self.machine.run(["ls", "-"], check=False)
+        result = self.machine.run(["ls", "-"], check=False, capture_output=True)
         assert result.returncode == 2
         assert "No such file or directory" in result.stderr
 
@@ -39,7 +46,7 @@ def test_infinite_command(self) -> None:
 
 
 def test_before_boot() -> None:
-    with test_skip_not_supported():
+    with skip_not_supported():
         m = Machine()
 
     if m.args.verb == Verb.shell:
@@ -49,7 +56,7 @@ def test_before_boot() -> None:
 
 
 def test_after_shutdown() -> None:
-    with test_skip_not_supported():
+    with skip_not_supported():
         with Machine() as m:
             pass