Repository: systemd/mkosi

Issue:
Allow building inside a container
It would be nice if mkosi could be launched inside a container to build an image.


Discussion:
well, that's hard, container managers generally don't support loopback devices as that's not safe, and that makes it very hard to do what we need to do...

What do you mean by "container" btw? docker?
Some kind of container runtime, docker, runc/crio.

Granted that loop devices are not easy, but maybe there is some other way.
This would be helpful when using mkosi in the context of a CI that uses containers (e.g. docker) to isolate builds - loopback devices should be accessible via running in priveleged mode.

There is a docker image containing the latest development version of mkosi here: https://github.com/42BV/docker-mkosi

However running an otherwise normal looking mkosi invocation results in error:

```
# sudo docker run --rm -i --privileged 42bv/mkosi:latest
[root@b1a811998513 mkosi]# mkosi
mkosi
Checking that no-one is using this disk right now ... OK

Disk /root/mkosi/.mkosi-we38uxya: 1 GiB, 1075838976 bytes, 2101248 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

>>> Script header accepted.
>>> Created a new GPT disklabel (GUID: 56EEC64D-06B7-C54C-9622-163AF9E6A859).
/root/mkosi/.mkosi-we38uxya1: Created a new partition 1 of type 'Linux root (x86-64)' and of size 1 GiB.
/root/mkosi/.mkosi-we38uxya2: Done.

New situation:
Disklabel type: gpt
Disk identifier: 56EEC64D-06B7-C54C-9622-163AF9E6A859

Device                       Start     End Sectors Size Type
/root/mkosi/.mkosi-we38uxya1  2048 2101214 2099167   1G Linux root (x86-64)

The partition table has been altered.
Syncing disks.
mke2fs 1.44.3 (10-July-2018)
The file /dev/loop0p1 does not exist and no size was specified.
DISTRIBUTION:
          Distribution: fedora
               Release: 29

OUTPUT:
         Output Format: gpt_ext4
                Output: /root/mkosi/image.raw
       Output Checksum: n/a
      Output Signature: n/a
           Output Bmap: n/a
Output nspawn Settings: n/a
           Incremental: no
             Read-only: no
        FS Compression: no
        XZ Compression: no
                 QCow2: no
            Encryption: no
                Verity: no
              Bootable: no

PACKAGES:
              Packages: none
    With Documentation: no
         Package Cache: none
           Extra Trees: none
        Skeleton Trees: none
          Build Script: none
         Build Sources: /root/mkosi
       Build Directory: none
        Build Packages: none
    Postinstall Script: none
       Finalize Script: none
  Scripts with network: no
       nspawn Settings: none

PARTITIONS:
        Root Partition: 1.0G
        Swap Partition: (disabled)
                   ESP: (disabled)
       /home Partition: (disabled)
        /srv Partition: (disabled)

VALIDATION:
              Checksum: no
                  Sign: no
               GPG Key: default
              Password: default

HOST CONFIGURATION:
    Extra search paths: none
‣ Detaching namespace...
‣ Detaching namespace complete.
‣ Setting up package cache...
‣ Setting up package cache /root/mkosi/.mkosi-u9bg57n5 complete.
‣ Setting up temporary workspace.
‣ Temporary workspace in /var/tmp/mkosi-97gvn68i is now set up.
‣ Creating partition table...
‣ Created partition table as /root/mkosi/.mkosi-we38uxya.
‣ Attaching image file...
‣ Attached image file as /dev/loop0.
‣ Formatting root partition...
‣ Detaching image file...
‣ Detaching image file complete.
Traceback (most recent call last):
  File "/usr/local/bin/mkosi", line 4, in <module>
    __import__('pkg_resources').run_script('mkosi==4', 'mkosi')
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 661, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1448, in run_script
    exec(script_code, namespace, namespace)
  File "/usr/local/lib/python3.7/site-packages/mkosi-4-py3.7.egg/EGG-INFO/scripts/mkosi", line 3906, in <module>
  File "/usr/local/lib/python3.7/site-packages/mkosi-4-py3.7.egg/EGG-INFO/scripts/mkosi", line 3896, in main
  File "/usr/local/lib/python3.7/site-packages/mkosi-4-py3.7.egg/EGG-INFO/scripts/mkosi", line 3741, in build_stuff
  File "/usr/local/lib/python3.7/site-packages/mkosi-4-py3.7.egg/EGG-INFO/scripts/mkosi", line 3576, in build_image
  File "/usr/local/lib/python3.7/site-packages/mkosi-4-py3.7.egg/EGG-INFO/scripts/mkosi", line 739, in prepare_root
  File "/usr/local/lib/python3.7/site-packages/mkosi-4-py3.7.egg/EGG-INFO/scripts/mkosi", line 591, in mkfs_ext4
  File "/usr/local/lib/python3.7/site-packages/mkosi-4-py3.7.egg/EGG-INFO/scripts/mkosi", line 72, in run
  File "/usr/lib64/python3.7/subprocess.py", line 481, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['mkfs.ext4', '-L', 'root', '-M', '/', '/dev/loop0p1']' returned non-zero exit status 1
```

I'm guessing, getting past this, mkosi's use of systemd-nspawn would require hacking to get working right as well.
Yep.  I was actually thinking to use it in order to build minimal images
for KubeVirt.

the linux kernel doesn't virtualize devices. All container environments that pretend they could pass through devices are pretty much lying to you. They can pass through individual device nodes, but generally devices coming and going, and they cannot provide /sys in correct fashion.

Now, if you try to use /dev/loop* inside of a docker container, then things fall apart as soon as you attach a file to it that contains a partition table, since that makes new block devices to pop up for the partitions, but the container payload will never see that...

Sorry, but containers and device access don't mix well. Some containre managers claim otherwise, but it's really broken all that.
When I try to build to a directory (mkosi -t directory) I get this errors:

```
Traceback (most recent call last):
  File "/usr/local/bin/mkosi", line 4, in <module>
    __import__('pkg_resources').run_script('mkosi==4', 'mkosi')
  File "/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line 741, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line 1509, in run_script
    exec(script_code, namespace, namespace)
  File "/usr/local/lib/python3.6/site-packages/mkosi-4-py3.6.egg/EGG-INFO/scripts/mkosi", line 3692, in <module>
  File "/usr/local/lib/python3.6/site-packages/mkosi-4-py3.6.egg/EGG-INFO/scripts/mkosi", line 3682, in main
  File "/usr/local/lib/python3.6/site-packages/mkosi-4-py3.6.egg/EGG-INFO/scripts/mkosi", line 3541, in build_stuff
  File "/usr/local/lib/python3.6/site-packages/mkosi-4-py3.6.egg/EGG-INFO/scripts/mkosi", line 3390, in build_image
  File "/usr/local/lib/python3.6/site-packages/mkosi-4-py3.6.egg/EGG-INFO/scripts/mkosi", line 1598, in install_distribution
  File "/usr/lib64/python3.6/contextlib.py", line 52, in inner
    return func(*args, **kwds)
  File "/usr/local/lib/python3.6/site-packages/mkosi-4-py3.6.egg/EGG-INFO/scripts/mkosi", line 1470, in install_arch
  File "/usr/local/lib/python3.6/site-packages/mkosi-4-py3.6.egg/EGG-INFO/scripts/mkosi", line 1415, in run_pacstrap
  File "/usr/lib64/python3.6/subprocess.py", line 418, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['pacstrap', '-C', '/home/nspuser/rootfs/.mkosi-q5v25lr9/pacman.conf', '-dGM', '/home/nspuser/rootfs/.mkosi-q5v25lr9/root', 'systemd-sysvcompat', 'util-linux', 'logrotate', 'findutils', 'glibc', 'which', 'nano', 'bash', 'texinfo', 'gettext', 'less', 'linux-firmware', 'iputils', 'grep', 'gzip', 'shadow', 'filesystem', 'procps-ng', 'psmisc', 'gcc-libs', 'pacman', 'pciutils', 'tar', 'diffutils', 'licenses', 'iproute2', 'file', 'perl', 'gawk', 'inetutils', 'usbutils', 'vi', 'bzip2', 'sysfsutils', 'coreutils', 's-nail', 'sed']' returned non-zero exit status 1.
```

I am using too Docker 42bv/mkosi:latest (Full log: https://travis-ci.org/localnet/nspawnci-mkosi)
just ran into this trying to get mkosi working with drone, our build system. it runs everything in docker by design, and mkosi is just super unhappy with it. however, not for device reasons, because drone allows setting containers as privileged per-job, and I can run `mount -t devtmpfs none /dev` inside the container to get updated device hooks.

where it gets hung up is in trying to do systemd-nspawn stuff after the image is built. systemd is of course not available inside the container, so this fails hard, even though I'm only building the image, not doing any of the execution stuff. I think if it was possible to run the build step without it trying to do anything systemd related, then it would work fine inside a (privileged) container.

is there something I've missed that will skip the systemd-nspawn registration after the container is built, since I don't need or want that step?

```
Traceback (most recent call last):
  File "/usr/bin/mkosi", line 4539, in <module>
    main()
  File "/usr/bin/mkosi", line 4535, in main
    run_verb(args)
  File "/usr/bin/mkosi", line 4507, in run_verb
    build_stuff(args)
  File "/usr/bin/mkosi", line 4333, in build_stuff
    raw, tar, root_hash = build_image(args, workspace, do_run_build_script=False, cleanup=True)
  File "/usr/bin/mkosi", line 4154, in build_image
    do_run_build_script=do_run_build_script, cached=cached)
  File "/usr/bin/mkosi", line 1987, in install_distribution
    install[args.distribution](args, workspace, do_run_build_script)
  File "/usr/lib/python3.7/contextlib.py", line 74, in inner
    return func(*args, **kwds)
  File "/usr/bin/mkosi", line 1717, in install_ubuntu
    install_debian_or_ubuntu(args, workspace, do_run_build_script=do_run_build_script, mirror=args.mirror)
  File "/usr/bin/mkosi", line 1706, in install_debian_or_ubuntu
    run_workspace_command(args, workspace, network=True, env=env, *cmdline)
  File "/usr/bin/mkosi", line 1181, in run_workspace_command
    run(cmdline, check=True)
  File "/usr/bin/mkosi", line 69, in run
    return subprocess.run(cmdline, **kwargs)
  File "/usr/lib/python3.7/subprocess.py", line 487, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['systemd-nspawn', '--quiet', '--directory=/var/tmp/mkosi-am6f_pl5/root', '--uuid=77e860ce4a24405fafddb5a07165dfd2', '--machine=mkosi-16afb08347f94174a378a4f3d94c2cef', '--as-pid2', '--register=no', '--bind=/var/tmp/mkosi-am6f_pl5/var-tmp:/var/tmp', '--setenv=SYSTEMD_OFFLINE=1', '--bind-ro=/etc/resolv.conf', '--setenv=DEBIAN_FRONTEND=noninteractive', '--setenv=DEBCONF_NONINTERACTIVE_SEEN=true', '--', '/usr/bin/apt-get', '--assume-yes', '--no-install-recommends', 'install', 'dbus', 'libpam-systemd', 'cloud-init', 'linux-aws', 'sudo', 'dracut', 'linux-generic', 'grub-pc']' returned non-zero exit status 100.
```

Resolving patch:
diff --git a/mkosi/__init__.py b/mkosi/__init__.py
--- a/mkosi/__init__.py
+++ b/mkosi/__init__.py
@@ -5589,6 +5589,11 @@ def create_parser() -> ArgumentParserMkosi:
     group.add_argument("--qemu-headless", action=BooleanAction, help="Configure image for qemu's -nographic mode")
     group.add_argument("--qemu-smp", help="Configure guest's SMP settings", metavar="SMP", default="2")
     group.add_argument("--qemu-mem", help="Configure guest's RAM size", metavar="MEM", default="1G")
+    group.add_argument(
+        "--nspawn-keep-unit",
+        action=BooleanAction,
+        help="If specified, underlying systemd-nspawn containers use the ressources of the current unit."
+    )
     group.add_argument(
         "--network-veth",
         dest="netdev",
@@ -7213,6 +7218,9 @@ def run_build_script(args: MkosiArgs, root: Path, raw: Optional[BinaryIO]) -> No
         if args.usr_only:
             cmdline += [f"--bind={root_home(args, root)}:/root"]
 
+        if args.nspawn_keep_unit:
+            cmdline += ["--keep-unit"]
+
         cmdline += [f"/root/{args.build_script.name}"]
         cmdline += args.cmdline
 
@@ -7474,6 +7482,9 @@ def run_shell_cmdline(args: MkosiArgs, pipe: bool = False, commands: Optional[Se
 
     cmdline += ["--machine", virt_name(args)]
 
+    if args.nspawn_keep_unit:
+        cmdline += ["--keep-unit"]
+
     if commands or args.cmdline:
         # If the verb is 'shell', args.cmdline contains the command to run.
         # Otherwise, the verb is 'boot', and we assume args.cmdline contains nspawn arguments.
diff --git a/mkosi/backend.py b/mkosi/backend.py
--- a/mkosi/backend.py
+++ b/mkosi/backend.py
@@ -502,6 +502,9 @@ class MkosiArgs:
     qemu_smp: str
     qemu_mem: str
 
+    # systemd-nspawn specific options
+    nspawn_keep_unit: bool
+
     # Some extra stuff that's stored in MkosiArgs for convenience but isn't populated by arguments
     machine_id_is_fixed: bool
     original_umask: int
@@ -646,6 +649,9 @@ def run_workspace_command(
         stdout = subprocess.PIPE
         nspawn += ["--console=pipe"]
 
+    if args.nspawn_keep_unit:
+        nspawn += ["--keep-unit"]
+
     result = run([*nspawn, "--", *cmd], check=False, stdout=stdout, text=capture_stdout)
     if result.returncode != 0:
         if "workspace-command" in ARG_DEBUG:


Test patch:
diff --git a/tests/test_config_parser.py b/tests/test_config_parser.py
--- a/tests/test_config_parser.py
+++ b/tests/test_config_parser.py
@@ -131,6 +131,7 @@ def add_reference_config(self, job_name: str = DEFAULT_JOB_NAME) -> None:
             "qemu_headless": False,
             "qemu_smp": "2",
             "qemu_mem": "1G",
+            "nspawn_keep_unit": False,
             "netdev": False,
             "ephemeral": False,
             "with_unified_kernel_images": True,