Repository: systemd/mkosi

Issue:
Encrypted gpt_squashfs images are broken
Encrypted gpt_squashfs images are broken in a couple of different ways in the latest main (7d2e091). Errors occur with either `-d arch` or `-d fedora -r 33`.

`mkosi --encrypt=data --home-size=128M --distribution=fedora --release=33 --format=gpt_squashfs` fails after creating the partitions with `sfdisk`:

```
  ‣ Inserting generated root partition...
  ‣ Resizing disk image to 178.4M...
  ‣ Inserting partition of 48.4M...
  label: gpt
  /dev/loop2p1 : start=        2048, size=      262144, type=933AC7E1-2EB4-4F13-B844-0E14E2AEF915, uuid=94EDBFE9-9FC6-B041-A6CC-1F880748835E, name="Home Partition"
  size=99128, type=4f68bce3-e8cd-4db1-96e7-fbcaf984b709, attrs=GUID:60, name="Root Partition"

  GPT PMBR size mismatch (6557695 != 365367) will be corrected by write.
  Checking that no-one is using this disk right now ... OK

  Disk /dev/loop2: 178.4 MiB, 187068416 bytes, 365368 sectors
  Units: sectors of 1 * 512 = 512 bytes
  Sector size (logical/physical): 512 bytes / 512 bytes
  I/O size (minimum/optimal): 512 bytes / 512 bytes
  Disklabel type: dos
  Disk identifier: 0x00000000

  Old situation:

  Device       Boot Start    End Sectors   Size Id Type
  /dev/loop2p1          1 365367  365367 178.4M ee GPT

  >>> Script header accepted.
  >>> Created a new GPT disklabel (GUID: AA1AE954-181A-2047-8D69-707F981543C2).
  /dev/loop2p1: Created a new partition 1 of type 'Linux home' and of size 128 MiB.
  Partition #1 contains a crypto_LUKS signature.
  /dev/loop2p2: Created a new partition 2 of type 'Linux root (x86-64)' and of size 48.4 MiB.
  /dev/loop2p3: Done.

  New situation:
  Disklabel type: gpt
  Disk identifier: AA1AE954-181A-2047-8D69-707F981543C2

  Device        Start    End Sectors  Size Type
  /dev/loop2p1   2048 264191  262144  128M Linux home
  /dev/loop2p2 264192 363319   99128 48.4M Linux root (x86-64)

  The partition table has been altered.
  Calling ioctl() to re-read partition table.
  Re-reading the partition table failed.: Device or resource busy
  The kernel still uses the old table. The new table will be used at the next reboot or after you run partprobe(8) or partx(8).
  Syncing disks.
  ‣ Writing partition...
  dd: failed to open '/dev/loop2p2': No such file or directory
  ‣ Closing LUKS home partition...
  ‣ Detaching image file...
  Traceback (most recent call last):
    File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main
      return _run_code(code, main_globals, None,
    File "/usr/lib/python3.9/runpy.py", line 87, in _run_code
      exec(code, run_globals)
    File "/home/nick/code/ext/mkosi/mkosi/__main__.py", line 31, in <module>
      main()
    File "/home/nick/code/ext/mkosi/mkosi/__main__.py", line 25, in main
      run_verb(a)
    File "/home/nick/code/ext/mkosi/mkosi/__init__.py", line 6565, in run_verb
      build_stuff(args)
    File "/home/nick/code/ext/mkosi/mkosi/__init__.py", line 6077, in build_stuff
      raw, tar, root_hash, sshkey = build_image(args, root, do_run_build_script=False, cleanup=True)
    File "/home/nick/code/ext/mkosi/mkosi/__init__.py", line 5875, in build_image
      insert_generated_root(args, root, raw, loopdev, generated_root, for_cache)
    File "/home/nick/code/ext/mkosi/mkosi/__init__.py", line 3603, in insert_generated_root
      args.root_size = insert_partition(
    File "/home/nick/code/ext/mkosi/mkosi/__init__.py", line 3575, in insert_partition
      run(["dd", f"if={blob.name}", f"of={path}", "conv=nocreat"])
    File "/home/nick/code/ext/mkosi/mkosi/__init__.py", line 209, in run
      return subprocess.run(cmdline, check=check, stdout=stdout, stderr=stderr, **kwargs)
    File "/usr/lib/python3.9/subprocess.py", line 524, in run
      raise CalledProcessError(retcode, process.args,
  subprocess.CalledProcessError: Command '['dd', 'if=/var/lib/machines/key/.mkosi-squashfstrq6h9zt', 'of=/dev/loop2p2', 'conv=nocreat']' returned non-zero exit status 1.
```

`mkosi --encrypt=all --distribution=fedora --release=33 --format=gpt_squashfs` fails because mkosi never LUKS formats the root partition.

```
  ‣ Writing partition...
  ‣ Opening LUKS root partition...
  Device /dev/loop2p2 is not a valid LUKS device.
  ‣ Detaching image file...
  Traceback (most recent call last):
    File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main
      return _run_code(code, main_globals, None,
    File "/usr/lib/python3.9/runpy.py", line 87, in _run_code
      exec(code, run_globals)
    File "/home/nick/code/ext/mkosi/mkosi/__main__.py", line 31, in <module>
      main()
    File "/home/nick/code/ext/mkosi/mkosi/__main__.py", line 25, in main
      run_verb(a)
    File "/home/nick/code/ext/mkosi/mkosi/__init__.py", line 6565, in run_verb
      build_stuff(args)
    File "/home/nick/code/ext/mkosi/mkosi/__init__.py", line 6077, in build_stuff
      raw, tar, root_hash, sshkey = build_image(args, root, do_run_build_script=False, cleanup=True)
    File "/home/nick/code/ext/mkosi/mkosi/__init__.py", line 5875, in build_image
      insert_generated_root(args, root, raw, loopdev, generated_root, for_cache)
    File "/home/nick/code/ext/mkosi/mkosi/__init__.py", line 3603, in insert_generated_root
      args.root_size = insert_partition(
    File "/home/nick/code/ext/mkosi/mkosi/__init__.py", line 3569, in insert_partition
      dev = luks_setup_root(args, loopdev, False, True)
    File "/home/nick/code/ext/mkosi/mkosi/__init__.py", line 1101, in luks_setup_root
      return luks_open(partition(loopdev, args.root_partno), args.passphrase)
    File "/home/nick/code/ext/mkosi/mkosi/__init__.py", line 994, in luks_open
      run(["cryptsetup", "open", "--type", "luks", dev, name], input=passphrase_content)
    File "/home/nick/code/ext/mkosi/mkosi/__init__.py", line 209, in run
      return subprocess.run(cmdline, check=check, stdout=stdout, stderr=stderr, **kwargs)
    File "/usr/lib/python3.9/subprocess.py", line 524, in run
      raise CalledProcessError(retcode, process.args,
  subprocess.CalledProcessError: Command '['cryptsetup', 'open', '--type', 'luks', '/dev/loop2p2', '8cd2181f-aafe-4e2a-9e3c-83ea6e0e58b2']' returned non-zero exit status 1.
```

For this `--encrypt=all` issue, I noticed that [`luks_format_root`](https://github.com/systemd/mkosi/blob/7d2e0919fba1e7d86d5d7ae2565f2b679f5077cc/mkosi/__init__.py#L1015) has an optional argument `inserting_generated_root: bool = False` that is never used at either call site. Piping `True` to that argument from `insert_generated_root` through `insert_partition` gets us a little further, but then `dd` fails once it runs out of space.

Edits:

```diff
diff --git a/mkosi/__init__.py b/mkosi/__init__.py
index f7f01a8..603983f 100644
--- a/mkosi/__init__.py
+++ b/mkosi/__init__.py
@@ -3523,6 +3523,7 @@ def insert_partition(
     type_uuid: uuid.UUID,
     read_only: bool,
     uuid_opt: Optional[uuid.UUID] = None,
+    inserting_generated_root: bool = False,
 ) -> int:
     if args.ran_sfdisk:
         old_table, last_partition_sector = read_partition_table(loopdev)
@@ -3565,7 +3566,7 @@ def insert_partition(
     MkosiPrinter.print_step("Writing partition...")
 
     if args.root_partno == partno:
-        luks_format_root(args, loopdev, False, True)
+        luks_format_root(args, loopdev, False, False, inserting_generated_root)
         dev = luks_setup_root(args, loopdev, False, True)
     else:
         dev = None
@@ -3610,6 +3611,7 @@ def insert_generated_root(
             "Root Partition",
             gpt_root_native(args.architecture).root,
             args.output_format.is_squashfs(),
+            inserting_generated_root=True
         )
```

Result:
```
  ‣ Writing partition...
  ‣ LUKS formatting root partition...
  ‣ Opening LUKS root partition...
  dd: writing to '/dev/mapper/e344e9de-090e-4e78-81d0-7732b5522ba3': No space left on device
  595497+0 records in
  595496+0 records out
  304893952 bytes (305 MB, 291 MiB) copied, 1.67822 s, 182 MB/s
  ‣ Closing LUKS root partition...
  ‣ Detaching image file...
  Traceback (most recent call last):
    File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main
      return _run_code(code, main_globals, None,
    File "/usr/lib/python3.9/runpy.py", line 87, in _run_code
      exec(code, run_globals)
    File "/home/nick/code/ext/mkosi/mkosi/__main__.py", line 31, in <module>
      main()
    File "/home/nick/code/ext/mkosi/mkosi/__main__.py", line 25, in main
      run_verb(a)
    File "/home/nick/code/ext/mkosi/mkosi/__init__.py", line 6567, in run_verb
      build_stuff(args)
    File "/home/nick/code/ext/mkosi/mkosi/__init__.py", line 6079, in build_stuff
      raw, tar, root_hash, sshkey = build_image(args, root, do_run_build_script=False, cleanup=True)
    File "/home/nick/code/ext/mkosi/mkosi/__init__.py", line 5877, in build_image
      insert_generated_root(args, root, raw, loopdev, generated_root, for_cache)
    File "/home/nick/code/ext/mkosi/mkosi/__init__.py", line 3604, in insert_generated_root
      args.root_size = insert_partition(
    File "/home/nick/code/ext/mkosi/mkosi/__init__.py", line 3576, in insert_partition
      run(["dd", f"if={blob.name}", f"of={path}", "conv=nocreat"])
    File "/home/nick/code/ext/mkosi/mkosi/__init__.py", line 209, in run
      return subprocess.run(cmdline, check=check, stdout=stdout, stderr=stderr, **kwargs)
    File "/usr/lib/python3.9/subprocess.py", line 524, in run
      raise CalledProcessError(retcode, process.args,
  subprocess.CalledProcessError: Command '['dd', 'if=/var/lib/machines/key/.mkosi-squashfsyyayzfv2', 'of=/dev/mapper/e344e9de-090e-4e78-81d0-7732b5522ba3', 'conv=nocreat']' returned non-zero exit status 1.
```

I'm on an Arch host with systemd 247.3.1 if that's relevant. Please let me know if there are any other details I can provide.


Discussion:
Thank you for the detailed bug report! 

Just to make sure whether this is a recent regression or has been lurking without notice so far: Has this feature ever worked for you, e.g. in an older release?
Nope, this never worked for me, so I don't have any info to help you bisect. Sorry!

Resolving patch:
diff --git a/mkosi/__init__.py b/mkosi/__init__.py
--- a/mkosi/__init__.py
+++ b/mkosi/__init__.py
@@ -3665,7 +3665,7 @@ def insert_partition(
         last_partition_sector = GPT_HEADER_SIZE
 
     blob_size = roundup512(os.stat(blob.name).st_size)
-    luks_extra = 2 * 1024 * 1024 if args.encrypt == "all" else 0
+    luks_extra = 16 * 1024 * 1024 if args.encrypt == "all" else 0
     new_size = last_partition_sector + blob_size + luks_extra + GPT_FOOTER_SIZE
 
     MkosiPrinter.print_step(f"Resizing disk image to {format_bytes(new_size)}...")
@@ -3694,11 +3694,12 @@ def insert_partition(
 
     run(["sfdisk", "--color=never", loopdev], input=table.encode("utf-8"))
     run(["sync"])
+    run(["partx", "--update", loopdev])
 
     MkosiPrinter.print_step("Writing partition...")
 
     if args.root_partno == partno:
-        luks_format_root(args, loopdev, False, True)
+        luks_format_root(args, loopdev, False, False, True)
         dev = luks_setup_root(args, loopdev, False, True)
     else:
         dev = None


Test patch: