Repository: osbuild/osbuild
Pull request #1565 (2024-01-30 21:19:07)

support user-defined partition numbers for GPT disks
Partitions by default are indexed starting at 1, but in
some cases, such as CoreOS for IBM Z, it may be usefull
to set the 'partnum' for GPT disks explicitly, without
creating dummy partitions.

Now user can define an image:

```
    mpp-define-images:
      - id: image
        size: 10737418240
        table:
          uuid: 00000000-0000-4000-a000-000000000001
          label: gpt
          partitions:
            - name: boot
              type: 0FC63DAF-8483-4772-8E79-3D69D8477DE4
              partnum: 3
              size: 786432
            - name: root
              type: 0FC63DAF-8483-4772-8E79-3D69D8477DE4
              partnum: 4
              size: 4194304
```

So target disk would look like:

```
    Disklabel type: gpt
    Disk identifier: 00000000-0000-4000-A000-000000000001
    Device        Start     End Sectors  Size Type
    /dev/loop0p3   2048  788479  786432  384M Linux filesystem
    /dev/loop0p4 788480 4982783 4194304    2G Linux filesystem
```

This patch updates the osbuild-mpp tool and the sgdisk and sfdisk
stages to support this.

Co-authored-by: Dusty Mabe <dusty@dustymabe.com>                 


dustymabe: This is a continuation of https://github.com/osbuild/osbuild/pull/1555
dustymabe: > I don't see anything checking for duplicate partition numbers. Maybe it's ok to let it fail 'naturally' :)

:) 

Yeah. You won't get far if it's invalid. Usually the tool that generates the manifest would be able to catch this. `osbuild-mpp` will fail at least.

> But it seems like a possible problem area. Also it looks like there could be duplicates in org.osbuild.sgdisk if there is a mix of entries with and without 'partition' field where it uses `i+1` if it is missing.

yeah.. I'm not opposed to adding some checking if you see easy ways/places to do it, but I don't think it's worth spending a lot of time on because the tool will just error when we get there. 


dustymabe: rebased to pick up latest changes (some CI fixes have been applied to `main`).

@bcl - want to add a review? maybe @mvo5 too?

Patch:
diff --git a/README.md b/README.md
index 96da8d34..eae4ea9f 100644
--- a/README.md
+++ b/README.md
@@ -12,11 +12,11 @@ of the pipeline description, and more.
 
 ## Project
 
- * **Website**: <https://www.osbuild.org>
- * **Bug Tracker**: <https://github.com/osbuild/osbuild/issues>
+ * **Website**: https://www.osbuild.org
+ * **Bug Tracker**: https://github.com/osbuild/osbuild/issues
  * **Matrix**: #image-builder on [fedoraproject.org](https://matrix.to/#/#image-builder:fedoraproject.org)
  * **Mailing List**: image-builder@redhat.com
- * **Changelog**: <https://github.com/osbuild/osbuild/releases>
+ * **Changelog**: https://github.com/osbuild/osbuild/releases
 
 ### Contributing
 
@@ -82,7 +82,7 @@ sudo dnf install ./rpmbuild/RPMS/noarch/*.rpm
 
 ## Repository
 
- - **web**:   <https://github.com/osbuild/osbuild>
+ - **web**:   https://github.com/osbuild/osbuild
  - **https**: `https://github.com/osbuild/osbuild.git`
  - **ssh**:   `git@github.com:osbuild/osbuild.git`
 
diff --git a/osbuild/util/fscache.py b/osbuild/util/fscache.py
index c8fc99aa..59039522 100644
--- a/osbuild/util/fscache.py
+++ b/osbuild/util/fscache.py
@@ -15,7 +15,7 @@ import json
 import os
 import subprocess
 import uuid
-from typing import Any, Dict, NamedTuple, Optional, Tuple, Union
+from typing import Any, Dict, List, NamedTuple, Optional, Tuple, Union
 
 from osbuild.util import ctx, linux, rmrf
 
@@ -101,6 +101,17 @@ class FsCacheInfo(NamedTuple):
         return data
 
 
+class FsCacheObjectInfo(NamedTuple):
+    """ File System Cache object information
+
+    This type represents information about a single cache object. The
+    last_used information is only guaranteed to be valid while the cache
+    is locked.
+    """
+    name: str
+    last_used: float
+
+
 class FsCache(contextlib.AbstractContextManager, os.PathLike):
     """File System Cache
 
@@ -672,8 +683,11 @@ class FsCache(contextlib.AbstractContextManager, os.PathLike):
             self._info_maximum_size = -1
         elif isinstance(info.maximum_size, int):
             self._info_maximum_size = info.maximum_size
-        else:
+        elif info.maximum_size is None:
             self._info_maximum_size = 0
+        else:
+            raise ValueError(
+                f"maximum-size can only be set to 'unlimited' or an integer value, got {type(info.maximum_size)}")
 
     def _is_active(self):
         # Internal helper to verify we are in an active context-manager.
@@ -931,19 +945,27 @@ class FsCache(contextlib.AbstractContextManager, os.PathLike):
             info["creation-boot-id"] = self._bootid
             info["size"] = self._calculate_space(path_data)
 
-            # Update the total cache-size. If it exceeds the limits, bail out
-            # but do not trigger an error. It behaves as if the entry was
-            # committed and immediately deleted by racing cache management. No
-            # need to tell the caller about it (if that is ever needed, we can
-            # provide for it).
+            # Exit early if it never is going to fit
+            if self._info_maximum_size > -1 and info["size"] > self._info_maximum_size:
+                return
+
+            # Update the total cache-size. If it exceeds the limits, remove
+            # least recently used objects until there is enough space.
             #
             # Note that if we crash after updating the total cache size, but
             # before committing the object information, the total cache size
-            # will be out of sync. However, it is never overcommitted, so we
-            # will never violate any cache invariants. The cache-size will be
-            # re-synchronized by any full cache-management operation.
+            # will be out of sync.
+            #
+            # However, it is never overcommitted, so we will never
+            # violate any cache invariants. Future code needs to resync
+            # the cache (e.g. on open with some simple journal strategy).
             if not self._update_cache_size(info["size"]):
-                return
+                # try to free space
+                self._remove_lru(info["size"])
+                # and see if the update can happen now
+                if not self._update_cache_size(info["size"]):
+                    # stil could not free enough space
+                    return
 
             try:
                 # Commit the object-information, thus marking it as fully
@@ -1059,6 +1081,85 @@ class FsCache(contextlib.AbstractContextManager, os.PathLike):
                 raise self.MissError() from None
             raise e
 
+    def _last_used_objs(self) -> List[FsCacheObjectInfo]:
+        """Return a list of FsCacheObjectInfo with name, last_used
+           information sorted by last_used time.
+
+        Note that this function will be racy when used without a lock and
+        the caller needs to handle this.
+        """
+        objs = []
+        for name in os.listdir(self._path(self._dirname_objects)):
+            try:
+                last_used = self._last_used(name)
+            except (OSError, FsCache.MissError):
+                continue
+            objs.append(FsCacheObjectInfo(name=name, last_used=last_used))
+        return sorted(objs, key=lambda obj: obj.last_used)
+
+    def _remove_lru(self, required_size):
+        """"
+        Make room in the cache for "required_size" by remove the least
+        recently used entry from the cache. Note that the cache may
+        clear more than required_size.
+        """
+        # To avoid having to take a global cache lock the strategy is:
+        # 1. Get list of (object, last_used) sorted from oldest to newest.
+        #    This is racy so we need to take care of that in step(2).
+        # 2. Start with the oldest entry, try to take a write_lock
+        #    (with O_NOATIME to be extra sure that atime information is
+        #    correct). Get the "last_used" (atime) time and compare to what
+        #    we expect in the list. If it diverges the object got load()ed
+        #    while we iterated. Skip it and go to (2) again.
+        # 3. Remove entry, update cache size after the entry is removed.
+        #
+        # Note that there is a risk to get out-of-sync in (3). If the
+        # process dies while removing and before updating the cache
+        # size the cache will be over reported.
+
+        # Try to clean at least twice the requested size to avoid having
+        # to do this all over again
+        try_to_free = required_size * 2
+        freed_so_far = 0
+        for name, last_used in self._last_used_objs():
+            # take write lock for the indivdual object
+            rpath = os.path.join(self._dirname_objects, name)
+            rpath_lock = os.path.join(rpath, self._filename_object_lock)
+            # Ideally there would some lock helper instead of the low-level
+            # file manipulation to abstract this a bit more.
+            try:
+                with self._atomic_open(
+                        rpath_lock,
+                        wait=False,
+                        write=True,
+                        # atime carries the "last-used" data so don't alter it
+                        oflags=os.O_EXCL | os.O_NOATIME,
+                ):
+                    if last_used != self._last_used(name):
+                        continue
+                    # This is racy right now if the process is killed
+                    # during "_rm_r_object(rpath)" because then the
+                    # cache size is never reduced by the amount that
+                    # was about to be deleted.
+                    #
+                    # To fix it we need to (atomic) rename the
+                    # "object.info" file in _rm_r_object() to
+                    # something like "object.removing". Then when
+                    # opening the cache scan for leftover
+                    # "object.removing" files and finish the cleanup
+                    # and update the cache size based on the size
+                    # recorded inside "object.removing".
+                    size = self._calculate_space(self._path(rpath))
+                    self._rm_r_object(rpath)
+                    self._update_cache_size(-size)
+                    freed_so_far += size
+                    if freed_so_far >= try_to_free:
+                        break
+            except BlockingIOError:
+                continue
+        # return True if at least the required size got freed
+        return freed_so_far > required_size
+
     @property
     def info(self) -> FsCacheInfo:
         """Query Cache Information
diff --git a/stages/org.osbuild.kickstart b/stages/org.osbuild.kickstart
index 03d30cf1..2153af72 100755
--- a/stages/org.osbuild.kickstart
+++ b/stages/org.osbuild.kickstart
@@ -223,6 +223,37 @@ SCHEMA = r"""
       }
     }
   },
+  "rootpw": {
+    "type": "object",
+    "anyOf":[
+      {"required": ["lock"], "not": {"required": ["allow_ssh"]}},
+      {"required": ["iscrypted", "password"], "not": {"required": ["plaintext"]}},
+      {"required": ["plaintext", "password"], "not": {"required": ["iscrypted"]}}
+    ],
+    "properties": {
+      "lock": {
+        "type": "boolean",
+        "description": "the root account is locked by default"
+      },
+      "plaintext": {
+         "type": "boolean",
+         "description": "the password argument is assumed to be in plain text"
+      },
+      "iscrypted": {
+         "type": "boolean",
+         "description": "the password argument is assumed to already be encrypted"
+      },
+      "allow_ssh": {
+         "type": "boolean",
+         "description": "This will allow remote root logins via ssh using only the password"
+      },
+      "password": {
+         "type": "string",
+         "description": "the root password",
+         "minLength": 1
+      }
+    }
+  },
   "lang": {
     "type": "string",
     "description": "The language code (e.g. en_US.UTF-8)"
@@ -254,6 +285,10 @@ SCHEMA = r"""
         "description": "Erases all partitions from the system",
         "type": "boolean"
       },
+      "initlabel": {
+        "description": "Initializes a disk (or disks) by creating a default disk label for all disks",
+        "type": "boolean"
+      },
       "drives": {
         "description": "Specifies which drives to clear partitions from",
         "type": "array",
@@ -363,6 +398,10 @@ SCHEMA = r"""
       "pbkdf-iterations": {
         "description": "Sets the number of iterations for passphrase processing directly",
         "type": "integer"
+      },
+      "nohome": {
+        "description": "Disables automatic creation of the /home partition",
+        "type": "boolean"
       }
     }
   },
@@ -438,6 +477,19 @@ def make_users(users: Dict) -> List[str]:
     return res
 
 
+def make_rootpw(rootpw: Dict) -> str:
+    arguments = []
+    for option in ["lock", "plaintext", "iscrypted", "allow_ssh", "password"]:
+        option_value = rootpw.get(option)
+        if option_value is True:
+            arguments.append(f"--{option.replace('_', '-')}")
+        elif isinstance(option_value, str) and option_value:
+            arguments.append(option_value)
+    if arguments:
+        return f"rootpw {' '.join(arguments)}"
+    return ""
+
+
 def make_clearpart(options: Dict) -> str:
     clearpart = options.get("clearpart")
     if clearpart is None:
@@ -458,6 +510,9 @@ def make_clearpart(options: Dict) -> str:
     linux = clearpart.get("linux", False)
     if linux:
         cmd += " --linux"
+    initlabel = clearpart.get("initlabel", False)
+    if initlabel:
+        cmd += " --initlabel"
     return cmd
 
 
@@ -481,7 +536,7 @@ def make_autopart(options: Dict) -> str:
     cmd = "autopart"
     for key in ["type", "fstype", "nolvm", "encrypted", "passphrase",
                 "escrowcert", "backuppassphrase", "cipher", "luks-version",
-                "pbkdf", "pbkdf-memory", "pbkdf-time", "pbkdf-iterations"]:
+                "pbkdf", "pbkdf-memory", "pbkdf-time", "pbkdf-iterations", "nohome"]:
         if key not in autopart:
             continue
         val = autopart[key]
@@ -568,6 +623,9 @@ def main(tree, options):  # pylint: disable=too-many-branches
 
     config += make_groups(options.get("groups", {}))
     config += make_users(options.get("users", {}))
+    rootpw_command = make_rootpw(options.get("rootpw", {}))
+    if rootpw_command:
+        config += [rootpw_command]
 
     lang = options.get("lang")
     if lang:
diff --git a/stages/org.osbuild.sfdisk b/stages/org.osbuild.sfdisk
index 9887a02e..108866c4 100755
--- a/stages/org.osbuild.sfdisk
+++ b/stages/org.osbuild.sfdisk
@@ -4,6 +4,7 @@ Partition a target using sfdisk(8)
 """
 
 import json
+import re
 import subprocess
 import sys
 from typing import Optional
@@ -50,6 +51,10 @@ SCHEMA_2 = r"""
             "description": "The partition name (GPT)",
             "type": "string"
           },
+          "partnum": {
+            "description": "The partition number",
+            "type": "integer"
+          },
           "size": {
             "description": "The size of this partition",
             "type": "integer"
@@ -88,6 +93,7 @@ SCHEMA_2 = r"""
 class Partition:
     def __init__(self,
                  pttype: str = None,
+                 partnum: int = None,
                  start: int = None,
                  size: int = None,
                  bootable: bool = False,
@@ -95,13 +101,15 @@ class Partition:
                  uuid: str = None,
                  attrs: int = None):
         self.type = pttype
+        self.partnum = partnum
         self.start = start
         self.size = size
         self.bootable = bootable
         self.name = name
         self.uuid = uuid
-        self.index = None
         self.attrs = attrs
+        self.index = partnum - 1 if partnum else None
+        self.partnum = partnum if partnum else None
 
     @property
     def start_in_bytes(self):
@@ -167,7 +175,10 @@ class PartitionTable:
                 fields += [f'{field}="{value}"']
             if partition.bootable:
                 fields += ["bootable"]
-            command += "\n" + ", ".join(fields)
+            if partition.partnum:
+                command += "\n" + f'{target}p{partition.partnum}: ' + ", ".join(fields)
+            else:
+                command += "\n" + ", ".join(fields)
 
         print(command)
 
@@ -190,7 +201,8 @@ class PartitionTable:
 
         assert len(disk_parts) == len(self.partitions)
         for i, part in enumerate(self.partitions):
-            part.index = i
+            part.partnum = int(re.findall(r'\d+$', disk_parts[i]["node"])[0])
+            part.index = part.partnum - 1
             part.start = disk_parts[i]["start"]
             part.size = disk_parts[i]["size"]
             part.type = disk_parts[i].get("type")
@@ -200,6 +212,7 @@ class PartitionTable:
 
 def partition_from_json(js) -> Partition:
     p = Partition(pttype=js.get("type"),
+                  partnum=js.get("partnum"),
                   start=js.get("start"),
                   size=js.get("size"),
                   bootable=js.get("bootable"),
diff --git a/stages/org.osbuild.sgdisk b/stages/org.osbuild.sgdisk
index e11e0119..f85a762a 100755
--- a/stages/org.osbuild.sgdisk
+++ b/stages/org.osbuild.sgdisk
@@ -35,6 +35,11 @@ SCHEMA_2 = r"""
       "type": "boolean",
       "default": true
     },
+    "label": {
+      "description": "The type of the partition table. Only 'gpt' supported.",
+      "type": "string",
+      "enum": ["gpt"]
+    },
     "partitions": {
       "description": "Partition layout ",
       "type": "array",
@@ -50,6 +55,10 @@ SCHEMA_2 = r"""
             "description": "The partition name",
             "type": "string"
           },
+          "partnum": {
+            "description": "The partition number",
+            "type": "integer"
+          },
           "size": {
             "description": "The size of this partition",
             "type": "integer"
@@ -88,6 +97,7 @@ SCHEMA_2 = r"""
 class Partition:
     def __init__(self,
                  pttype: str = None,
+                 partnum: int = None,
                  start: int = None,
                  size: int = None,
                  bootable: bool = False,
@@ -95,6 +105,7 @@ class Partition:
                  uuid: str = None,
                  attrs: int = None):
         self.type = pttype
+        self.partnum = partnum
         self.start = start
         self.size = size
         self.name = name
@@ -129,7 +140,7 @@ class PartitionTable:
             command += ["-U", self.uuid]
 
         for i, part in enumerate(self.partitions):
-            idx = i + 1  # partitions are 1-indexed
+            idx = part.partnum if part.partnum else i + 1  # partitions are 1-indexed
 
             # format is 'partnum:start:end'
             size = "0"
@@ -173,6 +184,7 @@ class PartitionTable:
 
 def partition_from_json(js) -> Partition:
     p = Partition(pttype=js.get("type"),
+                  partnum=js.get("partnum"),
                   start=js.get("start"),
                   size=js.get("size"),
                   bootable=js.get("bootable"),
diff --git a/stages/org.osbuild.tar b/stages/org.osbuild.tar
index f42ccd9d..4068c551 100755
--- a/stages/org.osbuild.tar
+++ b/stages/org.osbuild.tar
@@ -26,6 +26,25 @@ CAPABILITIES = ["CAP_MAC_ADMIN"]
 SCHEMA_2 = """
 "options": {
   "additionalProperties": false,
+  "anyOf": [
+    {
+      "required": ["root-node"],
+      "not": {
+        "required": ["paths"]
+      }
+    },
+    {
+      "required": ["paths"],
+      "not": {
+        "required": ["root-node"]
+      }
+    },
+    {
+      "not": {
+        "required": ["paths", "root-node"]
+      }
+    }
+  ],
   "required": ["filename"],
   "properties": {
     "filename": {
@@ -54,8 +73,15 @@ SCHEMA_2 = """
       "default": true
     },
     "root-node": {
-      "description": "How to handle the root node: include or omit",
+      "description": "How to handle the root node: include or omit, mutually exclusive with paths",
       "enum": ["include", "omit"]
+    },
+    "paths": {
+      "type": "array",
+      "items": {
+        "type": "string",
+        "description": "If specified, the archive will only contain the specified paths, in the specified order"
+      }
     }
   }
 },
@@ -78,6 +104,7 @@ def main(inputs, output_dir, options):
     filename = options["filename"].lstrip("/")
     tarfmt = options.get("format", "gnu")
     root_node = options.get("root-node", "include")
+    paths = options.get("paths", [])
 
     extra_args = []
     # Set environment variables for the tar operation.
@@ -106,13 +133,16 @@ def main(inputs, output_dir, options):
         "-C", tree,
     ]
 
-    # Should we include the root inode or not
-    # This will change the way files are listed:
-    # `./file` (include) vs `file` (omit)
-    if root_node == "include":
-        tar_cmd += ["."]
+    if len(paths) == 0:
+        # Should we include the root inode or not
+        # This will change the way files are listed:
+        # `./file` (include) vs `file` (omit)
+        if root_node == "include":
+            tar_cmd += ["."]
+        else:
+            tar_cmd += os.listdir(tree)
     else:
-        tar_cmd += os.listdir(tree)
+        tar_cmd += paths
 
     # Make a tarball of the tree.
     subprocess.run(
diff --git a/tools/osbuild-mpp b/tools/osbuild-mpp
index cf74488e..7bf93a18 100755
--- a/tools/osbuild-mpp
+++ b/tools/osbuild-mpp
@@ -357,6 +357,7 @@ import hashlib
 import json
 import os
 import pathlib
+import re
 import string
 import subprocess
 import sys
@@ -815,6 +816,7 @@ class Partition:
     def __init__(self,
                  uid: str = None,
                  pttype: str = None,
+                 partnum: int = None,
                  start: int = None,
                  size: int = None,
                  bootable: bool = False,
@@ -830,8 +832,8 @@ class Partition:
         self.name = name
         self.uuid = uuid
         self.attrs = attrs
-        self.index = None
-        self.partnum = None
+        self.index = partnum - 1 if partnum else None
+        self.partnum = partnum if partnum else None
 
     @property
     def start_in_bytes(self):
@@ -845,6 +847,7 @@ class Partition:
     def from_dict(cls, js):
         p = cls(uid=js.get("id"),
                 pttype=js.get("type"),
+                partnum=js.get("partnum"),
                 start=js.get("start"),
                 size=js.get("size"),
                 bootable=js.get("bootable"),
@@ -858,6 +861,8 @@ class Partition:
 
         if self.start:
             data["start"] = self.start
+        if self.partnum:
+            data["partnum"] = self.partnum
         if self.size:
             data["size"] = self.size
         if self.type:
@@ -915,7 +920,10 @@ class PartitionTable:
                 fields += [f'{field}="{value}"']
             if partition.bootable:
                 fields += ["bootable"]
-            command += "\n" + ", ".join(fields)
+            if partition.partnum:
+                command += "\n" + f'{target}p{partition.partnum}: ' + ", ".join(fields)
+            else:
+                command += "\n" + ", ".join(fields)
 
         subprocess.run(["sfdisk", "-q", "--no-tell-kernel", target],
                        input=command,
@@ -936,8 +944,8 @@ class PartitionTable:
 
         assert len(disk_parts) == len(self.partitions)
         for i, part in enumerate(self.partitions):
-            part.index = i
-            part.partnum = i + 1
+            part.partnum = int(re.findall(r'\d+$', disk_parts[i]["node"])[0])
+            part.index = part.partnum - 1
             part.start = disk_parts[i]["start"]
             part.size = disk_parts[i]["size"]
             part.type = disk_parts[i].get("type")


Test patch:
diff --git a/stages/test/test_kickstart.py b/stages/test/test_kickstart.py
index d4aab3a8..03ee9ee7 100644
--- a/stages/test/test_kickstart.py
+++ b/stages/test/test_kickstart.py
@@ -1,6 +1,7 @@
 #!/usr/bin/python3
 
 import os.path
+import re
 import subprocess
 
 import pytest
@@ -20,6 +21,21 @@ TEST_INPUT = [
         },
         "lang en_US.UTF-8\nkeyboard us\ntimezone UTC",
     ),
+    ({"rootpw": {"lock": True}}, "rootpw --lock"),
+    ({"rootpw": {"plaintext": True, "password": "plaintext-password"}}, "rootpw --plaintext plaintext-password"),
+    ({"rootpw": {"iscrypted": True, "password": "encrypted-password"}}, "rootpw --iscrypted encrypted-password"),
+    (
+        {"rootpw": {"iscrypted": True, "allow_ssh": True, "password": "encrypted-password"}},
+        "rootpw --iscrypted --allow-ssh encrypted-password",
+    ),
+    (
+        {"rootpw": {"plaintext": True, "allow_ssh": True, "password": "plaintext-password"}},
+        "rootpw --plaintext --allow-ssh plaintext-password",
+    ),
+    (
+        {"rootpw": {"plaintext": True, "lock": True, "password": "plaintext-password"}},
+        "rootpw --lock --plaintext plaintext-password",
+    ),
     (
         {
             "ostree": {
@@ -59,6 +75,7 @@ TEST_INPUT = [
     ),
     ({"zerombr": True}, "zerombr"),
     ({"clearpart": {"all": True}}, "clearpart --all"),
+    ({"clearpart": {"all": True, "initlabel": True}}, "clearpart --all --initlabel"),
     (
         {"clearpart": {"drives": ["sd*|hd*|vda", "/dev/vdc"]}},
         "clearpart --drives=sd*|hd*|vda,/dev/vdc",
@@ -68,6 +85,7 @@ TEST_INPUT = [
         {"clearpart": {"drives": ["disk/by-id/scsi-58095BEC5510947BE8C0360F604351918"]}},
         "clearpart --drives=disk/by-id/scsi-58095BEC5510947BE8C0360F604351918"
     ),
+    ({"clearpart": {"drives": ["hda"], "initlabel": True}}, "clearpart --drives=hda --initlabel"),
     ({"clearpart": {"list": ["sda2", "sda3"]}}, "clearpart --list=sda2,sda3"),
     ({"clearpart": {"list": ["sda2"]}}, "clearpart --list=sda2"),
     (
@@ -121,6 +139,8 @@ TEST_INPUT = [
     ({"autopart": {"pbkdf-memory": 64}}, "autopart --pbkdf-memory=64"),
     ({"autopart": {"pbkdf-time": 128}}, "autopart --pbkdf-time=128"),
     ({"autopart": {"pbkdf-iterations": 256}}, "autopart --pbkdf-iterations=256"),
+    ({"autopart": {"nohome": True}}, "autopart --nohome"),
+    ({"autopart": {"type": "plain", "fstype": "xfs", "nohome": True}}, "autopart --type=plain --fstype=xfs --nohome"),
     ({
         "lang": "en_US.UTF-8",
         "keyboard": "us",
@@ -346,7 +366,17 @@ def test_kickstart_valid(tmp_path, stage_module, test_input, expected):  # pylin
             },
             "is not valid under any of the given schemas",
         ),
-
+        ({"rootpw": {}}, "is not valid under any of the given schemas"),
+        ({"rootpw": {"lock": True, "allow_ssh": True}}, "is not valid under any of the given schemas"),
+        ({"rootpw": {"plaintext": True}}, "is not valid under any of the given schemas"),
+        # under py3.6 the message is "'' is too short" under other versions the message is "'' should be non-empty"
+        ({"rootpw": {"plaintext": True, "password": ""}}, re.compile("'' should be non-empty|'' is too short")),
+        ({"rootpw": {"iscrypted": True}}, "is not valid under any of the given schemas"),
+        ({"rootpw": {"password": "password"}}, "is not valid under any of the given schemas"),
+        (
+            {"rootpw": {"iscrypted": True, "plaintext": True, "password": "pass"}},
+            "is not valid under any of the given schemas"
+        ),
     ],
 )
 @pytest.mark.parametrize("stage_schema", ["1"], indirect=True)
diff --git a/stages/test/test_tar.py b/stages/test/test_tar.py
new file mode 100644
index 00000000..6a988a69
--- /dev/null
+++ b/stages/test/test_tar.py
@@ -0,0 +1,100 @@
+#!/usr/bin/python3
+
+import os.path
+import subprocess
+
+import pytest
+
+from osbuild import testutil
+from osbuild.testutil import has_executable, make_fake_input_tree
+
+STAGE_NAME = "org.osbuild.tar"
+
+
+@pytest.mark.parametrize("test_data,expected_err", [
+    # bad
+    ({}, "'filename' is a required property"),
+    ({
+        "filename": "out.tar",
+        "root-node": "include",
+        "paths": ["file1"]
+    }, "{'filename': 'out.tar', 'root-node': 'include', 'paths': ['file1']} is not valid under any of the given schemas"),
+    # good
+    ({"filename": "out.tar", "root-node": "include"}, ""),
+    ({"filename": "out.tar", "paths": ["file1"]}, ""),
+    ({"filename": "out.tar"}, ""),
+])
+def test_schema_validation_tar(stage_schema, test_data, expected_err):
+    test_input = {
+        "type": STAGE_NAME,
+        "options": {
+        }
+    }
+    test_input["options"].update(test_data)
+    res = stage_schema.validate(test_input)
+
+    if expected_err == "":
+        assert res.valid is True, f"err: {[e.as_dict() for e in res.errors]}"
+    else:
+        assert res.valid is False
+        testutil.assert_jsonschema_error_contains(res, expected_err, expected_num_errs=1)
+
+
+@pytest.fixture(name="fake_inputs")
+def make_fake_inputs(tmp_path):
+    fake_input_tree = make_fake_input_tree(tmp_path, {
+        "/file1": "A file to be tarred",
+        "/file2": "Another file to be tarred",
+    })
+    inputs = {
+        "tree": {
+            "path": fake_input_tree,
+        }
+    }
+    return inputs
+
+
+@pytest.mark.skipif(not has_executable("tar"), reason="no tar executable")
+def test_tar_filename(tmp_path, stage_module, fake_inputs):
+    options = {
+        "filename": "out.tar",
+    }
+    stage_module.main(fake_inputs, tmp_path, options)
+
+    tar_path = os.path.join(tmp_path, "out.tar")
+    assert os.path.exists(tar_path)
+    output = subprocess.check_output(["tar", "-tf", tar_path], encoding="utf-8").strip().split("\n")
+    assert sorted(["./", "./file1", "./file2"]) == sorted(output)
+
+
+@pytest.mark.skipif(not has_executable("tar"), reason="no tar executable")
+def test_tar_root_node_omit(tmp_path, stage_module, fake_inputs):
+    options = {
+        "filename": "out.tar",
+        "root-node": "omit",
+    }
+    stage_module.main(fake_inputs, tmp_path, options)
+
+    tar_path = os.path.join(tmp_path, "out.tar")
+    assert os.path.exists(tar_path)
+    output = subprocess.check_output(["tar", "-tf", tar_path], encoding="utf-8").strip().split("\n")
+    assert sorted(["file1", "file2"]) == sorted(output)
+
+
+@pytest.mark.skipif(not has_executable("tar"), reason="no tar executable")
+def test_tar_paths(tmp_path, stage_module, fake_inputs):
+    options = {
+        "filename": "out.tar",
+        "paths": [
+            "file2",
+            "file1",
+            "file1",
+            "file2",
+        ]
+    }
+    stage_module.main(fake_inputs, tmp_path, options)
+
+    tar_path = os.path.join(tmp_path, "out.tar")
+    assert os.path.exists(tar_path)
+    output = subprocess.check_output(["tar", "-tf", tar_path], encoding="utf-8").split("\n")
+    assert ["file2", "file1", "file1", "file2", ""] == output
diff --git a/test/data/manifests/fedora-coreos-container.json b/test/data/manifests/fedora-coreos-container.json
index fb56b861..ba1f0ad4 100644
--- a/test/data/manifests/fedora-coreos-container.json
+++ b/test/data/manifests/fedora-coreos-container.json
@@ -560,6 +560,7 @@
             "partitions": [
               {
                 "start": 2048,
+                "partnum": 1,
                 "size": 2048,
                 "type": "21686148-6449-6E6F-744E-656564454649",
                 "bootable": true,
@@ -567,18 +568,21 @@
               },
               {
                 "start": 4096,
+                "partnum": 2,
                 "size": 260096,
                 "type": "C12A7328-F81F-11D2-BA4B-00A0C93EC93B",
                 "name": "EFI-SYSTEM"
               },
               {
                 "start": 264192,
+                "partnum": 3,
                 "size": 786432,
                 "type": "0FC63DAF-8483-4772-8E79-3D69D8477DE4",
                 "name": "boot"
               },
               {
                 "start": 1050624,
+                "partnum": 4,
                 "size": 4194304,
                 "type": "0FC63DAF-8483-4772-8E79-3D69D8477DE4",
                 "name": "root"
@@ -806,6 +810,7 @@
             "partitions": [
               {
                 "start": 256,
+                "partnum": 1,
                 "size": 256,
                 "type": "21686148-6449-6E6F-744E-656564454649",
                 "bootable": true,
@@ -813,18 +818,21 @@
               },
               {
                 "start": 512,
+                "partnum": 2,
                 "size": 32512,
                 "type": "C12A7328-F81F-11D2-BA4B-00A0C93EC93B",
                 "name": "EFI-SYSTEM"
               },
               {
                 "start": 33024,
+                "partnum": 3,
                 "size": 98304,
                 "type": "0FC63DAF-8483-4772-8E79-3D69D8477DE4",
                 "name": "boot"
               },
               {
                 "start": 131328,
+                "partnum": 4,
                 "size": 524288,
                 "type": "0FC63DAF-8483-4772-8E79-3D69D8477DE4",
                 "name": "root"
diff --git a/test/data/manifests/fedora-coreos-container.mpp.yaml b/test/data/manifests/fedora-coreos-container.mpp.yaml
index 2494b867..d6f220d3 100644
--- a/test/data/manifests/fedora-coreos-container.mpp.yaml
+++ b/test/data/manifests/fedora-coreos-container.mpp.yaml
@@ -31,18 +31,22 @@ mpp-define-images:
           bootable: true
           size:
             mpp-format-int: "{bios_boot_size_mb * 1024 * 1024 / sector_size}"
+          partnum: 1
         - name: EFI-SYSTEM
           type: C12A7328-F81F-11D2-BA4B-00A0C93EC93B
           size:
             mpp-format-int: "{efi_system_size_mb * 1024 * 1024 / sector_size}"
+          partnum: 2
         - name: boot
           type: 0FC63DAF-8483-4772-8E79-3D69D8477DE4
           size:
             mpp-format-int: "{boot_size_mb * 1024 * 1024 / sector_size}"
+          partnum: 3
         - name: root
           type: 0FC63DAF-8483-4772-8E79-3D69D8477DE4
           size:
             mpp-format-int: "{root_size_mb * 1024 * 1024 / sector_size}"
+          partnum: 4
   - id: image4k
     sector_size:
         mpp-format-int: "{four_k_sector_size}"
diff --git a/test/data/manifests/fedora-ostree-image.json b/test/data/manifests/fedora-ostree-image.json
index 55c8d179..917d4ce6 100644
--- a/test/data/manifests/fedora-ostree-image.json
+++ b/test/data/manifests/fedora-ostree-image.json
@@ -1216,7 +1216,7 @@
           }
         },
         {
-          "type": "org.osbuild.sfdisk",
+          "type": "org.osbuild.sgdisk",
           "devices": {
             "device": {
               "type": "org.osbuild.loopback",
@@ -1231,6 +1231,7 @@
             "partitions": [
               {
                 "start": 2048,
+                "partnum": 1,
                 "size": 2048,
                 "type": "21686148-6449-6E6F-744E-656564454649",
                 "bootable": true,
@@ -1238,18 +1239,21 @@
               },
               {
                 "start": 4096,
+                "partnum": 2,
                 "size": 204800,
                 "type": "C12A7328-F81F-11D2-BA4B-00A0C93EC93B",
                 "uuid": "68B2905B-DF3E-4FB3-80FA-49D1E773AA33"
               },
               {
                 "start": 208896,
+                "partnum": 3,
                 "size": 204800,
                 "type": "0FC63DAF-8483-4772-8E79-3D69D8477DE4",
                 "uuid": "61B2905B-DF3E-4FB3-80FA-49D1E773AA32"
               },
               {
                 "start": 413696,
+                "partnum": 4,
                 "size": 20555776,
                 "type": "0FC63DAF-8483-4772-8E79-3D69D8477DE4",
                 "uuid": "CA7D7CCB-63ED-4C53-861C-1742536059CC"
diff --git a/test/data/manifests/fedora-ostree-image.mpp.yaml b/test/data/manifests/fedora-ostree-image.mpp.yaml
index 52bf80dd..ce0235e8 100644
--- a/test/data/manifests/fedora-ostree-image.mpp.yaml
+++ b/test/data/manifests/fedora-ostree-image.mpp.yaml
@@ -11,17 +11,21 @@ mpp-define-image:
         type: 21686148-6449-6E6F-744E-656564454649
         bootable: true
         uuid: FAC7F1FB-3E8D-4137-A512-961DE09A5549
+        partnum: 1
       - id: efi
         size: 204800
         type: C12A7328-F81F-11D2-BA4B-00A0C93EC93B
         uuid: 68B2905B-DF3E-4FB3-80FA-49D1E773AA33
+        partnum: 2
       - id: boot
         size: 204800
         type: 0FC63DAF-8483-4772-8E79-3D69D8477DE4
         uuid: 61B2905B-DF3E-4FB3-80FA-49D1E773AA32
+        partnum: 3
       - id: luks
         type: 0FC63DAF-8483-4772-8E79-3D69D8477DE4
         uuid: CA7D7CCB-63ED-4C53-861C-1742536059CC
+        partnum: 4
 pipelines:
   - mpp-import-pipelines:
       path: fedora-vars.ipp.yaml
@@ -139,7 +143,7 @@ pipelines:
           filename: disk.img
           size:
             mpp-format-string: '{image.size}'
-      - type: org.osbuild.sfdisk
+      - type: org.osbuild.sgdisk
         devices:
           device:
             type: org.osbuild.loopback
diff --git a/test/mod/test_util_fscache.py b/test/mod/test_util_fscache.py
index 97713ecf..b515d957 100644
--- a/test/mod/test_util_fscache.py
+++ b/test/mod/test_util_fscache.py
@@ -465,36 +465,124 @@ def test_cache_load_updates_last_used_on_noatime(tmp_path):
         test_cache_load_updates_last_used(mnt_path)
 
 
+@pytest.mark.skipif(not has_precise_fs_timestamps(), reason="need precise fs timestamps")
 def test_cache_full_behavior(tmp_path):
+    def _cache_size_from_file():
+        with open(cache._path(cache._filename_cache_size), encoding="utf8") as fp:
+            return json.load(fp)
+
     cache = fscache.FsCache("osbuild-cache-evict", tmp_path)
+    with cache:
+        # use big sizes to mask the effect of {dirs,cache.info} using
+        # 4k of space too
+        obj_size = 128 * 1024
+        # cache is big enough to hold 4 objects (add buffer)
+        max_cache_size = (4 * obj_size) + int(0.5 * obj_size)
+        cache.info = cache.info._replace(maximum_size=max_cache_size)
+        # add 4 object to the store, we are below the limit
+        for i in range(1, 5):
+            with cache.store(f"o{i}") as rpath:
+                rpath = os.path.join(tmp_path, rpath, f"f{i}")
+                with open(rpath, "wb") as fp:
+                    fp.write(b'a' * obj_size)
+            # info file is updated
+            assert _cache_size_from_file() >= i * obj_size
+            assert _cache_size_from_file() < (i + 1) * obj_size
+            # disk is updated
+            assert cache._calculate_space(tmp_path) >= i * obj_size
+            assert cache._calculate_space(tmp_path) < (i + 1) * obj_size
+            with cache.load(f"o{i}") as o:
+                assert o != ""
+            sleep_for_fs()
+        # adding one more
+        with cache.store("o-full") as rpath:
+            rpath = os.path.join(tmp_path, rpath, "f-full")
+            with open(rpath, "wb") as fp:
+                fp.write(b'b' * obj_size)
+        # cache file is updated, it will free twice the size of the
+        # requested obj
+        assert _cache_size_from_file() >= 2 * obj_size
+        assert _cache_size_from_file() < max_cache_size
+        # disk is updated
+        assert cache._calculate_space(tmp_path) >= 3 * obj_size
+        assert cache._calculate_space(tmp_path) < max_cache_size
+        # o1,o2 is least recently used and got removed
+        for obj in ["o1", "o2"]:
+            with pytest.raises(fscache.FsCache.MissError):
+                with cache.load(obj) as o:
+                    pass
+        # and o-full made it in
+        for obj in ["o3", "o4", "o-full"]:
+            with cache.load(obj) as o:
+                pass
+
+
+@pytest.mark.skipif(not has_precise_fs_timestamps(), reason="need precise fs timestamps")
+def test_cache_last_used_objs(tmpdir):
+    cache = fscache.FsCache("osbuild-cache-id", tmpdir)
     with cache:
         # use big sizes to mask the effect of dirs using 4k of space too
-        cache.info = cache.info._replace(maximum_size=192 * 1024)
-        # add one object to the store, we are below the limit
+        cache.info = cache.info._replace(maximum_size=256 * 1024)
+        # add objs to the store
+        for obj in ["o3", "o2", "o1"]:
+            with cache.store(obj):
+                pass
+            with cache.load(obj):
+                pass
+            sleep_for_fs()
+        sorted_objs = cache._last_used_objs()
+        assert [e[0] for e in sorted_objs] == ["o3", "o2", "o1"]
+        # access o2
+        with cache.load("o2"):
+            pass
+        sorted_objs = cache._last_used_objs()
+        assert [e[0] for e in sorted_objs] == ["o3", "o1", "o2"]
+
+
+@pytest.mark.skipif(not has_precise_fs_timestamps(), reason="need precise fs timestamps")
+def test_cache_remove_lru(tmpdir):
+    cache = fscache.FsCache("osbuild-cache-id", tmpdir)
+    with cache:
+        cache.info = cache.info._replace(maximum_size=-1)
+        # add objs to the store
+        for obj in ["o3", "o2", "o1"]:
+            with cache.store(obj):
+                pass
+            with cache.load(obj):
+                pass
+            sleep_for_fs()
+        # precondition check: we have least used o3,o2,o1
+        sorted_objs = cache._last_used_objs()
+        assert [e[0] for e in sorted_objs] == ["o3", "o2", "o1"]
+        # removed least recently used (o3), now o2 is least recently used
+        cache._remove_lru(1)
+        sorted_objs = cache._last_used_objs()
+        assert [e[0] for e in sorted_objs] == ["o2", "o1"]
+        # now load o2 (previously least recently used)
+        with cache.load("o2"):
+            pass
+        sleep_for_fs()
+        # and ensure that removing the lru removes "o1" now and keeps "o2"
+        cache._remove_lru(1)
+        sorted_objs = cache._last_used_objs()
+        assert [e[0] for e in sorted_objs] == ["o2"]
+        # removing last obj
+        cache._remove_lru(1)
+        sorted_objs = cache._last_used_objs()
+        assert sorted_objs == []
+        # and keep removing is fine
+        cache._remove_lru(1)
+        assert sorted_objs == []
+
+
+def test_cache_obj_too_big(tmp_path):
+    cache = fscache.FsCache("osbuild-cache-evict", tmp_path)
+    with cache:
+        max_cache_size = 16 * 1024
+        cache.info = cache.info._replace(maximum_size=max_cache_size)
         with cache.store("o1") as rpath:
-            rpath_f1 = os.path.join(tmp_path, rpath, "f1")
-            with open(rpath_f1, "wb") as fp:
-                fp.write(b'a' * 64 * 1024)
-        assert cache._calculate_space(tmp_path) > 64 * 1024
-        assert cache._calculate_space(tmp_path) < 128 * 1024
-        with cache.load("o1") as o:
-            assert o != ""
-        # and one more
-        with cache.store("o2") as rpath:
-            rpath_f2 = os.path.join(tmp_path, rpath, "f2")
-            with open(rpath_f2, "wb") as fp:
-                fp.write(b'b' * 64 * 1024)
-        assert cache._calculate_space(tmp_path) > 128 * 1024
-        assert cache._calculate_space(tmp_path) < 192 * 1024
-        with cache.load("o2") as o:
-            assert o != ""
-        # adding a third one will (silently) fail because the cache is full
-        with cache.store("o3") as rpath:
-            rpath_f3 = os.path.join(tmp_path, rpath, "f3")
-            with open(rpath_f3, "wb") as fp:
-                fp.write(b'b' * 128 * 1024)
-        assert cache._calculate_space(tmp_path) > 128 * 1024
-        assert cache._calculate_space(tmp_path) < 192 * 1024
+            with open(os.path.join(tmp_path, rpath, "f1"), "wb") as fp:
+                fp.write(b'a' * 2 * max_cache_size)
         with pytest.raises(fscache.FsCache.MissError):
-            with cache.load("o3") as o:
+            with cache.load("o1"):
                 pass