Repository: osbuild/osbuild
Pull request #1983 (2025-01-20 11:49:43)

Manifest tests: replace `manifest-db` with `images` CI cache (HMS-5360)
Replace `manifest-db`-based manifest tests with `images`-CI-cache-based manifest tests.

The manifest-db was hard to maintain and keep up to date. It also adds one more indirection to the workflow of adding a new distro version to test in the osbuild CI.

In `osbuild/images` CI, we generate manifests for all distros, arches, image types, and blueprint customizations. These manifests are then built using a pinned osbuild version. The built image artifact is then uploaded with its metadata and manifest to AWS S3 cache. As a natural consequence, the osbuild version used in `images` CI will always be older than the one tested in this repository on the `main` branch or in a PR, which is important for manifest tests.

Manifest tests aim to verify that osbuild image building behavior does not change unexpectedly. This is assured by building a specific manifest with an older and known-to-be-good osbuild version and rebuilding the same manifest with the osbuild version under test. The behavior consistency is assessed by comparing `osbuild-image-info` reports generated for both images. The expectation is that there should not be any differences (well, some UUIDs do differ but are ignored).

So the intention of this new script is to take advantage of the osbuild/images S3 cache. As part of the test case (for a specific distro / arch / image_type / config):
 - download the manifest from S3
 - download the image artifact, built from the manifest, from S3
 - generate image-info report for the downloaded image
 - rebuild the downloaded manifest using current version of osbuild
 - generate image-info report for the rebuilt image
 - compare the two image-info reports. If there is no difference, the
   test case PASS, otherwise it will FAIL.

For now, ignore image types known to fail with `osbuild-image-info` and test only manifests for the `empty` and `all-customizations` configurations.

This PR also includes a fix for how `osbuild-image-info` handles images with LVM layout. Previously, it would create a directory with the LVM VG name in the current working directory and then create device nodes inside it for each of the LVs on the VG. However, it never cleaned them up, so the result was that analyzing two images with a different LVM layout failed if done in the same CWD after each other. The LVM handling has been refactored and cleaned up to rely purely on the implementation of the osbuild devices, which managed the devices under `/dev` the whole time.

Depends on https://github.com/osbuild/images/pull/1155

/jira-epic COMPOSER-2318

JIRA: [HMS-5360](https://issues.redhat.com/browse/HMS-5360)

thozza: > This is great! Only a few nitpicks below.

Thanks for the review :partying_face: 

> One general remark: The chunking logic is a bit strange. I realise it's something we had in the previous scripts but I didn't like it there either. It doesn't really give much control over what will be run but it's useful for running builds in parallel. Maybe we could replace it with a `--jobs N` option that runs N jobs in parallel and then test selection can be handled by the new filters that select based on distro, image type, and config.

Yes, it is useful only for splitting test cases into chunks of the same size to run them in parallel.

If we decide to explicitly specify image types that will be tested, we won't need to specify any chunks since the GitLab CI testing matrix will take care of parallelizing the runs. The "chunking logic" enables us not to explicitly list image types (which may differ per each distro), and we will automatically start testing new image types as they are added in `images`. Anyway, the options to do this are implemented in the script, so it is a matter of how we execute it.

As for the `--jobs N`, I'm not sure how the specific run of the script would determine its order in the number of jobs to determine which subset of the test cases to run. :thinking: 

I think the best solution would be to generate dynamic Gitlab pipelines, similar to how it is done in `images`, but I didn't want to spend more time on this now. The ideal workflow would be for each distro to run a "generator" job to determine all test cases to run and then create dynamic pipelines for each test case.

From my PoV, the current implementation is the same as what we had, from a parallelization point of view, but it gets us rid of `manifest-db` and allows us to iterate on it and make it better in follow-ups.


achilleas-k: > As for the `--jobs N`, I'm not sure how the specific run of the script would determine its order in the number of jobs to determine which subset of the test cases to run. 🤔
> 

You're right.  I was just thinking about parallel jobs/builds, not splitting them across runners.

> I think the best solution would be to generate dynamic Gitlab pipelines, similar to how it is done in `images`, but I didn't want to spend more time on this now. The ideal workflow would be for each distro to run a "generator" job to determine all test cases to run and then create dynamic pipelines for each test case.
> 

Yup, that'd be nice.  Not for this PR though, I agree.

> From my PoV, the current implementation is the same as what we had, from a parallelization point of view, but it gets us rid of `manifest-db` and allows us to iterate on it and make it better in follow-ups.

Agreed.  Let's keep it as is.  We can revisit in the future if we do dynamic pipelines.


Patch:
diff --git a/.github/workflows/propagate_to_manifestdb.yml b/.github/workflows/propagate_to_manifestdb.yml
deleted file mode 100644
index b941b364..00000000
--- a/.github/workflows/propagate_to_manifestdb.yml
+++ /dev/null
@@ -1,78 +0,0 @@
-name: Update osbuild ref in manifest-db
-
-on:
-  push:
-    branches:
-      - main
-
-jobs:
-  propagate:
-    runs-on: ubuntu-latest
-    env:
-      SCHUTZBOT_GH: ${{ secrets.SCHUTZBOT_GITHUB_ACCESS_TOKEN }}
-    steps:
-      - name: Checkout OSBuild
-        uses: actions/checkout@v3
-        with:
-          path: osbuild
-          fetch-depth: 0
-
-      - name: Update Manifest-db
-        run: |
-          # update only if manifest-db SHA was updated in the last commit
-          cd osbuild
-          ACTUAL_MDB_SHA=$(git show HEAD:Schutzfile | jq -r '.global.dependencies."manifest-db".commit')
-          PREVIOUS_MDB_SHA=$(git show HEAD^:Schutzfile | jq -r '.global.dependencies."manifest-db".commit')
-          if [[ $ACTUAL_MDB_SHA == $PREVIOUS_MDB_SHA ]] ; then
-            echo "The manifest-db SHA was not updated in the last commit, ignoring the dependency update"
-            exit 0
-          fi
-          echo "updating the dependency in manifest-db"
-          OSBUILD_HEAD="${{ github.sha }}"
-          echo "Latest OSBuild SHA: $OSBUILD_HEAD"
-
-          # Clone manifest-db
-          cd ..
-          git clone https://github.com/osbuild/manifest-db.git
-
-          # Load the previous osbuild REF from manifest-db's Schutzfile and
-          # Generate the commit list between the two osbuild revisions for the
-          # PR
-          cd manifest-db
-          OLD_REF=$(jq -r '.global.dependencies.osbuild.commit' Schutzfile)
-          cd ../osbuild
-          COMMIT_LIST=$(git log --oneline $OLD_REF..$OSBUILD_HEAD | sed 's/.*/- https:\/\/github.com\/osbuild\/osbuild\/commit\/&/')
-
-          # Update the manifest-db's dependency to the newest OSBuild
-          #
-          # Login as Schutzbot
-          cd ../manifest-db
-          echo "${SCHUTZBOT_GH}" | gh auth login --with-token
-          git config --local user.name "SchutzBot"
-          git config --local user.email "imagebuilder-bots+schutzbot@redhat.com"
-
-          # Create a branch for the PR
-          now=$(date '+%Y-%m-%d-%H%M%S')
-          BRANCH_NAME="osbuild-update-$now"
-          git checkout -b $BRANCH_NAME
-
-          # change the value for the commit head in the schutzfile
-          jq --arg variable "$OSBUILD_HEAD" '.global.dependencies.osbuild.commit=$variable' Schutzfile > Schutzfile.tmp && mv Schutzfile.tmp Schutzfile
-
-          # create the PR
-          PR_BODY="$(cat <<-END
-          This PR updates the osbuild ref dependency for manifest-db. Between the
-          last time it was updated, and this new reference commit, these are the changes:
-
-          $COMMIT_LIST
-          END
-          )"
-          git remote add upstream https://schutzbot:"$SCHUTZBOT_GH"@github.com/schutzbot/manifest-db.git
-          git add -A && \
-              git commit -m "schutzfile: update osbuild ref $(date '+%Y-%m-%d')" && \
-              git push upstream "$BRANCH_NAME:$BRANCH_NAME" && \
-              gh pr create \
-                  --title "schutzfile: update osbuild ref $(date '+%Y-%m-%d')" \
-                  --body "$PR_BODY" \
-                  --repo "osbuild/manifest-db" \
-                  -r lavocatt
diff --git a/.github/workflows/update-images.yml b/.github/workflows/update-images.yml
new file mode 100644
index 00000000..d69b7f1e
--- /dev/null
+++ b/.github/workflows/update-images.yml
@@ -0,0 +1,52 @@
+# This action updates the images ref in the Schutzfile
+---
+    name: "Update images ref"
+    
+    on:
+      workflow_dispatch:
+      schedule:
+        # Every Mon at 8:00
+        - cron: "0 8 * * 1"
+    
+    jobs:
+      update-and-push:
+        runs-on: ubuntu-latest
+        steps:
+          - name: Apt update
+            run: sudo apt update
+    
+          - name: Check out main
+            uses: actions/checkout@v4
+            with:
+              path: osbuild
+              ref: main
+    
+          - name: Update Schutzfile
+            working-directory: ./osbuild
+            env:
+              GITHUB_TOKEN: ${{ secrets.SCHUTZBOT_GITHUB_ACCESS_TOKEN }}
+            run: |
+              ./schutzbot/update-schutzfile-images
+    
+          - name: Open PR
+            working-directory: ./osbuild
+            env:
+              GITHUB_TOKEN: ${{ secrets.SCHUTZBOT_GITHUB_ACCESS_TOKEN }}
+            run: |
+              if git diff --exit-code; then echo "No changes"; exit 0; fi
+              git config --unset-all http.https://github.com/.extraheader
+              git config user.name "schutzbot"
+              git config user.email "schutzbot@gmail.com"
+              branch="schutzfile-images-$(date -I)"
+              git checkout -b "${branch}"
+              git add Schutzfile
+              git commit -m "Schutzfile: Update images dependency ref to latest"
+              git push -f https://"$GITHUB_TOKEN"@github.com/schutzbot/osbuild.git
+              echo "Updating images dependency ref to current `main`" > body
+              gh pr create \
+                -t "Update images dependency ref to latest" \
+                -F "body" \
+                -r "osbuild/osbuild-reviewers" \
+                --repo "osbuild/osbuild" \
+                --base "main" \
+                --head "schutzbot:${branch}"
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 2961ed0c..af4600dc 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -88,21 +88,58 @@ Manifests:
   extends: .terraform
   script:
     - schutzbot/deploy.sh
-    - test/cases/manifest_tests.sh ${PARALLEL_EXEC}
+    - schutzbot/manifest-tests-install-deps.sh
+    - IMAGES_REPO_REF=$(jq -r '.global.dependencies.images.ref' Schutzfile)
+    - |
+        if [[ -n "$CHUNKS_COUNT" && -n "$CHUNK" ]]; then
+            echo "Running manifest tests with $CHUNKS_COUNT chunks, chunk $CHUNK"
+            SCRIPT_EXTRA_ARGS="--chunk-size $CHUNK $CHUNKS_COUNT"
+        fi
+        if [[ -n "$UNSUPPORTED_IMG_TYPES" ]]; then
+            echo "Skipping unsupported image types: $UNSUPPORTED_IMG_TYPES"
+            SCRIPT_EXTRA_ARGS="$SCRIPT_EXTRA_ARGS --skip-image-type $UNSUPPORTED_IMG_TYPES"
+        fi
+        if [[ -n "$CONFIG" ]]; then
+            echo "Testing only the following manifest generation configs: $CONFIG"
+            SCRIPT_EXTRA_ARGS="$SCRIPT_EXTRA_ARGS --config $CONFIG"
+        fi
+        if [[ -n "$IMAGES_REPO_REF" ]]; then
+            echo "Using osbuild/images repo ref: $IMAGES_REPO_REF"
+            SCRIPT_EXTRA_ARGS="$SCRIPT_EXTRA_ARGS --images-ref $IMAGES_REPO_REF"
+        fi
+        if [[ -n "$SCRIPT_EXTRA_ARGS" ]]; then
+            echo "Running manifest tests with extra args: $SCRIPT_EXTRA_ARGS"
+        fi
+    - |
+        source /etc/os-release
+        if [[ "$ID" == "rhel" || "$ID" == "centos" ]]; then
+            export GOFLAGS="-tags=exclude_graphdriver_btrfs"
+        fi
+    - test/cases/manifest_tests --rm-artifacts-after-test --workdir ./osbuild-manifest-tests-workdir $SCRIPT_EXTRA_ARGS
+  variables:
+    PYTHONUNBUFFERED: 1
+    # List of image types that are not supported by osbuild-image-info
+    UNSUPPORTED_IMG_TYPES: "ova *container live-installer wsl"
+  artifacts:
+    when: always
+    paths:
+      - ci-details-before-run
+      - ci-details-after-run
+      - ./osbuild-manifest-tests-results/
   parallel:
     matrix:
-      - PARALLEL_EXEC: ["1/8", "2/8", "3/8", "4/8", "5/8", "6/8", "7/8", "8/8"]
-        RUNNER:
-          - aws/fedora-39-x86_64
-          - aws/fedora-39-aarch64
+      - RUNNER:
+          - aws/fedora-41-x86_64
+          - aws/fedora-41-aarch64
           - aws/centos-stream-9-x86_64
           - aws/centos-stream-9-aarch64
           - aws/centos-stream-10-x86_64
           - aws/centos-stream-10-aarch64
-      - PARALLEL_EXEC: ["1/8", "2/8", "3/8", "4/8", "5/8", "6/8", "7/8", "8/8"]
-        RUNNER:
-          - aws/rhel-8.10-ga-x86_64
-          - aws/rhel-8.10-ga-aarch64
+        # Test only the following manifest generation configs
+        CONFIG: "empty all-customizations"
+        CHUNK: [1, 2, 3, 4, 5]
+        CHUNKS_COUNT: 5
+      - RUNNER:
           - aws/rhel-9.4-ga-x86_64
           - aws/rhel-9.4-ga-aarch64
           - aws/rhel-9.6-nightly-x86_64
@@ -110,6 +147,10 @@ Manifests:
           - aws/rhel-10.0-nightly-x86_64
           - aws/rhel-10.0-nightly-aarch64
         INTERNAL_NETWORK: "true"
+        # Test only the following manifest generation configs
+        CONFIG: "empty all-customizations"
+        CHUNK: [1, 2, 3, 4, 5]
+        CHUNKS_COUNT: 5
 
 finish:
   stage: finish
diff --git a/Schutzfile b/Schutzfile
index 9451d875..9ab5b948 100644
--- a/Schutzfile
+++ b/Schutzfile
@@ -1,8 +1,8 @@
 {
   "global": {
     "dependencies": {
-      "manifest-db": {
-        "commit": "8bf14138336dfcebe6486068b5856cae2489c632"
+      "images": {
+        "ref": "main"
       }
     }
   },
diff --git a/osbuild/monitor.py b/osbuild/monitor.py
index d0f117fa..9ece10f3 100644
--- a/osbuild/monitor.py
+++ b/osbuild/monitor.py
@@ -16,6 +16,7 @@ import json
 import os
 import sys
 import time
+from threading import Lock
 from typing import Dict, Optional, Set, Union
 
 import osbuild
@@ -234,6 +235,7 @@ class BaseMonitor(abc.ABC):
     def result(self, result: Union[BuildResult, DownloadResult]) -> None:
         """Called when a module (stage/assembler) is done with its result"""
 
+    # note that this should be re-entrant
     def log(self, message: str, origin: Optional[str] = None):
         """Called for all module log outputs"""
 
@@ -316,6 +318,7 @@ class JSONSeqMonitor(BaseMonitor):
         self._ctx_ids: Set[str] = set()
         self._progress = Progress("pipelines/sources", total_steps)
         self._context = Context(origin="org.osbuild")
+        self._jsonseq_mu = Lock()
 
     def begin(self, pipeline: osbuild.Pipeline):
         self._context.set_pipeline(pipeline)
@@ -362,10 +365,11 @@ class JSONSeqMonitor(BaseMonitor):
         ))
 
     def _jsonseq(self, entry: dict) -> None:
-        # follow rfc7464 (application/json-seq)
-        self.out.write("\x1e")
-        json.dump(entry, self.out)
-        self.out.write("\n")
+        with self._jsonseq_mu:
+            # follow rfc7464 (application/json-seq)
+            self.out.write("\x1e")
+            json.dump(entry, self.out)
+            self.out.write("\n")
 
 
 def make(name: str, fd: int, total_steps: int) -> BaseMonitor:
diff --git a/schutzbot/update-schutzfile-images b/schutzbot/update-schutzfile-images
new file mode 100755
index 00000000..214beca8
--- /dev/null
+++ b/schutzbot/update-schutzfile-images
@@ -0,0 +1,46 @@
+#!/usr/bin/env python3
+import json
+import os
+import sys
+import urllib.request
+
+SCHUTZFILE = "Schutzfile"
+
+
+def images_main_commit_id():
+    token = os.environ.get("GITHUB_TOKEN")
+    req = urllib.request.Request("https://api.github.com/repos/osbuild/images/commits/main")
+    req.add_header("Accept", "application/vnd.github+json")
+    if token:
+        # this API request doesn't necessarily require a token, but let's use it if we have one
+        req.add_header("Authorization", f"Bearer {token}")
+    try:
+        with urllib.request.urlopen(req, timeout=30) as resp:
+            body = resp.read()
+    except urllib.error.HTTPError as http_error:
+        print(http_error)
+        sys.exit(1)
+
+    data = json.loads(body)
+    return data["sha"]
+
+
+def update_images_ref(new):
+    with open(SCHUTZFILE, encoding="utf-8") as schutzfile:
+        data = json.load(schutzfile)
+
+    data.setdefault("global", {}).setdefault("dependencies", {}).setdefault("images", {})["ref"] = new
+
+    with open(SCHUTZFILE, encoding="utf-8", mode="w") as schutzfile:
+        json.dump(data, schutzfile, indent="  ")
+
+
+def main():
+    main_id = images_main_commit_id()
+    print(f"osbuild/images main commit ID: {main_id}")
+    print("Updating Schutzfile")
+    update_images_ref(main_id)
+
+
+if __name__ == "__main__":
+    main()
diff --git a/tools/osbuild-image-info b/tools/osbuild-image-info
index e4b9910c..c66f7e8a 100755
--- a/tools/osbuild-image-info
+++ b/tools/osbuild-image-info
@@ -19,7 +19,7 @@ import tempfile
 import time
 import xml.etree.ElementTree
 from collections import OrderedDict
-from typing import Any, Dict
+from typing import Any, Dict, List
 
 import jsonschema
 import yaml
@@ -41,28 +41,52 @@ def run_ostree(*args, _input=None, _check=True, **kwargs):
     return res
 
 
-def loop_open(devmgr: devices.DeviceManager, name: str, image, size, offset=0):
+class OSBuildDeviceManager(devices.DeviceManager):
     """
-    Uses a DeviceManager to open the `name` at `offset`.
-    Returns a Device object and the path onto which the image was loopback mounted.
+    Thin wrapper around the DeviceManager for opening devices
     """
-    info = index.get_module_info("Device", "org.osbuild.loopback")
-    fname = os.path.basename(image)
-    options = {
-        "filename": fname,
-        "start": offset // SECTOR_SIZE,
-        "size": size // SECTOR_SIZE
-    }
-    if not info:
-        raise RuntimeError("Can't load org.osbuild.loopback")
 
-    jsonschema.validate(options, info.get_schema())
-    dev = devices.Device(name, info, None, options)
-    reply = devmgr.open(dev)
-    return {
-        "Device": dev,
-        "path": os.path.join("/dev", reply["path"])
-    }
+    def open_loopback(self, name, image, size, offset=0) -> Dict[str, Any]:
+        """
+        Uses a DeviceManager to open the `name` at `offset`.
+        Returns a Device object and the path onto which the image was loopback mounted.
+        """
+        info = index.get_module_info("Device", "org.osbuild.loopback")
+        if not info:
+            raise RuntimeError("Can't load org.osbuild.loopback")
+
+        fname = os.path.basename(image)
+        options = {
+            "filename": fname,
+            "start": offset // SECTOR_SIZE,
+            "size": size // SECTOR_SIZE
+        }
+        jsonschema.validate(options, info.get_schema())
+        dev = devices.Device(name, info, None, options)
+        reply = self.open(dev)
+        return {
+            "Device": dev,
+            "path": os.path.join(self.devpath, reply["path"])
+        }
+
+    def open_lvm_lv(self, lv_name: str, parent: devices.Device):
+        """
+        Open a logical volume and return the path to the device node
+        """
+        info = index.get_module_info("Device", "org.osbuild.lvm2.lv")
+        if not info:
+            raise RuntimeError("can't find org.osbuild.lvm2.lv")
+
+        options = {
+            "volume": lv_name,
+        }
+        jsonschema.validate(options, info.get_schema())
+        dev = devices.Device(lv_name, info, parent, options)
+        reply = self.open(dev)
+        return {
+            "Device": dev,
+            "path": os.path.join(self.devpath, reply["path"])
+        }
 
 
 @contextlib.contextmanager
@@ -2453,8 +2477,10 @@ def append_filesystem(report, tree, *, is_ostree=False):
         print("EFI partition", file=sys.stderr)
 
 
-def volume_group_for_device(device: str) -> str:
-    # Find the volume group that belongs to the device specified via `parent`
+def lvm_vg_for_device(device: str) -> str:
+    """
+    Find the volume group name for the specified device.
+    """
     vg_name = None
     count = 0
 
@@ -2471,7 +2497,7 @@ def volume_group_for_device(device: str) -> str:
 
         if res.returncode == 5:
             if count == 10:
-                raise RuntimeError("Could not find parent device")
+                raise RuntimeError(f"Could not find parent device: {res.stderr.strip()}")
             time.sleep(1 * count)
             count += 1
             continue
@@ -2486,26 +2512,13 @@ def volume_group_for_device(device: str) -> str:
     return vg_name
 
 
-def ensure_device_file(path: str, major: int, minor: int):
-    """Ensure the device file with the given major, minor exists"""
-    os.makedirs(os.path.dirname(path), exist_ok=True)
-    if not os.path.exists(path):
-        os.mknod(path, 0o600 | stat.S_IFBLK, os.makedev(major, minor))
-
-
-def discover_lvm(dev: str, parent: devices.Device, devmgr: devices.DeviceManager):
-    # find the volume group name for the device file
-    vg_name = volume_group_for_device(dev)
-
-    # activating LVM is done OSBuild side.
-    # However we still have to get OSBuild the name of the VG to open
-
-    # Find all logical volumes in the volume group
+def lvm_lvs_for_vg(vg_name: str) -> List[str]:
+    """
+    Get the list of logical volumes for a given volume group.
+    """
     cmd = [
         "lvdisplay", "-C", "--noheadings",
-        "-o", "lv_name,path,lv_kernel_major,lv_kernel_minor",
-        "--separator", ";",
-        vg_name
+        "-o", "lv_name", vg_name
     ]
 
     res = subprocess.run(cmd,
@@ -2517,51 +2530,40 @@ def discover_lvm(dev: str, parent: devices.Device, devmgr: devices.DeviceManager
     if res.returncode != 0:
         raise RuntimeError(res.stderr.strip())
 
-    data = res.stdout.strip()
-    parsed = list(map(lambda line: line.split(";"), data.split("\n")))
-    volumes = OrderedDict()
+    return [lv.strip() for lv in res.stdout.strip().split("\n")]
+
 
+def discover_lvm(dev: str, parent: devices.Device, devmgr: OSBuildDeviceManager):
+    # NB: activating LVM is done by the OSBuild device implementation,
+    # however, the LV name must be passed to the OSBuild device implementation.
+    vg_name = lvm_vg_for_device(dev)
+    lv_names = lvm_lvs_for_vg(vg_name)
+
+    # NB: the order of the volumes is important, we want to mount the root
+    # volume first, so that we can mount the other volumes on top of it.
+    volumes = OrderedDict()
     # devices_map stores for each device path onto the system the corresponding
     # OSBuild's Device object
     devices_map = {}
 
-    for vol in parsed:
-        vol = list(map(lambda v: v.strip(), vol))
-        assert len(vol) == 4
-        name, _, _, _ = vol
+    for lv_name in lv_names:
+        ret = devmgr.open_lvm_lv(lv_name, parent)
+        voldev = ret["path"]
+        device = ret["Device"]
 
-        options = {
-            "volume": name,
-        }
-
-        # Create an OSBuild device object for the LVM partition
-        info = index.get_module_info("Device", "org.osbuild.lvm2.lv")
-        device = devices.Device(
-            name,
-            info,
-            parent,
-            options)
-        if not info:
-            raise RuntimeError("can't find org.osbuild.lvm2.lv")
-        jsonschema.validate(options, info.get_schema())
-        reply = devmgr.open(device)
-        voldev = reply["path"]  # get the path where is mounted the device
-        minor = reply["node"]["minor"]
-        major = reply["node"]["major"]
-
-        info = {
-            "device": voldev
+        # NB: add the device path to the partition info, so that it can be mounted by the caller.
+        part_info = {
+            "device": voldev,
         }
-        ensure_device_file(voldev, int(major), int(minor))
 
-        read_partition(voldev, info)
-        volumes[name] = info
-        if name.startswith("root"):
-            volumes.move_to_end(name, last=False)
+        volumes[lv_name] = read_partition(voldev, part_info)
+        if lv_name.startswith("root"):
+            volumes.move_to_end(lv_name, last=False)
 
         # associate the device path with the Device object, we will need it to
         # mount later on.
         devices_map[voldev] = device
+
     # get back both the device map and the result that'll go in the JSON report
     return devices_map, {
         "lvm": True,
@@ -2632,8 +2634,7 @@ def append_partitions(report, image):
     partitions = report["partitions"]
     with tempfile.TemporaryDirectory() as mountpoint:
         with host.ServiceManager(monitor=monitor.NullMonitor(1)) as mgr:
-
-            devmgr = devices.DeviceManager(mgr, "/dev", os.path.dirname(image))
+            devmgr = OSBuildDeviceManager(mgr, "/dev", os.path.dirname(image))
 
             # Device map associate a path onto where the device is mounted with its
             # corresponding Device object. Mount will require both the path and the
@@ -2642,8 +2643,7 @@ def append_partitions(report, image):
             filesystems = {}
             for part in partitions:
                 start, size = part["start"], part["size"]
-                ret = loop_open(
-                    devmgr,
+                ret = devmgr.open_loopback(
                     part["partuuid"],
                     image,
                     size,
@@ -2780,8 +2780,7 @@ def analyse_image(image) -> Dict[str, Any]:
     with convert_image(image, imgfmt) as target:
         size = os.stat(target).st_size
         with host.ServiceManager(monitor=monitor.NullMonitor(1)) as mgr:
-            device = loop_open(
-                devices.DeviceManager(mgr, "/dev", os.path.dirname(target)),
+            device = OSBuildDeviceManager(mgr, "/dev", os.path.dirname(target)).open_loopback(
                 os.path.basename(target),
                 target,
                 size,


Test patch:
diff --git a/schutzbot/manifest-tests-install-deps.sh b/schutzbot/manifest-tests-install-deps.sh
new file mode 100755
index 00000000..14a48177
--- /dev/null
+++ b/schutzbot/manifest-tests-install-deps.sh
@@ -0,0 +1,78 @@
+#!/bin/bash
+set -euxo pipefail
+
+# Get OS details.
+source /etc/os-release
+ARCH=$(uname -m)
+
+# btrfs is not available on RHEL and CentOS.
+if [[ "${ID}" == "fedora" ]]; then
+    sudo dnf install -y btrfs-progs-devel
+fi
+
+sudo dnf -y install \
+    gcc \
+    git-core \
+    go \
+    gpgme-devel \
+    osbuild-depsolve-dnf \
+    osbuild-tools \
+    device-mapper-devel \
+    unzip
+
+# Install awscli
+WORKDIR=$(mktemp -d)
+pushd "${WORKDIR}"
+
+# Import the AWS CLI public key.
+AWSCLI_PUB_KEY="awscliv2.pubkey"
+cat <<EOF | tee "${AWSCLI_PUB_KEY}"
+-----BEGIN PGP PUBLIC KEY BLOCK-----
+
+mQINBF2Cr7UBEADJZHcgusOJl7ENSyumXh85z0TRV0xJorM2B/JL0kHOyigQluUG
+ZMLhENaG0bYatdrKP+3H91lvK050pXwnO/R7fB/FSTouki4ciIx5OuLlnJZIxSzx
+PqGl0mkxImLNbGWoi6Lto0LYxqHN2iQtzlwTVmq9733zd3XfcXrZ3+LblHAgEt5G
+TfNxEKJ8soPLyWmwDH6HWCnjZ/aIQRBTIQ05uVeEoYxSh6wOai7ss/KveoSNBbYz
+gbdzoqI2Y8cgH2nbfgp3DSasaLZEdCSsIsK1u05CinE7k2qZ7KgKAUIcT/cR/grk
+C6VwsnDU0OUCideXcQ8WeHutqvgZH1JgKDbznoIzeQHJD238GEu+eKhRHcz8/jeG
+94zkcgJOz3KbZGYMiTh277Fvj9zzvZsbMBCedV1BTg3TqgvdX4bdkhf5cH+7NtWO
+lrFj6UwAsGukBTAOxC0l/dnSmZhJ7Z1KmEWilro/gOrjtOxqRQutlIqG22TaqoPG
+fYVN+en3Zwbt97kcgZDwqbuykNt64oZWc4XKCa3mprEGC3IbJTBFqglXmZ7l9ywG
+EEUJYOlb2XrSuPWml39beWdKM8kzr1OjnlOm6+lpTRCBfo0wa9F8YZRhHPAkwKkX
+XDeOGpWRj4ohOx0d2GWkyV5xyN14p2tQOCdOODmz80yUTgRpPVQUtOEhXQARAQAB
+tCFBV1MgQ0xJIFRlYW0gPGF3cy1jbGlAYW1hem9uLmNvbT6JAlQEEwEIAD4CGwMF
+CwkIBwIGFQoJCAsCBBYCAwECHgECF4AWIQT7Xbd/1cEYuAURraimMQrMRnJHXAUC
+ZqFYbwUJCv/cOgAKCRCmMQrMRnJHXKYuEAC+wtZ611qQtOl0t5spM9SWZuszbcyA
+0xBAJq2pncnp6wdCOkuAPu4/R3UCIoD2C49MkLj9Y0Yvue8CCF6OIJ8L+fKBv2DI
+yWZGmHL0p9wa/X8NCKQrKxK1gq5PuCzi3f3SqwfbZuZGeK/ubnmtttWXpUtuU/Iz
+VR0u/0sAy3j4uTGKh2cX7XnZbSqgJhUk9H324mIJiSwzvw1Ker6xtH/LwdBeJCck
+bVBdh3LZis4zuD4IZeBO1vRvjot3Oq4xadUv5RSPATg7T1kivrtLCnwvqc6L4LnF
+0OkNysk94L3LQSHyQW2kQS1cVwr+yGUSiSp+VvMbAobAapmMJWP6e/dKyAUGIX6+
+2waLdbBs2U7MXznx/2ayCLPH7qCY9cenbdj5JhG9ibVvFWqqhSo22B/URQE/CMrG
++3xXwtHEBoMyWEATr1tWwn2yyQGbkUGANneSDFiTFeoQvKNyyCFTFO1F2XKCcuDs
+19nj34PE2TJilTG2QRlMr4D0NgwLLAMg2Los1CK6nXWnImYHKuaKS9LVaCoC8vu7
+IRBik1NX6SjrQnftk0M9dY+s0ZbAN1gbdjZ8H3qlbl/4TxMdr87m8LP4FZIIo261
+Eycv34pVkCePZiP+dgamEiQJ7IL4ZArio9mv6HbDGV6mLY45+l6/0EzCwkI5IyIf
+BfWC9s/USgxchg==
+=ptgS
+-----END PGP PUBLIC KEY BLOCK-----
+EOF
+gpg --import "$AWSCLI_PUB_KEY"
+
+# Download the AWS CLI v2 binary and its signature.
+AWSCLI_ARCHIVE="awscliv2.zip"
+curl "https://awscli.amazonaws.com/awscli-exe-linux-${ARCH}.zip" -o "${AWSCLI_ARCHIVE}"
+AWSCLI_SIG="awscliv2.sig"
+curl  "https://awscli.amazonaws.com/awscli-exe-linux-${ARCH}.zip.sig" -o "${AWSCLI_SIG}"
+
+# Verify the signature.
+gpg --verify "${AWSCLI_SIG}" "${AWSCLI_ARCHIVE}"
+
+# Unzip the AWS CLI v2 binary and install it.
+unzip "${AWSCLI_ARCHIVE}"
+sudo ./aws/install 1> /dev/null
+
+# Verify the installation.
+aws --version
+popd
+rm -rf "${WORKDIR}"
\ No newline at end of file
diff --git a/test/cases/manifest_tests b/test/cases/manifest_tests
new file mode 100755
index 00000000..c0a6cbdf
--- /dev/null
+++ b/test/cases/manifest_tests
@@ -0,0 +1,471 @@
+#!/usr/bin/python3
+
+"""
+Test manifest building consistency using osbuild-image-info
+
+The test case downloads the image build cache (manifests, build info and artifacts) for the osbuild/images repository
+from S3. It then runs osbuild-image-info on the downloaded image artifact file and produces an image-info report.
+The test case then builds the manifest using the current version of osbuild and produces the image artifact.
+It runs osbuild-image-info on the newly built image artifact file and produces an image-info report.
+Finally, it compares the image-info reports and fails if they are different.
+"""
+
+import argparse
+import json
+import os
+import subprocess
+import sys
+import tempfile
+from typing import Dict, List, Optional, Tuple
+
+OSBUILD_IMAGES_REPO_URL = os.environ.get("OSBUILD_IMAGES_REPO_URL", "https://github.com/osbuild/images.git")
+
+OS_RELEASE_FILE = "/etc/os-release"
+
+
+def read_osrelease() -> Dict[str, str]:
+    """Read Operating System Information from `os-release`
+
+    This creates a dictionary with information describing the running operating system. It reads the information from
+    the path array provided as `paths`.  The first available file takes precedence. It must be formatted according to
+    the rules in `os-release(5)`.
+    """
+    osrelease = {}
+
+    with open(OS_RELEASE_FILE, encoding="utf-8") as orf:
+        for line in orf:
+            line = line.strip()
+            if not line:
+                continue
+            if line[0] == "#":
+                continue
+            key, value = line.split("=", 1)
+            osrelease[key] = value.strip('"')
+
+    return osrelease
+
+
+def get_host_distro() -> str:
+    """
+    Get the host distro version based on data in the os-release file.
+    The format is <distro>-<version> (e.g. fedora-41).
+    """
+    osrelease = read_osrelease()
+    return f"{osrelease['ID']}-{osrelease['VERSION_ID']}"
+
+
+def get_host_arch() -> str:
+    """
+    Get the host architecture.
+    """
+    return os.uname().machine
+
+
+def manifest_pipeline_names(path: str) -> List[str]:
+    """
+    Read the manifest file and return a list of pipeline names
+    """
+    with open(path, "r", encoding="utf-8") as f:
+        manifest = json.load(f)
+
+    return [pipeline["name"] for pipeline in manifest["pipelines"]]
+
+
+def find_manifest_file(build_path: str) -> str:
+    """
+    Return the path to the manifest file in the build directory
+    """
+    return os.path.join(build_path, "manifest.json")
+
+
+def find_image_file(build_path: str, export_name: Optional[str] = None) -> str:
+    """
+    Find the path to the image by searching for the file under the directory named 'export_name'. If the name of the
+    export pipeline is not provided, determine it by reading the manifest to get the name of the last pipeline.
+    Raises RuntimeError if no or multiple files are found in the expected path.
+    """
+    if export_name is None:
+        manifest_file = find_manifest_file(build_path)
+        export_name = manifest_pipeline_names(manifest_file)[-1]
+    files = os.listdir(os.path.join(build_path, export_name))
+    if len(files) > 1:
+        error = "Multiple files found in build path while searching for image file"
+        error += "\n".join(files)
+        raise RuntimeError(error)
+
+    if len(files) == 0:
+        raise RuntimeError("No found in build path while searching for image file")
+
+    return os.path.join(build_path, export_name, files[0])
+
+
+def checkout_images_repo(ref, workdir: os.PathLike) -> str:
+    """
+    Checkout the 'images' repository at a specific commit and return the path to the directory
+    If the repository is already checked-out, switch to the specified commit.
+    """
+    images_path = os.path.join(workdir, "images")
+
+    if not os.path.exists(images_path):
+        print(f"Checking out '{OSBUILD_IMAGES_REPO_URL}' repository at ref '{ref}'")
+        try:
+            subprocess.check_call(
+                ["git", "clone", "--depth=1", "--no-single-branch", OSBUILD_IMAGES_REPO_URL, "images"],
+                cwd=workdir, stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
+            )
+        except subprocess.CalledProcessError as e:
+            print(f"Failed to clone 'images' repository: {e.stdout.decode()}")
+            sys.exit(1)
+    else:
+        print(f"'images' repository is already checked-out at '{images_path}'")
+
+    subprocess.check_call(["git", "checkout", ref], cwd=images_path, stdout=subprocess.DEVNULL)
+    return images_path
+
+
+def download_image_build_artifact(images_path: os.PathLike, build_dir: os.PathLike) -> None:
+    """
+    Download the image build artifact from S3 for a specific image build.
+    """
+    cmd = ["./test/scripts/dl-one-image-build-cache", build_dir]
+    print(" ".join(cmd))
+    try:
+        subprocess.check_call(cmd, cwd=images_path, env=os.environ)
+    except subprocess.CalledProcessError as _:
+        print("⚠️ Failed to download image build cache")
+        sys.exit(1)
+
+
+def download_image_build_cache_md(
+        images_path, output_dir, distros: List[str], arch: str, configs: Optional[List[str]] = None,
+        image_types: Optional[List[str]] = None, skip_img_types: Optional[List[str]] = None) -> None:
+    """
+    Download the image build cache metadata from S3 for a specific distro / arch / configs.
+    The image artifacts is not downloaded, only the metadata.
+    """
+    cmd = [
+        "./test/scripts/dl-image-build-cache",
+        "--arch", arch,
+        "--output", output_dir,
+    ]
+    for distro in distros:
+        cmd += ["--distro", distro]
+    for config in configs or []:
+        cmd += ["--config", config]
+
+    # The image_types and skip_img_types are mutually exclusive,
+    # but this is enforced in the argument parser.
+    for imag_type in image_types or []:
+        cmd += ["--image-type", imag_type]
+    for skip_img_type in skip_img_types or []:
+        cmd += ["--skip-image-type", skip_img_type]
+
+    print(" ".join(cmd))
+    try:
+        subprocess.check_call(cmd, cwd=images_path, env=os.environ)
+    except subprocess.CalledProcessError as _:
+        print("⚠️ Failed to download image build cache")
+        sys.exit(1)
+
+
+def gen_image_info_report(image_path: str, report_path: str) -> None:
+    """
+    Run osbuild-image-info on the image file and return the parsed JSON output
+    """
+    cmd = ["sudo", "osbuild-image-info", image_path]
+    output = subprocess.check_output(cmd)
+    with open(report_path, "w", encoding="utf-8") as f:
+        f.write(output.decode())
+
+
+def diff_files(file1: str, file2: str) -> str:
+    """
+    Run diff on two files and return the output
+    """
+    cmd = [
+        "diff", "-u",
+        "-I", r'"[0-9a-f]\{8\}-\([0-9a-f]\{4\}-\)\{3\}[0-9a-f]\{12\}"',  # Ignore UUIDs
+        "-I", r'"[0-9a-zA-Z]\{6\}-\([0-9a-zA-Z]\{4\}-\)\{5\}[0-9a-zA-Z]\{6\}"',  # Ignore LVM quasi-UUIDs
+        "-I", r'"volid": "[0-9a-f]\{8\}"',  # Ignore volid
+        file1, file2,
+    ]
+    print(" ".join(cmd))
+    run = subprocess.run(cmd, stdout=subprocess.PIPE, check=False)
+    return run.stdout.decode()
+
+
+class OSBuild:
+    def __init__(self, store, outdir):
+        self.store = store
+        self.outdir = outdir
+
+    def run(self, manifest, exports, checkpoints=None) -> Tuple[int, str, str]:
+        cmd = [
+            "sudo",
+            "osbuild",
+            "--cache-max-size", "unlimited",
+            "--store", os.fspath(self.store),
+            "--output-directory", os.fspath(self.outdir),
+            os.fspath(manifest)
+        ]
+
+        for checkpoint in checkpoints or []:
+            cmd += [
+                "--checkpoint", checkpoint
+            ]
+
+        for export in exports:
+            cmd += [
+                "--export", export
+            ]
+
+        print(" ".join(cmd))
+        run = subprocess.run(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, check=False)
+
+        return run.returncode, run.stdout.decode(), run.stderr.decode()
+
+
+def run_manifest_behavior_test(
+        build_dir: str, results_dir: str, osbuild_store: str, rm_artifacts_after_test: bool = False) -> None:
+    """
+    This function implements the test case for a single image build cache directory which contains:
+     - the manifest.json file
+     - the image artifact file built by an older version of osbuild pinned in the images repository
+
+    The test case does the following:
+     - runs osbuild-image-info on the downloaded image artifact file and produces image-info report
+     - builds the manifest using the current version of osbuild and produces image artifact
+     - runs osbuild-image-info on the newly built image artifact file and produces image-info report
+     - compares the image-info reports
+
+    The test case fails if the image-info reports are different
+
+    The function can optionally remove the downloaded and rebuilt image artifacts after the test case
+    to save disk space.
+    """
+    manifest = find_manifest_file(build_dir)
+    manifest_pipelines = manifest_pipeline_names(manifest)
+    downloaded_image = find_image_file(build_dir)
+    downloaded_image_iminfo = os.path.join(results_dir, "downloaded_image_iminfo.json")
+    print(f"📜 Generating image info report for downloaded image '{downloaded_image}' to '{downloaded_image_iminfo}'")
+    gen_image_info_report(downloaded_image, downloaded_image_iminfo)
+
+    if rm_artifacts_after_test:
+        print("🗑️ Removing downloaded image artifact")
+        os.remove(downloaded_image)
+
+    rebuild_dir = os.path.join(build_dir, "rebuild")
+    os.makedirs(rebuild_dir, exist_ok=True)
+    osbuild = OSBuild(osbuild_store, rebuild_dir)
+    print("Rebuilding the image artifact using installed osbuild version")
+    retcode, stdout, stderr = osbuild.run(manifest, [manifest_pipelines[-1]], ["build"])
+    with open(os.path.join(rebuild_dir, "image_rebuild_osbuild.log"), "w", encoding="utf-8") as f:
+        f.write(stdout)
+        f.write(stderr)
+    if retcode != 0:
+        raise RuntimeError(f"Failed to rebuild the image artifact:\nSTDOUT:\n{stdout}\nSTDERR:\n{stderr}")
+
+    rebuilt_image = find_image_file(rebuild_dir, manifest_pipelines[-1])
+    rebuilt_image_iminfo = os.path.join(results_dir, "rebuilt_image_iminfo.json")
+    print(f"📜 Generating image info report for rebuilt image: '{rebuilt_image}' to '{rebuilt_image_iminfo}'")
+    gen_image_info_report(rebuilt_image, rebuilt_image_iminfo)
+
+    if rm_artifacts_after_test:
+        print("🗑️ Removing the dir with rebuilt image artifact")
+        # NB: use sudo to remove the dir, because the files are owned by root after osbuild run
+        subprocess.run(["sudo", "rm", "-rf", rebuild_dir], check=False)
+
+    diff = diff_files(downloaded_image_iminfo, rebuilt_image_iminfo)
+    with open(os.path.join(results_dir, "iminfo.diff"), "w", encoding="utf-8") as f:
+        f.write(diff)
+    if diff:
+        raise RuntimeError(f"Image info reports are different:\n{diff}")
+
+
+def get_argparser():
+
+    class ExtendAction(argparse.Action):
+        """
+        Custom argparse action to append multiple values to a list option
+        to prevent overwriting the list with each new value.
+
+        This may be removed when Python 3.8 is the minimum supported version.
+        """
+
+        def __call__(self, parser, namespace, values, option_string=None):
+            items = getattr(namespace, self.dest) or []
+            items.extend(values)
+            setattr(namespace, self.dest, items)
+
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.register('action', 'extend', ExtendAction)
+
+    parser.add_argument(
+        "--distro",
+        metavar="DISTRO",
+        action="extend",
+        nargs="+",
+        help="Distro to test. Default is the host distro."
+    )
+    parser.add_argument(
+        "--image-type", action="extend", metavar="TYPE", nargs="+",
+        help="Image type for which the image build cache is downloaded. Can be specified multiple times. " +
+             "If not provided, all image types are downloaded. " +
+             "The option is mutually exclusive with --skip-image-type.",
+    )
+    parser.add_argument(
+        "--skip-image-type", action="extend", metavar="TYPE_GLOB", nargs="+",
+        help="Image types to skip when downloading the image build cache. Can be specified multiple times. " +
+             "This is useful for image types which can't be analyzed using osbuild-image-info.",
+    )
+    parser.add_argument(
+        "--config", action="extend", metavar="NAME_GLOB", nargs="+",
+        help="Config name globs used to filter which image build cache files are downloaded. " +
+             "Can be specified multiple times. If not provided, all configs are downloaded.",
+    )
+    parser.add_argument(
+        "--images-ref",
+        metavar="REF",
+        default="main",
+        help="Git ref to checkout in the osbuild/images repository."
+    )
+    parser.add_argument(
+        "--osbuild-store",
+        metavar="PATH",
+        type=os.path.abspath,
+        default=None,
+        help="Directory where intermediary os trees are stored."
+    )
+    parser.add_argument(
+        "--workdir",
+        metavar="PATH",
+        type=os.path.abspath,
+        help="Working directory where the images repository is checked out and the image build cache is downloaded. " +
+             "If not provided, a temporary directory will be used and deleted on exit."
+    )
+    parser.add_argument(
+        "--results-dir",
+        metavar="PATH",
+        type=os.path.abspath,
+        default="./osbuild-manifest-tests-results",
+        help="Directory where the test results are stored."
+    )
+    parser.add_argument(
+        "--chunk-size", nargs=2, metavar=("CHUNK_NUMBER", "TOTAL_CHUNKS"),
+        type=int, default=None,
+        help="Run tests only for a specific chunk of the image build cache. " +
+             "The first argument is the chunk number (starting from 1) and the second argument is the total " +
+             "number of chunks. This is useful for running the tests in parallel."
+    )
+    parser.add_argument(
+        "--rm-artifacts-after-test",
+        action="store_true",
+        default=False,
+        help="Remove the downloaded and rebuilt image artifacts after the test case. " +
+             "This is useful for saving disk space."
+    )
+    return parser
+
+
+def main():
+    parser = get_argparser()
+    args = parser.parse_args()
+
+    if not args.distro:
+        args.distro = [get_host_distro()]
+
+    chunk_number, total_chunks = None, None
+    if args.chunk_size:
+        chunk_number, total_chunks = args.chunk_size
+        if chunk_number < 1 or total_chunks < 1:
+            parser.error("Both values for '--chunk-size' must be greater than zero.")
+        if chunk_number > total_chunks:
+            parser.error("The chunk number must be less than or equal to the total number of chunks.")
+
+    if args.image_type and args.skip_image_type:
+        parser.error("Options --image-type and --skip-image-type are mutually exclusive.")
+
+    try:
+        tmpdir = tempfile.TemporaryDirectory(dir='/var/tmp', prefix='osbuild-manifest-tests-workdir')
+        workdir = args.workdir or tmpdir
+        os.makedirs(workdir, exist_ok=True)
+        print(f"👷 Using working directory: {workdir}")
+        os.chdir(workdir)
+
+        osbuild_store_dir = args.osbuild_store or os.path.join(workdir, "osbuild-store")
+        os.makedirs(osbuild_store_dir, exist_ok=True)
+        print(f"💾 Using osbuild store directory: {osbuild_store_dir}")
+
+        # Checkout the images repository, since we will need scripts from it to download the image build cache
+        images_path = checkout_images_repo(args.images_ref, workdir)
+
+        # Create the directory where the image build cache will be downloaded
+        image_build_cache = os.path.join(workdir, "image-build-cache")
+        os.makedirs(image_build_cache, exist_ok=True)
+
+        download_image_build_cache_md(
+            images_path, image_build_cache, args.distro, get_host_arch(),
+            args.config, args.image_type, args.skip_image_type
+        )
+
+        # The test case is run for every directory in the image build cache directory
+        test_cases = sorted(os.listdir(image_build_cache))
+
+        if not test_cases:
+            print("⚠️ No image build cache directories found -> nothing to test", file=sys.stderr)
+            sys.exit(1)
+
+        print(f"📦 Found {len(test_cases)} image build cache directories")
+        if chunk_number and total_chunks:
+            all_test_cases = test_cases
+            print(f"📦 Will run subset of tests for chunk {chunk_number} of {total_chunks}:")
+            chunk_size = len(test_cases) // total_chunks
+            chunk_size_remainder = len(test_cases) % total_chunks
+
+            # determine the chunk index range
+            start = 0
+            end = 0
+            for i in range(chunk_number):
+                current_chunk_size = chunk_size + 1 if i < chunk_size_remainder else chunk_size
+                start = end
+                end = start + current_chunk_size
+
+            test_cases = test_cases[start:end]
+            idx_range = range(start, end)
+            for i, test_case in enumerate(all_test_cases):
+                print(f"    {'🟢' if i in idx_range else '🚫'} {test_case}")
+
+        # Dictionary holding the test case name as key and a boolean indicating if the test case failed as value
+        test_cases_failed: Dict[str, bool] = {}
+        print(f"🏃 Running {len(test_cases)} test cases:\n{os.linesep.join(test_cases)}")
+        for test_case in test_cases:
+            print(f"🏃 Running test case for {test_case}")
+            test_case_build_dir = os.path.join(image_build_cache, test_case)
+            download_image_build_artifact(images_path, test_case_build_dir)
+            test_case_results_dir = os.path.join(args.results_dir, test_case)
+            os.makedirs(test_case_results_dir, exist_ok=True)
+            try:
+                run_manifest_behavior_test(
+                    test_case_build_dir, test_case_results_dir, osbuild_store_dir, args.rm_artifacts_after_test)
+            # pylint: disable=broad-exception-caught
+            except Exception as e:
+                print(f"❌ {test_case} FAILED")
+                print(e)
+                test_cases_failed[test_case] = True
+            else:
+                print(f"✅ {test_case} PASSED")
+                test_cases_failed[test_case] = False
+    finally:
+        # We can't use the context manager here, because some files in it may be owned by root
+        subprocess.run(["sudo", "rm", "-rf", tmpdir.name], check=False)
+
+    print("Test results:")
+    for test_case, failed in test_cases_failed.items():
+        print(f"    {'❌' if failed else '✅'} {test_case}")
+
+    if any(test_cases_failed.values()):
+        sys.exit(1)
+
+
+if __name__ == "__main__":
+    main()
diff --git a/test/cases/manifest_tests.sh b/test/cases/manifest_tests.sh
deleted file mode 100755
index 7233ef7f..00000000
--- a/test/cases/manifest_tests.sh
+++ /dev/null
@@ -1,23 +0,0 @@
-#!/bin/bash
-set -euxo pipefail
-# Get OS details.
-source /etc/os-release
-ARCH=$(uname -m)
-DISTRO_CODE="${DISTRO_CODE:-${ID}-${VERSION_ID}}"
-# get manifest-db at the version specified for this arch+distro
-MANIFEST_DB_COMMIT=$(cat Schutzfile | jq -r '.global.dependencies."manifest-db".commit')
-MANIFEST_DB_REPO="https://github.com/osbuild/manifest-db"
-git clone "$MANIFEST_DB_REPO" manifest-db
-cd manifest-db
-git checkout --force "$MANIFEST_DB_COMMIT"
-
-# set the maximum cache size to unlimited
-echo "{}" | sudo osbuild --store /var/lib/osbuild/store --cache-max-size unlimited -
-
-IFS='/' read -r -a array <<< $1
-
-sudo dnf install -y osbuild-tools
-
-# run the tests from the manifest-db for this arch+distro
-echo "Running the osbuild-image-test for arch $ARCH and ditribution $DISTRO_CODE"
-sudo tools/osbuild-image-test --arch=$ARCH --distro=$DISTRO_CODE --image-info-path="$(which osbuild-image-info)" --instance-number="${array[0]}" --total-number-of-instances="${array[1]}"
diff --git a/test/mod/test_monitor.py b/test/mod/test_monitor.py
index d8f3895e..148a4957 100644
--- a/test/mod/test_monitor.py
+++ b/test/mod/test_monitor.py
@@ -7,6 +7,7 @@ import json
 import os
 import sys
 import tempfile
+import threading
 import time
 import unittest
 from collections import defaultdict
@@ -416,3 +417,28 @@ def test_jsonseq_download_unhappy(mocked_download, tmp_path):
         assert log[1]["result"]["name"] == "source org.osbuild.curl"
         assert log[1]["result"]["success"] is False
         assert log[1]["result"]["output"] == "RuntimeError: curl: error download ...\n error stack"
+
+
+def test_json_progress_monitor_handles_racy_writes(tmp_path):
+    output_path = tmp_path / "jsonseq.log"
+    with output_path.open("w") as fp:
+        mon = JSONSeqMonitor(fp.fileno(), 10)
+
+        def racy_write(s):
+            for i in range(20):
+                mon.log(f"{s}: {i}")
+                time.sleep(0.0001)
+        t1 = threading.Thread(target=racy_write, args=("msg from t1",))
+        t2 = threading.Thread(target=racy_write, args=("msg from t2",))
+        t1.start()
+        t2.start()
+        t1.join()
+        t2.join()
+    # ensure the file is valid jsonseq
+    with output_path.open() as fp:
+        for line in fp.readlines():
+            line = line.strip().strip("\1xe")
+            try:
+                json.loads(line)
+            except json.decoder.JSONDecodeError:
+                pytest.fail(f"the jsonseq stream is not valid json, got {line}")