Repository: osbuild/osbuild-composer
Pull request #1187 (2021-01-29 12:57:09)

image-info: Add workaround for listing services by status
This pull request includes:

- [x] adequate testing for the new functionality or fixed issue
  - Fixes an issue, affected test cases were updated 
- [ ] adequate documentation informing people about the change
  - No documentation other than comments in the code is included. This is not user-visible change
 
**Summary**
`image-info` tools parses output of `systemctl list-unit-files` run on a
different tree (with `--root` option), to determine the list of enabled
and disabled services on the inspected image. However since Fedora 33
(and presumably since systemd v246), the output of `systemctl
list-unit-files` changed. Some units previously reported as "enabled" or
"disabled" are now reported as "alias", which means, that they are just
a symlink to a different unit.

There is no systemd command, that would take an "alias" unit and would
report its state as "enabled" or "disabled" and could run on a different
tree (with "--root" option).

To make the list of reported services in the given state consistent on
systems with older and new (v246+) systemd version, check all "alias"
units and append them to the list of services with a specific status,
if their target is also listed in in the list.

Example of the `systemctl list-unit-files` output change:
```
~]# rpm -q systemd
systemd-246.6-3.fc33.x86_64
~]# systemctl list-unit-files ctrl-alt-del.target
UNIT FILE           STATE VENDOR PRESET
ctrl-alt-del.target alias -
```
```
~]# rpm -q systemd
systemd-245.8-2.fc32.x86_64
~]# systemctl list-unit-files ctrl-alt-del.target
UNIT FILE           STATE   VENDOR PRESET
ctrl-alt-del.target enabled disabled
```
This change makes it possible to produce consistent output for an
inspected image, regardless if the `image-info` tool is run on Fedora
32, Fedora 33 or RHEL-8.

Also regenerate all Fedora 33 test cases, since this commit changes the
content of produced list of enabled / disabled services since Fedora 33.
The list is now consistent with what would be produced by `image-info`
for an image on older Fedora (e.g. 32) or RHEL-8.




Patch:
diff --git a/HACKING.md b/HACKING.md
index eaaa27b27..123a1af65 100644
--- a/HACKING.md
+++ b/HACKING.md
@@ -1,5 +1,7 @@
 # Hacking on osbuild-composer
 
+## Virtual Machine
+
 *osbuild-composer* cannot be run from the source tree, but has to be installed
 onto a system. We recommend doing this by building rpms, with:
 
@@ -42,3 +44,65 @@ For a quick compile and debug cycle, we recommend iterating code using thorough
 unit tests before going through the full workflow described above.
 
 [1]: https://wiki.qemu.org/Documentation/Networking#User_Networking_.28SLIRP.29
+
+## Containers
+
+*osbuild-composer* and *osbuild-composer-worker* can be run using Docker
+containers. Building and starting containers is generally faster than building
+RPMs and installing them in a VM, so this method is more convenient for
+developing and testing changes quickly. However, using this method has several
+downsides:
+- It doesn't build the RPMs so the `.spec` file isn't tested.
+- The environment is quite different from production (e.g., installation paths,
+  privileges and permissions).
+- The setup is not complete for all required services, so some functionality
+  isn't available for testing this way (e.g., Koji Hub and all its dependent
+  services).
+
+The containers are a good way to quickly test small changes, but before
+submitting a Pull Request, it's recommended to run through all the tests using
+the [Virtual Machine](#virtual-machine) setup described above.
+
+### Configuration
+
+Each service (*composer* and *worker*) requires a configuration file and a set
+of certificates. The storage location for these is shared between the
+containers for simplicity. By default it's configured to be at
+`./containers/config`, but this can be changed in the
+[`./distribution/.env`](./distribution/.env) file by modifying the value of the
+`$CONTAIN£R_CONFIG_DIR` variable (both absolute and relative paths work).
+
+Use the [`tools/gen-certs.sh`](./tools/gen-certs.sh) script to
+generate the certificates (using the test OpenSSL config file):
+
+    ./tools/gen-certs.sh ./test/data/x509/openssl.cnf ./containers/config  ./containers/config/ca
+
+Note that the two arguments `./containers/config` and `./containers/config/ca`
+should be the same location as the `$CONTAINER_CONFIG_DIR` described above
+
+The services also require a config file each which they expect to be in the
+same directory. The following test files can be copied into it:
+
+    cp ./test/data/composer/osbuild-composer.toml ./test/data/composer/osbuild-worker.toml ./containers/config/
+
+The `$CONTAINER_CONFIG_DIR` (default `containers/config`) directory will be mounted inside both containers (see
+the [`docker-composer.yml`](./distribution/docker-compose.yml) file).
+
+### Build and run
+
+To build the containers, change into the `distribution/` directory and run:
+
+    docker-compose build
+
+To start the containers, change into the `distribution/` directory and run:
+
+    docker-compose up
+
+You can send requests to the *osbuild-composer* container directly using the
+generated certificate and client key. For example, from the project root, run:
+
+    curl -k --cert ./containers/config/client-crt.pem --key ./containers/config/client-key.pem https://172.30.0.10:9196/api/composer-koji/v1/status
+
+To rebuild the containers after a change, add the `--build` flag to the `docker-compose` command:
+
+    docker-compose up --build
diff --git a/distribution/.env b/distribution/.env
new file mode 100644
index 000000000..4e3c577af
--- /dev/null
+++ b/distribution/.env
@@ -0,0 +1,2 @@
+COMPOSE_PROJECT_NAME=osbuild
+CONTAINER_CONFIG_DIR=../containers/config
diff --git a/distribution/Dockerfile-ubi b/distribution/Dockerfile-ubi
index 8a3bf0cf8..234ee2f20 100644
--- a/distribution/Dockerfile-ubi
+++ b/distribution/Dockerfile-ubi
@@ -4,7 +4,7 @@ ENV GOFLAGS=-mod=vendor
 RUN go install ./cmd/osbuild-composer/
 
 FROM registry.access.redhat.com/ubi8/ubi-minimal:latest
-RUN microdnf install python3
+RUN microdnf install python3 python3-dnf
 RUN mkdir -p "/usr/libexec/osbuild-composer"
 RUN mkdir -p "/etc/osbuild-composer/"
 RUN mkdir -p "/run/osbuild-composer/"
diff --git a/distribution/Dockerfile-worker b/distribution/Dockerfile-worker
new file mode 100644
index 000000000..ee02a91de
--- /dev/null
+++ b/distribution/Dockerfile-worker
@@ -0,0 +1,15 @@
+FROM registry.access.redhat.com/ubi8/go-toolset:latest AS builder
+COPY . .
+ENV GOFLAGS=-mod=vendor
+RUN go install ./cmd/osbuild-worker
+
+FROM fedora
+RUN dnf install -y qemu-img osbuild osbuild-ostree
+RUN mkdir -p "/usr/libexec/osbuild-composer"
+RUN mkdir -p "/etc/osbuild-composer/"
+RUN mkdir -p "/run/osbuild-composer/"
+RUN mkdir -p "/var/cache/osbuild-composer/"
+RUN mkdir -p "/var/lib/osbuild-composer/"
+COPY --from=builder /opt/app-root/src/go/bin/osbuild-worker /usr/libexec/osbuild-composer/
+
+ENTRYPOINT ["/usr/libexec/osbuild-composer/osbuild-worker"]
diff --git a/distribution/docker-compose.yml b/distribution/docker-compose.yml
new file mode 100644
index 000000000..79cf838f6
--- /dev/null
+++ b/distribution/docker-compose.yml
@@ -0,0 +1,40 @@
+version: '2.4'
+services:
+  composer:
+    image: local/osbuild-composer
+    build:
+      context: ..
+      dockerfile: ./distribution/Dockerfile-ubi
+    volumes:
+      - ${CONTAINER_CONFIG_DIR}/:/etc/osbuild-composer
+    networks:
+      net:
+        ipv4_address: 172.30.0.10
+  worker:
+    image: local/osbuild-worker
+    build:
+      context: ..
+      dockerfile: ./distribution/Dockerfile-worker
+    # override the entrypoint to specify composer hostname and port
+    entrypoint: /usr/libexec/osbuild-composer/osbuild-worker composer:8700
+    volumes:
+      - ${CONTAINER_CONFIG_DIR}/:/etc/osbuild-composer
+    environment:
+      - CACHE_DIRECTORY=/var/cache/osbuild-worker
+    privileged: true
+    cap_add:
+      - MKNOD
+      - SYS_ADMIN
+      - NET_ADMIN
+    networks:
+      net:
+        ipv4_address: 172.30.0.20
+    depends_on:
+      - "composer"
+
+networks:
+  net:
+    ipam:
+      driver: default
+      config:
+        - subnet: 172.30.0.0/16
diff --git a/docs/news/unreleased/rhel84-differences-fixed.md b/docs/news/unreleased/rhel84-differences-fixed.md
index d0141aa21..39378b5cc 100644
--- a/docs/news/unreleased/rhel84-differences-fixed.md
+++ b/docs/news/unreleased/rhel84-differences-fixed.md
@@ -6,12 +6,12 @@ and disabled services. To remedy these differences the following changes have
 been made:
 
 The following packages have been added to our qcow2 image: oddjob, 
-oddjob-mkhomedir, psmisc, authselect-compat, rng-tools, dbxtool.
+oddjob-mkhomedir, psmisc, authselect-compat, dbxtool.
 
 The following packages have been removed from our qcow2 image: 
 dnf-plugin-spacewalk, fwupd, nss, and udisks2.
 
-The following services have been enabled: rngd.service, nfs-convert.service.
+The following services have been enabled: nfs-convert.service.
 
 The following services have been removed/disabled: mdmonitor.service, 
 udisks2.service, fwupd-refresh.timer, mdcheck_continue.timer, 
diff --git a/internal/distro/rhel84/distro.go b/internal/distro/rhel84/distro.go
index 9dfc5e5a0..d4f175313 100644
--- a/internal/distro/rhel84/distro.go
+++ b/internal/distro/rhel84/distro.go
@@ -958,11 +958,9 @@ func New() distro.Distro {
 			"libertas-usb8388-firmware",
 			"nss",
 			"plymouth",
+			"rng-tools",
 			"udisks2",
 		},
-		enabledServices: []string{
-			"rngd.service",
-		},
 		defaultTarget:           "multi-user.target",
 		kernelOptions:           "console=tty0 console=ttyS0,115200n8 no_timer_check net.ifnames=0 crashkernel=auto",
 		bootable:                true,
diff --git a/internal/weldr/api.go b/internal/weldr/api.go
index cbbfd2062..166a86661 100644
--- a/internal/weldr/api.go
+++ b/internal/weldr/api.go
@@ -666,6 +666,19 @@ func (api *API) sourceNewHandler(writer http.ResponseWriter, request *http.Reque
 		return
 	}
 
+	// Is there an existing System Repo using this id?
+	for _, n := range api.systemRepoNames() {
+		if n == source.GetKey() {
+			// Users cannot replace system repos
+			errors := responseError{
+				ID:  "SystemSource",
+				Msg: fmt.Sprintf("%s is a system source, it cannot be changed.", source.GetKey()),
+			}
+			statusResponseError(writer, http.StatusBadRequest, errors)
+			return
+		}
+	}
+
 	api.store.PushSource(source.GetKey(), source.SourceConfig())
 
 	statusResponseOK(writer)
diff --git a/osbuild-composer.spec b/osbuild-composer.spec
index c56403de5..87aef2ae3 100644
--- a/osbuild-composer.spec
+++ b/osbuild-composer.spec
@@ -179,6 +179,7 @@ install -m 0755 -vp _bin/osbuild-auth-tests                     %{buildroot}%{_l
 install -m 0755 -vp _bin/osbuild-koji-tests                     %{buildroot}%{_libexecdir}/osbuild-composer-test/
 install -m 0755 -vp _bin/cloud-cleaner                          %{buildroot}%{_libexecdir}/osbuild-composer-test/
 install -m 0755 -vp tools/provision.sh                          %{buildroot}%{_libexecdir}/osbuild-composer-test/
+install -m 0755 -vp tools/gen-certs.sh                          %{buildroot}%{_libexecdir}/osbuild-composer-test/
 install -m 0755 -vp tools/image-info                            %{buildroot}%{_libexecdir}/osbuild-composer-test/
 install -m 0755 -vp tools/run-koji-container.sh                 %{buildroot}%{_libexecdir}/osbuild-composer-test/
 install -m 0755 -vp tools/koji-compose.py                       %{buildroot}%{_libexecdir}/osbuild-composer-test/
diff --git a/tools/gen-certs.sh b/tools/gen-certs.sh
new file mode 100755
index 000000000..c60327990
--- /dev/null
+++ b/tools/gen-certs.sh
@@ -0,0 +1,97 @@
+#!/bin/bash
+if (( $# != 3 )); then
+    echo "Usage: $0 <openssl-config> <certdir> <cadir>"
+    echo
+    echo "Positional arguments"
+    echo "  <openssl-config>  OpenSSL configuration file"
+    echo "  <certdir>         Destination directory for the generated files"
+    echo "  <cadir>           Working directory for the generation process"
+    exit 1
+fi
+
+set -euxo pipefail
+# Generate all X.509 certificates for the tests
+# The whole generation is done in a $CADIR to better represent how osbuild-ca
+# it.
+OPENSSL_CONFIG="$1"
+CERTDIR="$2"
+CADIR="$3"
+
+# The $CADIR might exist from a previous test (current Schutzbot's imperfection)
+rm -rf "$CADIR" || true
+mkdir -p "$CADIR" "$CERTDIR"
+
+# Convert the arguments to real paths so we can safely change working directory
+OPENSSL_CONFIG="$(realpath "${OPENSSL_CONFIG}")"
+CERTDIR="$(realpath "${CERTDIR}")"
+CADIR="$(realpath "${CADIR}")"
+
+pushd "$CADIR"
+    mkdir certs private
+    touch index.txt
+
+    # Generate a CA.
+    openssl req -config "$OPENSSL_CONFIG" \
+        -keyout private/ca.key.pem \
+        -new -nodes -x509 -extensions osbuild_ca_ext \
+        -out ca.cert.pem -subj "/CN=osbuild.org"
+
+    # Copy the private key to the location expected by the tests
+    cp ca.cert.pem "$CERTDIR"/ca-crt.pem
+
+    # Generate a composer certificate.
+    openssl req -config "$OPENSSL_CONFIG" \
+        -keyout "$CERTDIR"/composer-key.pem \
+        -new -nodes \
+        -out /tmp/composer-csr.pem \
+        -subj "/CN=localhost/emailAddress=osbuild@example.com" \
+        -addext "subjectAltName=DNS:localhost, DNS:composer"
+
+    openssl ca -batch -config "$OPENSSL_CONFIG" \
+        -extensions osbuild_server_ext \
+        -in /tmp/composer-csr.pem \
+        -out "$CERTDIR"/composer-crt.pem
+
+    # Generate a worker certificate.
+    openssl req -config "$OPENSSL_CONFIG" \
+        -keyout "$CERTDIR"/worker-key.pem \
+        -new -nodes \
+        -out /tmp/worker-csr.pem \
+        -subj "/CN=localhost/emailAddress=osbuild@example.com" \
+        -addext "subjectAltName=DNS:localhost, DNS:worker"
+
+    openssl ca -batch -config "$OPENSSL_CONFIG" \
+        -extensions osbuild_client_ext \
+        -in /tmp/worker-csr.pem \
+        -out "$CERTDIR"/worker-crt.pem
+
+    # Generate a client certificate.
+    openssl req -config "$OPENSSL_CONFIG" \
+        -keyout "$CERTDIR"/client-key.pem \
+        -new -nodes \
+        -out /tmp/client-csr.pem \
+        -subj "/CN=client.osbuild.org/emailAddress=osbuild@example.com" \
+        -addext "subjectAltName=DNS:client.osbuild.org"
+
+    openssl ca -batch -config "$OPENSSL_CONFIG" \
+        -extensions osbuild_client_ext \
+        -in /tmp/client-csr.pem \
+        -out "$CERTDIR"/client-crt.pem
+
+    # Client keys are used by tests to access the composer APIs. Allow all users access.
+    chmod 644 "$CERTDIR"/client-key.pem
+
+    # Generate a kojihub certificate.
+    openssl req -config "$OPENSSL_CONFIG" \
+        -keyout "$CERTDIR"/kojihub-key.pem \
+        -new -nodes \
+        -out /tmp/kojihub-csr.pem \
+        -subj "/CN=localhost/emailAddress=osbuild@example.com" \
+        -addext "subjectAltName=DNS:localhost"
+
+    openssl ca -batch -config "$OPENSSL_CONFIG" \
+        -extensions osbuild_server_ext \
+        -in /tmp/kojihub-csr.pem \
+        -out "$CERTDIR"/kojihub-crt.pem
+
+popd
diff --git a/tools/image-info b/tools/image-info
index cb4e548be..1c2a5adc7 100755
--- a/tools/image-info
+++ b/tools/image-info
@@ -134,9 +134,6 @@ def parse_unit_files(s, expected_state):
             continue
         r.append(unit)
 
-    # deduplicate and sort
-    r = list(set(r))
-    r.sort()
     return r
 
 
@@ -265,8 +262,69 @@ def rpm_packages(tree, is_ostree):
     return list(sorted(pkgs))
 
 
+@contextlib.contextmanager
+def change_root(root):
+    real_root = os.open("/", os.O_RDONLY)
+    try:
+        os.chroot(root)
+        yield None
+    finally:
+        os.fchdir(real_root)
+        os.chroot(".")
+        os.close(real_root)
+
+
 def read_services(tree, state):
-    return subprocess_check_output(["systemctl", f"--root={tree}", "list-unit-files"], (lambda s: parse_unit_files(s, state)))
+    services_state = subprocess_check_output(["systemctl", f"--root={tree}", "list-unit-files"], (lambda s: parse_unit_files(s, state)))
+
+    # Since systemd v246, some services previously reported as "enabled" /
+    # "disabled" are now reported as "alias". There is no systemd command, that
+    # would take an "alias" unit and report its state as enabled/disabled
+    # and could run on a different tree (with "--root" option).
+    # To make the produced list of services in the given state consistent on
+    # pre/post v246 systemd versions, check all "alias" units and append them
+    # to the list, if their target is also listed in 'services_state'.
+    if state != "alias":
+        services_alias = subprocess_check_output(["systemctl", f"--root={tree}", "list-unit-files"], (lambda s: parse_unit_files(s, "alias")))
+
+        for alias in services_alias:
+            # The service may be in one of the following places (output of
+            # "systemd-analyze unit-paths", it should not change too often).
+            unit_paths = [
+                "/etc/systemd/system.control",
+                "/run/systemd/system.control",
+                "/run/systemd/transient",
+                "/run/systemd/generator.early",
+                "/etc/systemd/system",
+                "/run/systemd/system",
+                "/run/systemd/generator",
+                "/usr/local/lib/systemd/system",
+                "/usr/lib/systemd/system",
+                "/run/systemd/generator.late"
+            ]
+
+            with change_root(tree):
+                for path in unit_paths:
+                    unit_path = os.path.join(path, alias)
+                    if os.path.exists(unit_path):
+                        real_unit_path = os.path.realpath(unit_path)
+                        # Skip the alias, if there was a symlink cycle.
+                        # When symbolic link cycles occur, the returned path will
+                        # be one member of the cycle, but no guarantee is made about
+                        # which member that will be.
+                        if os.path.islink(real_unit_path):
+                            continue
+
+                        # Append the alias unit to the list, if its target is
+                        # already there.
+                        if os.path.basename(real_unit_path) in services_state:
+                            services_state.append(alias)
+
+    # deduplicate and sort
+    services_state = list(set(services_state))
+    services_state.sort()
+
+    return services_state
 
 
 def read_default_target(tree):
diff --git a/tools/provision.sh b/tools/provision.sh
index f90f2419e..1ed1e4478 100755
--- a/tools/provision.sh
+++ b/tools/provision.sh
@@ -37,81 +37,9 @@ CERTDIR=/etc/osbuild-composer
 OPENSSL_CONFIG=/usr/share/tests/osbuild-composer/x509/openssl.cnf
 CADIR=/etc/osbuild-composer-test/ca
 
-# The $CADIR might exist from a previous test (current Schutzbot's imperfection)
-sudo rm -rf $CADIR || true
-sudo mkdir -p $CADIR
-
-pushd $CADIR
-    sudo mkdir certs private
-    sudo touch index.txt
-
-    # Generate a CA.
-    sudo openssl req -config $OPENSSL_CONFIG \
-        -keyout private/ca.key.pem \
-        -new -nodes -x509 -extensions osbuild_ca_ext \
-        -out ca.cert.pem -subj "/CN=osbuild.org"
-
-    # Copy the private key to the location expected by the tests
-    sudo cp ca.cert.pem "$CERTDIR"/ca-crt.pem
-
-    # Generate a composer certificate.
-    sudo openssl req -config $OPENSSL_CONFIG \
-        -keyout "$CERTDIR"/composer-key.pem \
-        -new -nodes \
-        -out /tmp/composer-csr.pem \
-        -subj "/CN=localhost/emailAddress=osbuild@example.com" \
-        -addext "subjectAltName=DNS:localhost"
-
-    sudo openssl ca -batch -config $OPENSSL_CONFIG \
-        -extensions osbuild_server_ext \
-        -in /tmp/composer-csr.pem \
-        -out "$CERTDIR"/composer-crt.pem
-
-    sudo chown _osbuild-composer "$CERTDIR"/composer-*.pem
-
-    # Generate a worker certificate.
-    sudo openssl req -config $OPENSSL_CONFIG \
-        -keyout "$CERTDIR"/worker-key.pem \
-        -new -nodes \
-        -out /tmp/worker-csr.pem \
-        -subj "/CN=localhost/emailAddress=osbuild@example.com" \
-        -addext "subjectAltName=DNS:localhost"
-
-    sudo openssl ca -batch -config $OPENSSL_CONFIG \
-        -extensions osbuild_client_ext \
-        -in /tmp/worker-csr.pem \
-        -out "$CERTDIR"/worker-crt.pem
-
-    # Generate a client certificate.
-    sudo openssl req -config $OPENSSL_CONFIG \
-        -keyout "$CERTDIR"/client-key.pem \
-        -new -nodes \
-        -out /tmp/client-csr.pem \
-        -subj "/CN=client.osbuild.org/emailAddress=osbuild@example.com" \
-        -addext "subjectAltName=DNS:client.osbuild.org"
-
-    sudo openssl ca -batch -config $OPENSSL_CONFIG \
-        -extensions osbuild_client_ext \
-        -in /tmp/client-csr.pem \
-        -out "$CERTDIR"/client-crt.pem
-
-    # Client keys are used by tests to access the composer APIs. Allow all users access.
-    sudo chmod 644 "$CERTDIR"/client-key.pem
-
-    # Generate a kojihub certificate.
-    sudo openssl req -config $OPENSSL_CONFIG \
-        -keyout "$CERTDIR"/kojihub-key.pem \
-        -new -nodes \
-        -out /tmp/kojihub-csr.pem \
-        -subj "/CN=localhost/emailAddress=osbuild@example.com" \
-        -addext "subjectAltName=DNS:localhost"
-
-    sudo openssl ca -batch -config $OPENSSL_CONFIG \
-        -extensions osbuild_server_ext \
-        -in /tmp/kojihub-csr.pem \
-        -out "$CERTDIR"/kojihub-crt.pem
-
-popd
+scriptloc=$(dirname "$0")
+sudo "${scriptloc}/gen-certs.sh" "${OPENSSL_CONFIG}" "${CERTDIR}" "${CADIR}"
+sudo chown _osbuild-composer "${CERTDIR}"/composer-*.pem
 
 sudo systemctl start osbuild-remote-worker.socket
 sudo systemctl start osbuild-composer.socket


Test patch:
diff --git a/internal/client/source_test.go b/internal/client/source_test.go
index 8d26101ad..0040ae397 100644
--- a/internal/client/source_test.go
+++ b/internal/client/source_test.go
@@ -9,6 +9,7 @@
 package client
 
 import (
+	"fmt"
 	"strings"
 	"testing"
 
@@ -88,6 +89,73 @@ func TestPOSTInvalidJSONSourceV1(t *testing.T) {
 	require.False(t, resp.Status, "did not return an error")
 }
 
+// POST a JSON system source using V0 API
+func TestPOSTSystemJSONSourceV0(t *testing.T) {
+	sources_list, api, err := ListSourcesV0(testState.socket)
+	require.NoError(t, err, "GET source failed with a client error")
+	require.Nil(t, api, "ListSources failed: %#v", api)
+
+	// Cannot override system sources
+	source := `{
+		"name": "REPO-NAME",
+		"url": "file://REPO-PATH",
+		"type": "yum-baseurl",
+		"proxy": "https://proxy-url/",
+		"check_ssl": true,
+		"check_gpg": true,
+		"gpgkey_urls": ["https://url/path/to/gpg-key"]
+	}`
+
+	for _, repoName := range []string{"test-system-repo", "fedora", "baseos"} {
+		// skip repository names which are not present b/c this test can be
+		// executed both as a unit test and as an integration test
+		if !Include(sources_list, repoName) {
+			continue
+		}
+		useSource := strings.Replace(source, "REPO-NAME", repoName, 1)
+
+		resp, err := PostJSONSourceV0(testState.socket, useSource)
+		require.NoError(t, err, "POST source failed with a client error")
+		require.False(t, resp.Status, "did not return an error")
+		msg := fmt.Sprintf("%s is a system source, it cannot be changed.", repoName)
+		require.Equal(t, APIErrorMsg{ID: "SystemSource", Msg: msg}, resp.Errors[0])
+	}
+}
+
+// POST a JSON system source using V1 API
+func TestPOSTSystemJSONSourceV1(t *testing.T) {
+	sources_list, api, err := ListSourcesV1(testState.socket)
+	require.NoError(t, err, "GET source failed with a client error")
+	require.Nil(t, api, "ListSources failed: %#v", api)
+
+	// Cannot override system sources
+	source := `{
+        "id": "REPO-NAME",
+		"name": "json package system repo",
+		"url": "file://REPO-PATH",
+		"type": "yum-baseurl",
+		"proxy": "https://proxy-url/",
+		"check_ssl": true,
+		"check_gpg": true,
+		"gpgkey_urls": ["https://url/path/to/gpg-key"]
+	}`
+
+	for _, repoName := range []string{"test-system-repo", "fedora", "baseos"} {
+		// skip repository names which are not present b/c this test can be
+		// executed both as a unit test and as an integration test
+		if !Include(sources_list, repoName) {
+			continue
+		}
+		useSource := strings.Replace(source, "REPO-NAME", repoName, 1)
+
+		resp, err := PostJSONSourceV1(testState.socket, useSource)
+		require.NoError(t, err, "POST source failed with a client error")
+		require.False(t, resp.Status, "did not return an error")
+		msg := fmt.Sprintf("%s is a system source, it cannot be changed.", repoName)
+		require.Equal(t, APIErrorMsg{ID: "SystemSource", Msg: msg}, resp.Errors[0])
+	}
+}
+
 // POST a new TOML source using V0 API
 func TestPOSTTOMLSourceV0(t *testing.T) {
 	source := `
@@ -223,6 +291,71 @@ func TestPOSTWrongTOMLSourceV1(t *testing.T) {
 	require.False(t, resp.Status, "did not return an error")
 }
 
+// POST a TOML system source using V0 API
+func TestPOSTTOMLSystemSourceV0(t *testing.T) {
+	sources_list, api, err := ListSourcesV0(testState.socket)
+	require.NoError(t, err, "GET source failed with a client error")
+	require.Nil(t, api, "ListSources failed: %#v", api)
+
+	source := `
+		name = "REPO-NAME"
+		url = "file://REPO-PATH"
+		type = "yum-baseurl"
+		proxy = "https://proxy-url/"
+		check_ssl = true
+		check_gpg = true
+		gpgkey_urls = ["https://url/path/to/gpg-key"]
+	`
+	source = strings.Replace(source, "REPO-PATH", testState.repoDir, 1)
+	for _, repoName := range []string{"test-system-repo", "fedora", "baseos"} {
+		// skip repository names which are not present b/c this test can be
+		// executed both as a unit test and as an integration test
+		if !Include(sources_list, repoName) {
+			continue
+		}
+		useSource := strings.Replace(source, "REPO-NAME", repoName, 1)
+
+		resp, err := PostTOMLSourceV0(testState.socket, useSource)
+		require.NoError(t, err, "POST source failed with a client error")
+		require.False(t, resp.Status, "did not return an error")
+		msg := fmt.Sprintf("%s is a system source, it cannot be changed.", repoName)
+		require.Equal(t, APIErrorMsg{ID: "SystemSource", Msg: msg}, resp.Errors[0])
+	}
+}
+
+// POST a new TOML system source using V1 API
+func TestPOSTTOMLSystemSourceV1(t *testing.T) {
+	sources_list, api, err := ListSourcesV1(testState.socket)
+	require.NoError(t, err, "GET source failed with a client error")
+	require.Nil(t, api, "ListSources failed: %#v", api)
+
+	source := `
+		id = "REPO-NAME"
+		name = "toml package repo"
+		url = "file://REPO-PATH"
+		type = "yum-baseurl"
+		proxy = "https://proxy-url/"
+		check_ssl = true
+		check_gpg = true
+		gpgkey_urls = ["https://url/path/to/gpg-key"]
+	`
+	source = strings.Replace(source, "REPO-PATH", testState.repoDir, 1)
+	for _, repoName := range []string{"test-system-repo", "fedora", "baseos"} {
+		// skip repository names which are not present b/c this test can be
+		// executed both as a unit test and as an integration test
+		if !Include(sources_list, repoName) {
+			continue
+		}
+		useSource := strings.Replace(source, "REPO-NAME", repoName, 1)
+
+		resp, err := PostTOMLSourceV1(testState.socket, useSource)
+		require.NoError(t, err, "POST source failed with a client error")
+		require.False(t, resp.Status, "did not return an error")
+		msg := fmt.Sprintf("%s is a system source, it cannot be changed.", repoName)
+		require.Equal(t, APIErrorMsg{ID: "SystemSource", Msg: msg}, resp.Errors[0])
+	}
+}
+
 // list sources using the v0 API
 func TestListSourcesV0(t *testing.T) {
 	sources := []string{`{
diff --git a/internal/weldr/api_test.go b/internal/weldr/api_test.go
index 9f32dec3e..8af9c1fc8 100644
--- a/internal/weldr/api_test.go
+++ b/internal/weldr/api_test.go
@@ -925,6 +925,8 @@ func TestSourcesNew(t *testing.T) {
 		{"POST", "/api/v0/projects/source/new", `{"url": "https://download.opensuse.org/repositories/shells:/fish:/release:/3/Fedora_29/","type": "yum-baseurl","check_ssl": false,"check_gpg": false}`, http.StatusBadRequest, `{"errors": [{"id": "ProjectsError","msg": "Problem parsing POST body: 'name' field is missing from request"}],"status":false}`},
 		{"POST", "/api/v0/projects/source/new", `{"name": "fish", "type": "yum-baseurl","check_ssl": false,"check_gpg": false}`, http.StatusBadRequest, `{"errors": [{"id": "ProjectsError","msg": "Problem parsing POST body: 'url' field is missing from request"}],"status":false}`},
 		{"POST", "/api/v0/projects/source/new", `{"name": "fish", "url": "https://download.opensuse.org/repositories/shells:/fish:/release:/3/Fedora_29/","check_ssl": false,"check_gpg": false}`, http.StatusBadRequest, `{"errors": [{"id": "ProjectsError","msg": "Problem parsing POST body: 'type' field is missing from request"}],"status":false}`},
+		{"POST", "/api/v0/projects/source/new", `{"name": "test-id", "url": "https://download.opensuse.org/repositories/shells:/fish:/release:/3/Fedora_29/","type": "yum-baseurl","check_ssl": false,"check_gpg": false}`, http.StatusBadRequest, `{"errors": [{"id": "SystemSource","msg": "test-id is a system source, it cannot be changed."}],"status":false}`},
+		{"POST", "/api/v1/projects/source/new", `{"id": "test-id", "name": "test system repo", "url": "https://download.opensuse.org/repositories/shells:/fish:/release:/3/Fedora_29/","type": "yum-baseurl","check_ssl": false,"check_gpg": false}`, http.StatusBadRequest, `{"errors": [{"id": "SystemSource","msg": "test-id is a system source, it cannot be changed."}],"status":false}`},
 	}
 
 	tempdir, err := ioutil.TempDir("", "weldr-tests-")
diff --git a/test/README.md b/test/README.md
index d04571cea..99cbd7d0b 100644
--- a/test/README.md
+++ b/test/README.md
@@ -69,6 +69,20 @@ boot-test the image.
 
 To (re)generate these test cases use the tool
 `tools/test-case-generators/generate-test-cases`.
+Note that the `generate-test-cases` tool must be run on a host with
+the same architecture, as the one intended for the generated test
+cases. In other words, you need to generate e.g test cases for `aarch64`
+images on an `aarch64` host.
+
+Alternatively to (re)generate test cases for all architectures, or just
+the ones different from your host's architecture, you can use the tool
+`tools/test-case-generators/generate-all-test-cases`. It creates
+an ephemeral virtual machine for each necessary architecture using the
+`qemu-system-<arch>` command and generates test cases using the
+`generate-test-cases` tool inside the virtual machine. It is important
+to note that test case generation in virtual machines may take several
+hours. The `generate-all-test-cases` currently does not work with RHEL
+images because of missing "9p" filesystem support.
 
 ### Setting up Azure upload tests
 
diff --git a/test/data/manifests/fedora_33-aarch64-ami-boot.json b/test/data/manifests/fedora_33-aarch64-ami-boot.json
index 029c829d5..c3c4dbd3a 100644
--- a/test/data/manifests/fedora_33-aarch64-ami-boot.json
+++ b/test/data/manifests/fedora_33-aarch64-ami-boot.json
@@ -9817,12 +9817,12 @@
         "version": "5.8.15-301.fc33.aarch64"
       }
     ],
+    "default-target": "graphical.target",
     "firewall-enabled": [
       "ssh",
       "mdns",
       "dhcpv6-client"
     ],
-    "default-target": "graphical.target",
     "fstab": [
       [
         "UUID=46BB-8120",
@@ -10372,6 +10372,7 @@
       "rdisc.service",
       "rdma.service",
       "remote-cryptsetup.target",
+      "runlevel0.target",
       "serial-getty@.service",
       "sshd-keygen@.service",
       "sshd.socket",
@@ -10397,9 +10398,16 @@
       "NetworkManager-wait-online.service",
       "NetworkManager.service",
       "auditd.service",
+      "autovt@.service",
       "chronyd.service",
       "cloud-init.service",
+      "ctrl-alt-del.target",
       "dbus-broker.service",
+      "dbus-org.fedoraproject.FirewallD1.service",
+      "dbus-org.freedesktop.home1.service",
+      "dbus-org.freedesktop.nm-dispatcher.service",
+      "dbus-org.freedesktop.resolve1.service",
+      "dbus.service",
       "dbus.socket",
       "dnf-makecache.timer",
       "firewalld.service",
@@ -10409,6 +10417,7 @@
       "reboot.target",
       "remote-fs.target",
       "rpmdb-rebuild.service",
+      "runlevel6.target",
       "selinux-autorelabel-mark.service",
       "sshd.service",
       "sssd-kcm.socket",
diff --git a/test/data/manifests/fedora_33-x86_64-ami-boot.json b/test/data/manifests/fedora_33-x86_64-ami-boot.json
index 825e37472..26c838f06 100644
--- a/test/data/manifests/fedora_33-x86_64-ami-boot.json
+++ b/test/data/manifests/fedora_33-x86_64-ami-boot.json
@@ -9396,12 +9396,12 @@
         "version": "5.8.15-301.fc33.x86_64"
       }
     ],
+    "default-target": "graphical.target",
     "firewall-enabled": [
       "ssh",
       "mdns",
       "dhcpv6-client"
     ],
-    "default-target": "graphical.target",
     "fstab": [
       [
         "UUID=76a22bf4-f153-4541-b6c7-0332c0dfaeac",
@@ -9904,6 +9904,7 @@
       "rdisc.service",
       "rdma.service",
       "remote-cryptsetup.target",
+      "runlevel0.target",
       "serial-getty@.service",
       "sshd-keygen@.service",
       "sshd.socket",
@@ -9929,9 +9930,16 @@
       "NetworkManager-wait-online.service",
       "NetworkManager.service",
       "auditd.service",
+      "autovt@.service",
       "chronyd.service",
       "cloud-init.service",
+      "ctrl-alt-del.target",
       "dbus-broker.service",
+      "dbus-org.fedoraproject.FirewallD1.service",
+      "dbus-org.freedesktop.home1.service",
+      "dbus-org.freedesktop.nm-dispatcher.service",
+      "dbus-org.freedesktop.resolve1.service",
+      "dbus.service",
       "dbus.socket",
       "dnf-makecache.timer",
       "firewalld.service",
@@ -9941,6 +9949,7 @@
       "reboot.target",
       "remote-fs.target",
       "rpmdb-rebuild.service",
+      "runlevel6.target",
       "selinux-autorelabel-mark.service",
       "sshd.service",
       "sssd-kcm.socket",
diff --git a/test/data/manifests/fedora_33-x86_64-fedora_iot_commit-boot.json b/test/data/manifests/fedora_33-x86_64-fedora_iot_commit-boot.json
index ca77d2257..ce3366bac 100644
--- a/test/data/manifests/fedora_33-x86_64-fedora_iot_commit-boot.json
+++ b/test/data/manifests/fedora_33-x86_64-fedora_iot_commit-boot.json
@@ -11143,6 +11143,7 @@
       "remote-cryptsetup.target",
       "rpm-ostree-bootstatus.service",
       "rpm-ostreed-automatic.timer",
+      "runlevel0.target",
       "serial-getty@.service",
       "sshd-keygen@.service",
       "sshd.socket",
@@ -11164,10 +11165,19 @@
       "NetworkManager-wait-online.service",
       "NetworkManager.service",
       "auditd.service",
+      "autovt@.service",
       "bluetooth.service",
       "chronyd.service",
+      "ctrl-alt-del.target",
       "dbus-broker.service",
+      "dbus-org.bluez.service",
+      "dbus-org.fedoraproject.FirewallD1.service",
+      "dbus-org.freedesktop.ModemManager1.service",
+      "dbus-org.freedesktop.home1.service",
+      "dbus-org.freedesktop.nm-dispatcher.service",
+      "dbus-org.freedesktop.resolve1.service",
       "dbus-parsec.service",
+      "dbus.service",
       "dbus.socket",
       "dm-event.socket",
       "firewalld.service",
@@ -11190,6 +11200,7 @@
       "remote-fs.target",
       "rngd.service",
       "rpmdb-rebuild.service",
+      "runlevel6.target",
       "selinux-autorelabel-mark.service",
       "sshd.service",
       "systemd-homed.service",
diff --git a/test/data/manifests/fedora_33-x86_64-openstack-boot.json b/test/data/manifests/fedora_33-x86_64-openstack-boot.json
index b332bade7..ee3ba22df 100644
--- a/test/data/manifests/fedora_33-x86_64-openstack-boot.json
+++ b/test/data/manifests/fedora_33-x86_64-openstack-boot.json
@@ -9711,12 +9711,12 @@
         "version": "5.8.15-301.fc33.x86_64"
       }
     ],
+    "default-target": "graphical.target",
     "firewall-enabled": [
       "ssh",
       "mdns",
       "dhcpv6-client"
     ],
-    "default-target": "graphical.target",
     "fstab": [
       [
         "UUID=76a22bf4-f153-4541-b6c7-0332c0dfaeac",
@@ -10235,6 +10235,7 @@
       "rdisc.service",
       "rdma.service",
       "remote-cryptsetup.target",
+      "runlevel0.target",
       "serial-getty@.service",
       "sshd-keygen@.service",
       "sshd.socket",
@@ -10260,12 +10261,19 @@
       "NetworkManager-wait-online.service",
       "NetworkManager.service",
       "auditd.service",
+      "autovt@.service",
       "chronyd.service",
       "cloud-config.service",
       "cloud-final.service",
       "cloud-init-local.service",
       "cloud-init.service",
+      "ctrl-alt-del.target",
       "dbus-broker.service",
+      "dbus-org.fedoraproject.FirewallD1.service",
+      "dbus-org.freedesktop.home1.service",
+      "dbus-org.freedesktop.nm-dispatcher.service",
+      "dbus-org.freedesktop.resolve1.service",
+      "dbus.service",
       "dbus.socket",
       "dnf-makecache.timer",
       "firewalld.service",
@@ -10276,6 +10284,7 @@
       "reboot.target",
       "remote-fs.target",
       "rpmdb-rebuild.service",
+      "runlevel6.target",
       "selinux-autorelabel-mark.service",
       "sshd.service",
       "sssd-kcm.socket",
diff --git a/test/data/manifests/fedora_33-x86_64-qcow2-boot.json b/test/data/manifests/fedora_33-x86_64-qcow2-boot.json
index 82db9e53e..a5fe9f963 100644
--- a/test/data/manifests/fedora_33-x86_64-qcow2-boot.json
+++ b/test/data/manifests/fedora_33-x86_64-qcow2-boot.json
@@ -9790,6 +9790,7 @@
       "rdisc.service",
       "rdma.service",
       "remote-cryptsetup.target",
+      "runlevel0.target",
       "serial-getty@.service",
       "sshd-keygen@.service",
       "sshd.socket",
@@ -9815,12 +9816,18 @@
       "NetworkManager-wait-online.service",
       "NetworkManager.service",
       "auditd.service",
+      "autovt@.service",
       "chronyd.service",
       "cloud-config.service",
       "cloud-final.service",
       "cloud-init-local.service",
       "cloud-init.service",
+      "ctrl-alt-del.target",
       "dbus-broker.service",
+      "dbus-org.freedesktop.home1.service",
+      "dbus-org.freedesktop.nm-dispatcher.service",
+      "dbus-org.freedesktop.resolve1.service",
+      "dbus.service",
       "dbus.socket",
       "dnf-makecache.timer",
       "fstrim.timer",
@@ -9829,6 +9836,7 @@
       "reboot.target",
       "remote-fs.target",
       "rpmdb-rebuild.service",
+      "runlevel6.target",
       "selinux-autorelabel-mark.service",
       "sshd.service",
       "sssd-kcm.socket",
diff --git a/test/data/manifests/fedora_33-x86_64-qcow2-customize.json b/test/data/manifests/fedora_33-x86_64-qcow2-customize.json
index ac9d7a327..06d7a9eb4 100644
--- a/test/data/manifests/fedora_33-x86_64-qcow2-customize.json
+++ b/test/data/manifests/fedora_33-x86_64-qcow2-customize.json
@@ -9892,6 +9892,7 @@
       "rdisc.service",
       "rdma.service",
       "remote-cryptsetup.target",
+      "runlevel0.target",
       "serial-getty@.service",
       "sshd-keygen@.service",
       "sssd-autofs.socket",
@@ -9916,12 +9917,18 @@
       "NetworkManager-wait-online.service",
       "NetworkManager.service",
       "auditd.service",
+      "autovt@.service",
       "chronyd.service",
       "cloud-config.service",
       "cloud-final.service",
       "cloud-init-local.service",
       "cloud-init.service",
+      "ctrl-alt-del.target",
       "dbus-broker.service",
+      "dbus-org.freedesktop.home1.service",
+      "dbus-org.freedesktop.nm-dispatcher.service",
+      "dbus-org.freedesktop.resolve1.service",
+      "dbus.service",
       "dbus.socket",
       "dnf-makecache.timer",
       "fstrim.timer",
@@ -9930,6 +9937,7 @@
       "reboot.target",
       "remote-fs.target",
       "rpmdb-rebuild.service",
+      "runlevel6.target",
       "selinux-autorelabel-mark.service",
       "sshd.service",
       "sshd.socket",
diff --git a/test/data/manifests/fedora_33-x86_64-vhd-boot.json b/test/data/manifests/fedora_33-x86_64-vhd-boot.json
index 605acb153..4d1e98551 100644
--- a/test/data/manifests/fedora_33-x86_64-vhd-boot.json
+++ b/test/data/manifests/fedora_33-x86_64-vhd-boot.json
@@ -8997,12 +8997,12 @@
         "version": "5.8.15-301.fc33.x86_64"
       }
     ],
+    "default-target": "graphical.target",
     "firewall-enabled": [
       "ssh",
       "mdns",
       "dhcpv6-client"
     ],
-    "default-target": "graphical.target",
     "fstab": [
       [
         "UUID=76a22bf4-f153-4541-b6c7-0332c0dfaeac",
@@ -9479,6 +9479,7 @@
       "rdisc.service",
       "rdma.service",
       "remote-cryptsetup.target",
+      "runlevel0.target",
       "serial-getty@.service",
       "sshd-keygen@.service",
       "sshd.socket",
@@ -9504,8 +9505,15 @@
       "NetworkManager-wait-online.service",
       "NetworkManager.service",
       "auditd.service",
+      "autovt@.service",
       "chronyd.service",
+      "ctrl-alt-del.target",
       "dbus-broker.service",
+      "dbus-org.fedoraproject.FirewallD1.service",
+      "dbus-org.freedesktop.home1.service",
+      "dbus-org.freedesktop.nm-dispatcher.service",
+      "dbus-org.freedesktop.resolve1.service",
+      "dbus.service",
       "dbus.socket",
       "dnf-makecache.timer",
       "firewalld.service",
@@ -9515,6 +9523,7 @@
       "reboot.target",
       "remote-fs.target",
       "rpmdb-rebuild.service",
+      "runlevel6.target",
       "selinux-autorelabel-mark.service",
       "sshd.service",
       "sssd-kcm.socket",
diff --git a/test/data/manifests/fedora_33-x86_64-vmdk-boot.json b/test/data/manifests/fedora_33-x86_64-vmdk-boot.json
index 0e07e87b7..25c40a293 100644
--- a/test/data/manifests/fedora_33-x86_64-vmdk-boot.json
+++ b/test/data/manifests/fedora_33-x86_64-vmdk-boot.json
@@ -9790,6 +9790,7 @@
       "rdisc.service",
       "rdma.service",
       "remote-cryptsetup.target",
+      "runlevel0.target",
       "serial-getty@.service",
       "sshd-keygen@.service",
       "sshd.socket",
@@ -9815,12 +9816,18 @@
       "NetworkManager-wait-online.service",
       "NetworkManager.service",
       "auditd.service",
+      "autovt@.service",
       "chronyd.service",
       "cloud-config.service",
       "cloud-final.service",
       "cloud-init-local.service",
       "cloud-init.service",
+      "ctrl-alt-del.target",
       "dbus-broker.service",
+      "dbus-org.freedesktop.home1.service",
+      "dbus-org.freedesktop.nm-dispatcher.service",
+      "dbus-org.freedesktop.resolve1.service",
+      "dbus.service",
       "dbus.socket",
       "dnf-makecache.timer",
       "fstrim.timer",
@@ -9829,6 +9836,7 @@
       "reboot.target",
       "remote-fs.target",
       "rpmdb-rebuild.service",
+      "runlevel6.target",
       "selinux-autorelabel-mark.service",
       "sshd.service",
       "sssd-kcm.socket",
diff --git a/test/data/manifests/rhel_84-x86_64-qcow2-boot.json b/test/data/manifests/rhel_84-x86_64-qcow2-boot.json
index df83a71e5..7a313cc18 100644
--- a/test/data/manifests/rhel_84-x86_64-qcow2-boot.json
+++ b/test/data/manifests/rhel_84-x86_64-qcow2-boot.json
@@ -828,9 +828,6 @@
           "sha256:9d2a2ef762444163e91c433c83ce91ef37acb2318ba168a81ac0dd4d65b834ef": {
             "url": "https://rpmrepo.osbuild.org/v1/psi/el8/el8-x86_64-baseos-8.4.0.n-20201010/Packages/subscription-manager-1.27.16-1.el8.x86_64.rpm"
           },
-          "sha256:9d9d1bdfba4cd852f90232ebe2c4247e7b3a88c4476578c820788a91d6838e55": {
-            "url": "https://rpmrepo.osbuild.org/v1/psi/el8/el8-x86_64-baseos-8.4.0.n-20201010/Packages/rng-tools-6.8-3.el8.x86_64.rpm"
-          },
           "sha256:9e08c0338eac83abf9a0118cb05fb646d4554c1b2513ab6801e9587aede40b28": {
             "url": "https://rpmrepo.osbuild.org/v1/psi/el8/el8-x86_64-appstream-8.4.0.n-20201010/Packages/python3-babel-2.5.1-5.el8.noarch.rpm"
           },
@@ -3097,9 +3094,6 @@
               {
                 "checksum": "sha256:9be187fd62ada3a61cb494a383b8a60fd6c705c93897a13051600dbbeb1d914f"
               },
-              {
-                "checksum": "sha256:9d9d1bdfba4cd852f90232ebe2c4247e7b3a88c4476578c820788a91d6838e55"
-              },
               {
                 "checksum": "sha256:ad060f60303b6764e08657436147cc24cfc4bb1ccc1431796407ba005a7bcec0"
               },
@@ -3460,9 +3454,6 @@
         {
           "name": "org.osbuild.systemd",
           "options": {
-            "enabled_services": [
-              "rngd.service"
-            ],
             "default_target": "multi-user.target"
           }
         },
@@ -8554,15 +8545,6 @@
         "remote_location": "https://rpmrepo.osbuild.org/v1/psi/el8/el8-x86_64-baseos-8.4.0.n-20201010/Packages/rhsm-icons-1.27.16-1.el8.noarch.rpm",
         "checksum": "sha256:9be187fd62ada3a61cb494a383b8a60fd6c705c93897a13051600dbbeb1d914f"
       },
-      {
-        "name": "rng-tools",
-        "epoch": 0,
-        "version": "6.8",
-        "release": "3.el8",
-        "arch": "x86_64",
-        "remote_location": "https://rpmrepo.osbuild.org/v1/psi/el8/el8-x86_64-baseos-8.4.0.n-20201010/Packages/rng-tools-6.8-3.el8.x86_64.rpm",
-        "checksum": "sha256:9d9d1bdfba4cd852f90232ebe2c4247e7b3a88c4476578c820788a91d6838e55"
-      },
       {
         "name": "rootfiles",
         "epoch": 0,
@@ -9524,7 +9506,6 @@
       "polkitd:x:996:",
       "redhat:x:1000:",
       "render:x:998:",
-      "rngd:x:988:",
       "root:x:0:",
       "rpc:x:32:",
       "rpcuser:x:29:",
@@ -9952,7 +9933,6 @@
       "redhat-release-8.4-0.5.el8.x86_64",
       "redhat-release-eula-8.4-0.5.el8.x86_64",
       "rhsm-icons-1.27.16-1.el8.noarch",
-      "rng-tools-6.8-3.el8.x86_64",
       "rootfiles-8.1-22.el8.noarch",
       "rpcbind-1.2.5-7.el8.x86_64",
       "rpm-4.14.3-4.el8.x86_64",
@@ -10064,7 +10044,6 @@
       "operator:x:11:0:operator:/root:/sbin/nologin",
       "polkitd:x:998:996:User for polkitd:/:/sbin/nologin",
       "redhat:x:1000:1000::/home/redhat:/bin/bash",
-      "rngd:x:991:988:Random Number Generator Daemon:/var/lib/rngd:/sbin/nologin",
       "root:x:0:0:root:/root:/bin/bash",
       "rpc:x:32:32:Rpcbind Daemon:/var/lib/rpcbind:/sbin/nologin",
       "rpcuser:x:29:29:RPC Service User:/var/lib/nfs:/sbin/nologin",
@@ -10120,7 +10099,6 @@
       "remote-cryptsetup.target",
       "rhsm-facts.service",
       "rhsm.service",
-      "rngd-wake-threshold.service",
       "runlevel0.target",
       "runlevel6.target",
       "serial-getty@.service",
@@ -10163,7 +10141,6 @@
       "nis-domainname.service",
       "remote-fs.target",
       "rhsmcertd.service",
-      "rngd.service",
       "rpcbind.service",
       "rpcbind.socket",
       "rsyslog.service",
diff --git a/test/data/manifests/rhel_84-x86_64-qcow2-customize.json b/test/data/manifests/rhel_84-x86_64-qcow2-customize.json
index a6e28f48a..a526971e6 100644
--- a/test/data/manifests/rhel_84-x86_64-qcow2-customize.json
+++ b/test/data/manifests/rhel_84-x86_64-qcow2-customize.json
@@ -880,9 +880,6 @@
           "sha256:9d2a2ef762444163e91c433c83ce91ef37acb2318ba168a81ac0dd4d65b834ef": {
             "url": "https://rpmrepo.osbuild.org/v1/psi/el8/el8-x86_64-baseos-8.4.0.n-20201010/Packages/subscription-manager-1.27.16-1.el8.x86_64.rpm"
           },
-          "sha256:9d9d1bdfba4cd852f90232ebe2c4247e7b3a88c4476578c820788a91d6838e55": {
-            "url": "https://rpmrepo.osbuild.org/v1/psi/el8/el8-x86_64-baseos-8.4.0.n-20201010/Packages/rng-tools-6.8-3.el8.x86_64.rpm"
-          },
           "sha256:9e08c0338eac83abf9a0118cb05fb646d4554c1b2513ab6801e9587aede40b28": {
             "url": "https://rpmrepo.osbuild.org/v1/psi/el8/el8-x86_64-appstream-8.4.0.n-20201010/Packages/python3-babel-2.5.1-5.el8.noarch.rpm"
           },
@@ -3149,9 +3146,6 @@
               {
                 "checksum": "sha256:9be187fd62ada3a61cb494a383b8a60fd6c705c93897a13051600dbbeb1d914f"
               },
-              {
-                "checksum": "sha256:9d9d1bdfba4cd852f90232ebe2c4247e7b3a88c4476578c820788a91d6838e55"
-              },
               {
                 "checksum": "sha256:ad060f60303b6764e08657436147cc24cfc4bb1ccc1431796407ba005a7bcec0"
               },
@@ -3554,7 +3548,6 @@
           "name": "org.osbuild.systemd",
           "options": {
             "enabled_services": [
-              "rngd.service",
               "sshd.socket"
             ],
             "disabled_services": [
@@ -8651,15 +8644,6 @@
         "remote_location": "https://rpmrepo.osbuild.org/v1/psi/el8/el8-x86_64-baseos-8.4.0.n-20201010/Packages/rhsm-icons-1.27.16-1.el8.noarch.rpm",
         "checksum": "sha256:9be187fd62ada3a61cb494a383b8a60fd6c705c93897a13051600dbbeb1d914f"
       },
-      {
-        "name": "rng-tools",
-        "epoch": 0,
-        "version": "6.8",
-        "release": "3.el8",
-        "arch": "x86_64",
-        "remote_location": "https://rpmrepo.osbuild.org/v1/psi/el8/el8-x86_64-baseos-8.4.0.n-20201010/Packages/rng-tools-6.8-3.el8.x86_64.rpm",
-        "checksum": "sha256:9d9d1bdfba4cd852f90232ebe2c4247e7b3a88c4476578c820788a91d6838e55"
-      },
       {
         "name": "rootfiles",
         "epoch": 0,
@@ -9622,7 +9606,6 @@
       "nobody:x:65534:",
       "polkitd:x:996:",
       "render:x:998:",
-      "rngd:x:988:",
       "root:x:0:",
       "rpc:x:32:",
       "rpcuser:x:29:",
@@ -10051,7 +10034,6 @@
       "redhat-release-8.4-0.5.el8.x86_64",
       "redhat-release-eula-8.4-0.5.el8.x86_64",
       "rhsm-icons-1.27.16-1.el8.noarch",
-      "rng-tools-6.8-3.el8.x86_64",
       "rootfiles-8.1-22.el8.noarch",
       "rpcbind-1.2.5-7.el8.x86_64",
       "rpm-4.14.3-4.el8.x86_64",
@@ -10162,7 +10144,6 @@
       "nobody:x:65534:65534:Kernel Overflow User:/:/sbin/nologin",
       "operator:x:11:0:operator:/root:/sbin/nologin",
       "polkitd:x:998:996:User for polkitd:/:/sbin/nologin",
-      "rngd:x:991:988:Random Number Generator Daemon:/var/lib/rngd:/sbin/nologin",
       "root:x:0:0:root:/root:/bin/bash",
       "rpc:x:32:32:Rpcbind Daemon:/var/lib/rpcbind:/sbin/nologin",
       "rpcuser:x:29:29:RPC Service User:/var/lib/nfs:/sbin/nologin",
@@ -10221,7 +10202,6 @@
       "remote-cryptsetup.target",
       "rhsm-facts.service",
       "rhsm.service",
-      "rngd-wake-threshold.service",
       "runlevel0.target",
       "runlevel6.target",
       "serial-getty@.service",
@@ -10263,7 +10243,6 @@
       "nis-domainname.service",
       "remote-fs.target",
       "rhsmcertd.service",
-      "rngd.service",
       "rpcbind.service",
       "rpcbind.socket",
       "rsyslog.service",
diff --git a/tools/deploy/gen-test-data/files/run/provision-scripts/deploy.sh b/tools/deploy/gen-test-data/files/run/provision-scripts/deploy.sh
new file mode 100755
index 000000000..f27c7810b
--- /dev/null
+++ b/tools/deploy/gen-test-data/files/run/provision-scripts/deploy.sh
@@ -0,0 +1,9 @@
+#!/bin/bash
+
+set -euxo pipefail
+
+dnf -y install  osbuild \
+                osbuild-selinux \
+                osbuild-ostree \
+                osbuild-composer \
+                golang
diff --git a/tools/deploy/gen-test-data/user-data.yml b/tools/deploy/gen-test-data/user-data.yml
new file mode 100644
index 000000000..7d4d68eff
--- /dev/null
+++ b/tools/deploy/gen-test-data/user-data.yml
@@ -0,0 +1,20 @@
+#cloud-config
+yum_repos:
+  # Fetch osbuild packages from a repository served on the host.
+  #
+  # In qemu user networking, 10.0.2.2 always points to the host:
+  # https://wiki.qemu.org/Documentation/Networking#User_Networking_.28SLIRP.29
+  osbuild:
+    name: osbuild
+    baseurl: "http://10.0.2.2:8000"
+    enabled: true
+    gpgcheck: false
+    skip_if_unavailable: true
+user: admin
+password: foobar
+ssh_pwauth: True
+chpasswd:
+  expire: False
+sudo: 'ALL=(ALL) NOPASSWD:ALL'
+runcmd:
+  - /run/provision-scripts/deploy.sh
diff --git a/tools/test-case-generators/distro-arch-imagetype-map.json b/tools/test-case-generators/distro-arch-imagetype-map.json
new file mode 100644
index 000000000..7c79172d0
--- /dev/null
+++ b/tools/test-case-generators/distro-arch-imagetype-map.json
@@ -0,0 +1,69 @@
+{
+    "fedora-32": {
+        "x86_64": [
+            "ami",
+            "fedora-iot-commit",
+            "openstack",
+            "qcow2",
+            "qcow2-customize",
+            "vhd",
+            "vmdk"
+        ],
+        "aarch64": [
+            "ami",
+            "openstack",
+            "qcow2"
+        ]
+    },
+    "fedora-33": {
+        "x86_64": [
+            "ami",
+            "fedora-iot-commit",
+            "openstack",
+            "qcow2",
+            "qcow2-customize",
+            "vhd",
+            "vmdk"
+        ],
+        "aarch64": [
+            "ami"
+        ]
+    },
+    "rhel-8": {
+        "x86_64": [
+            "ami",
+            "openstack",
+            "qcow2",
+            "qcow2-customize",
+            "rhel-edge-commit",
+            "tar",
+            "vhd",
+            "vmdk"
+        ],
+        "aarch64": [
+            "ami",
+            "openstack",
+            "qcow2",
+            "rhel-edge-commit",
+            "tar"
+        ],
+        "ppc64le": [
+            "qcow2",
+            "tar"
+        ],
+        "s390x": [
+            "qcow2",
+            "tar"
+        ]
+    },
+    "rhel-84": {
+        "x86_64": [
+            "ami",
+            "openstack",
+            "qcow2",
+            "qcow2-customize",
+            "vhd",
+            "vmdk"
+        ]
+    }
+}
\ No newline at end of file
diff --git a/tools/test-case-generators/generate-all-test-cases b/tools/test-case-generators/generate-all-test-cases
new file mode 100755
index 000000000..00cbc8401
--- /dev/null
+++ b/tools/test-case-generators/generate-all-test-cases
@@ -0,0 +1,933 @@
+#!/usr/bin/python3
+
+# pylint: disable=line-too-long
+"""
+    generate-all-test-cases
+
+    Script to generate all image test cases based on distro x arch x image-type
+    matrix read from `distro-arch-imagetype-map.json` or passed file. One can
+    filter the matrix just to a subset using `--distro`, `--arch` or
+    `--image-types` arguments.
+
+    The script is intended to be run from the osbuild-composer sources directory
+    root, for which the image test cases should be (re)generated.
+
+    Example (builds rhel-8 qcow2 images on aarch64 s390x ppc64le):
+    tools/test-case-generators/generate-all-test-cases \
+        --output=test/data/manifests \
+        --image-x86_64 ~/Downloads/Images/Fedora-Cloud-Base-33-1.2.x86_64.qcow2 \
+        --image-ppc64le ~/Downloads/Images/Fedora-Cloud-Base-33-1.2.ppc64le.qcow2 \
+        --image-aarch64 ~/Downloads/Images/Fedora-Cloud-Base-33-1.2.aarch64.qcow2 \
+        --image-s390x ~/Downloads/Images/Fedora-Cloud-Base-33-1.2.s390x.qcow2 \
+        --arch aarch64 s390x ppc64le \
+        --distro rhel-8 \
+        --image-types qcow2
+
+    The script spins up an ephemeral QEMU VM (called Runner) per each required
+    architecture. The CWD (sources dir root) is attached to the Runner using
+    virtfs as readonly and later mounted into /mnt/sources on the Runner.
+    The 'output' directory is also attached to the Runner using virtfs as r/w
+    and later mounted into /mnt/output on the Runner. The next execution on
+    Runners is as follows:
+    - Wait for the runner to be configured using cloud-init.
+        - includes installing osbuild, osbuild-composer and golang
+    - Create /mnt/sources and /mnt/output and mount appropriate devices
+    - in /mnt/sources execute tools/test-case-generators/generate-test-cases
+      for each requested distro and image type combination on the particular
+      architecture. Output manifest is written into /mnt/output
+
+    One can use e.q. Fedora cloud qcow2 images:
+    x86_64: https://download.fedoraproject.org/pub/fedora/linux/releases/33/Cloud/x86_64/images/Fedora-Cloud-Base-33-1.2.x86_64.qcow2
+    aarch64: https://download.fedoraproject.org/pub/fedora/linux/releases/33/Cloud/aarch64/images/Fedora-Cloud-Base-33-1.2.aarch64.qcow2
+    ppc64le: https://download.fedoraproject.org/pub/fedora-secondary/releases/33/Cloud/ppc64le/images/Fedora-Cloud-Base-33-1.2.ppc64le.qcow2
+    s390x: https://download.fedoraproject.org/pub/fedora-secondary/releases/33/Cloud/s390x/images/Fedora-Cloud-Base-33-1.2.s390x.qcow2
+
+    aarch64 special note:
+    make sure to have the *edk2-aarch64* package installed, which provides UEFI
+    builds for QEMU and AARCH64 (/usr/share/edk2/aarch64/QEMU_EFI.fd)
+    https://fedoraproject.org/wiki/Architectures/AArch64/Install_with_QEMU
+
+    Images need to have enough disk space to be able to build images using
+    osbuild. You can resize them using 'qemu-img resize <image> 20G' command.
+
+    Known issues:
+    - The tool does not work with RHEL qcow2 images, becuase the "9p" filesystem
+    is not supported on RHEL.
+
+    HW requirements:
+    - The x86_64 VM uses 1 CPU and 1GB of RAM
+    - The aarch64, s390x and ppc64le VMs each uses 2CPU and 2GB of RAM
+    - Unless filtered using `--arch` option, the script starts 4 VMs in parallel
+
+    Tested with:
+    - Fedora 32 (x86_64) and QEMU version 4.2.1
+
+    Not tested:
+    - installation of newer 'osbuild-composer' or 'osbuild' packages from the
+      local 'osbuild' repository, which is configured by cloud-init. Not sure
+      how dnf will behave if there are packages for multiple architectures.
+"""
+
+
+import argparse
+import subprocess
+import json
+import os
+import tempfile
+import shutil
+import time
+import socket
+import contextlib
+import multiprocessing
+import logging
+
+import yaml
+import paramiko
+
+
+# setup logging
+log = logging.getLogger("generate-all-test-cases")
+log.setLevel(logging.INFO)
+formatter = logging.Formatter("%(asctime)s [%(levelname)s] - %(processName)s: %(message)s")
+sh = logging.StreamHandler()
+sh.setFormatter(formatter)
+log.addHandler(sh)
+
+# suppress all errors logged by paramiko
+paramiko.util.log_to_file(os.devnull)
+
+
+class RunnerMountPoint:
+    """
+    Data structure to represent basic data used by Runners to attach host
+    directory as virtfs to the guest and then to mount it.
+    """
+    def __init__(self, src_host, dst_guest, mount_tag, readonly):
+        self.src_host = src_host
+        self.dst_guest = dst_guest
+        self.readonly = readonly
+        self.mount_tag = mount_tag
+
+    @staticmethod
+    def get_default_runner_mount_points(output_dir, sources_dir=None):
+        """
+        Returns a list of default mount points used by Runners when generating
+        image test cases.
+        """
+        sources_dir = os.getcwd() if sources_dir is None else sources_dir
+        mount_points = [
+            RunnerMountPoint(sources_dir, "/mnt/sources", "sources", True),
+            RunnerMountPoint(output_dir, "/mnt/output", "output", False)
+        ]
+        return mount_points
+
+
+class BaseRunner(contextlib.AbstractContextManager):
+    """
+    Base class representing a QEMU VM runner, which is used for generating image
+    test case definitions.
+
+    Each architecture-specific runner should inherit from this class and define
+    QEMU_BIN, QEMU_CMD class variable. These will be used to successfully boot
+    VM for the given architecture.
+    """
+
+    # name of the QEMU binary to use for running the VM
+    QEMU_BIN = None
+    # the actual command to use for running QEMU VM
+    QEMU_CMD = None
+
+    def __init__(self, image, user, passwd, cdrom_iso=None, mount_points=None):
+        self._check_qemu_bin()
+
+        # path to image to run
+        self.image = image
+        # path to cdrom iso to attach (for cloud-init)
+        self.cdrom_iso = cdrom_iso
+        # host directories to share with the VM as virtfs devices
+        self.mount_points = mount_points if mount_points else list()
+        # Popen object of the qemu process
+        self.vm = None
+        self.vm_ready = False
+        # following values are set after the VM is terminated
+        self.vm_return_code = None
+        self.vm_stdout = None
+        self.vm_stderr = None
+        # credentials used to SSH to the VM
+        self.vm_user = user
+        self.vm_pass = passwd
+        # port on host to forward the guest's SSH port (22) to
+        self.host_fwd_ssh_port = None
+
+    def _check_qemu_bin(self):
+        """
+        Checks whether QEMU binary used for the particular runner is present
+        on the system.
+        """
+        try:
+            subprocess.check_call([self.QEMU_BIN, "--version"])
+        except subprocess.CalledProcessError as _:
+            raise RuntimeError("QEMU binary {} not found".format(self.QEMU_BIN))
+
+    def _get_qemu_cdrom_option(self):
+        """
+        Get the appropriate options for attaching CDROM device to the VM, if
+        the path to ISO has been provided.
+
+        This method may be reimplemented by architecture specific runner class
+        if needed. Returns a list of strings to be appended to the QEMU command.
+        """
+        if self.cdrom_iso:
+            return ["-cdrom", self.cdrom_iso]
+
+        return list()
+
+    def _get_qemu_boot_image_option(self):
+        """
+        Get the appropriate options for specifying the image to boot from.
+
+        This method may be reimplemented by architecture specific runner class
+        if needed.
+
+        Returns a list of strings to be appended to the QEMU command.
+        """
+        return [self.image]
+
+    def _get_qemu_ssh_fwd_option(self):
+        """
+        Get the appropriate options for forwarding guest's port 22 to host's
+        random available port.
+        """
+        # get a random free TCP port. This should work in majority of cases
+        with contextlib.closing(socket.socket(socket.AF_INET, socket.SOCK_STREAM)) as sock:
+            sock.bind(('localhost', 0))
+            self.host_fwd_ssh_port = sock.getsockname()[1]
+
+            return ["-net", "user,hostfwd=tcp::{}-:22".format(self.host_fwd_ssh_port)]
+
+    def _run_qemu_cmd(self, qemu_cmd):
+        """
+        Assembles the QEMU command to run and executes using subprocess.
+        """
+        # handle CDROM
+        qemu_cmd.extend(self._get_qemu_cdrom_option())
+
+        # handle mount points
+        for mount_point in self.mount_points:
+            src_host = mount_point.src_host
+            tag = mount_point.mount_tag
+            readonly = ",readonly" if mount_point.readonly else ""
+            qemu_cmd.extend(["-virtfs", f"local,path={src_host},mount_tag={tag},security_model=mapped-xattr{readonly}"])
+
+        # handle boot image
+        qemu_cmd.extend(self._get_qemu_boot_image_option())
+
+        # handle forwarding of guest's SSH port to host
+        qemu_cmd.extend(self._get_qemu_ssh_fwd_option())
+
+        log.debug("Starting VM using command: '%s'", " ".join(qemu_cmd))
+        self.vm = subprocess.Popen(
+            qemu_cmd,
+            stdout=subprocess.PIPE,
+            stderr=subprocess.PIPE
+        )
+
+    def start(self):
+        """
+        Starts the QEMU process running the VM
+        """
+        if not self.QEMU_CMD:
+            raise NotImplementedError("The way to spin up QEMU VM is not implemented")
+
+        # don't start the qemu process if there is already one running
+        if self.vm is None:
+            self._run_qemu_cmd(list(self.QEMU_CMD))
+        log.info(
+            "Runner started. You can SSH to it once it has been configured:" +\
+            "'ssh %s@localhost -p %d' using password: '%s'",
+            self.vm_user,
+            self.host_fwd_ssh_port,
+            self.vm_pass
+        )
+
+    def stop(self):
+        """
+        Stops the QEMU process running the VM
+        """
+        if self.vm:
+            self.vm.terminate()
+            try:
+                # give the process some time to terminate
+                self.vm.wait(timeout=15)
+            except subprocess.TimeoutExpired as _:
+                self.vm.kill()
+                self.vm.wait(timeout=15)
+
+            if self.vm.stdout:
+                self.vm_stdout = self.vm.stdout.read().decode()
+            if self.vm.stderr:
+                self.vm_stderr = self.vm.stderr.read().decode()
+            self.vm_return_code = self.vm.returncode
+
+            if self.vm_return_code == 0:
+                log.debug("%s process ended with return code %d\n\n" + \
+                          "stdout:\n%s\nstderr:\n%s", self.QEMU_BIN,
+                          self.vm_return_code, self.vm_stdout, self.vm_stderr)
+            else:
+                log.error("%s process ended with return code %d\n\n" + \
+                          "stdout:\n%s\nstderr:\n%s", self.QEMU_BIN,
+                          self.vm_return_code, self.vm_stdout, self.vm_stderr)
+
+            self.vm = None
+            self.vm_ready = False
+
+    def run_command(self, command):
+        """
+        Runs a given command on the VM over ssh in a blocking fashion.
+
+        Calling this method before is_ready() returned True has undefined
+        behavior.
+
+        Returns stdin, stdout, stderr from the run command.
+        """
+        ssh = paramiko.SSHClient()
+        # don't ask / fail on unknown remote host fingerprint, just accept any
+        ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+        try:
+            ssh.connect("localhost", self.host_fwd_ssh_port, self.vm_user, self.vm_pass)
+            ssh_tansport = ssh.get_transport()
+            channel = ssh_tansport.open_session()
+            # don't log commands when the vm is not yet ready for use
+            if self.vm_ready:
+                log.debug("Running on VM: '%s'", command)
+            channel.exec_command(command)
+            stdout = ""
+            stderr = ""
+            # wait for the command to finish
+            while True:
+                while channel.recv_ready():
+                    stdout += channel.recv(1024).decode()
+                while channel.recv_stderr_ready():
+                    stderr += channel.recv_stderr(1024).decode()
+                if channel.exit_status_ready():
+                    break
+                time.sleep(0.01)
+            returncode = channel.recv_exit_status()
+        except Exception as e:
+            # don't log errors when vm is not ready yet, because there are many errors
+            if self.vm_ready:
+                log.error("Running command over ssh failed: %s", str(e))
+            raise e
+        finally:
+            # closes the underlying transport
+            ssh.close()
+
+        return stdout, stderr, returncode
+
+    def run_command_check_call(self, command):
+        """
+        Runs a command on the VM over ssh in a similar fashion as subprocess.check_call()
+        """
+        _, _, ret = self.run_command(command)
+        if ret != 0:
+            raise subprocess.CalledProcessError(ret, command)
+
+    def wait_until_ready(self, timeout=None):
+        """
+        Waits for the VM to be ready for use (cloud-init configuration finished).
+
+        If timeout is provided
+        """
+        now = time.time()
+        while not self.is_ready():
+            if timeout is not None and time.time() > (now + timeout):
+                raise subprocess.TimeoutExpired("wait_until_ready()", timeout)
+            time.sleep(15)
+
+    def is_ready(self):
+        """
+        Returns True if the VM is ready to be used.
+        VM is ready after the cloud-init setup is finished.
+        """
+        if self.vm_ready:
+            return True
+
+        # check if the runner didn't terminate unexpectedly before being ready
+        try:
+            if self.vm:
+                self.vm.wait(1)
+        except subprocess.TimeoutExpired as _:
+            # process still running
+            pass
+        else:
+            # process not running, call .stop() to log stdout, stderr and retcode
+            self.stop()
+            qemu_bin = self.QEMU_BIN
+            raise RuntimeError(f"'{qemu_bin}' process ended before being ready to use")
+
+        try:
+            # cloud-init touches /var/lib/cloud/instance/boot-finished after it finishes
+            self.run_command_check_call("ls /var/lib/cloud/instance/boot-finished")
+        except (paramiko.ChannelException,
+                paramiko.ssh_exception.NoValidConnectionsError,
+                paramiko.ssh_exception.SSHException,
+                EOFError,
+                socket.timeout,
+                subprocess.CalledProcessError) as _:
+            # ignore all reasonable paramiko exceptions, this is useful when the VM is still stating up
+            pass
+        else:
+            log.debug("VM is ready for use")
+            self.vm_ready = True
+
+        return self.vm_ready
+
+    def mount_mount_points(self):
+        """
+        This method mounts the needed mount points on the VM.
+
+        It should be called only after is_vm_ready() returned True. Otherwise it will fail.
+        """
+        for mount_point in self.mount_points:
+            dst_guest = mount_point.dst_guest
+            mount_tag = mount_point.mount_tag
+            self.run_command_check_call(f"sudo mkdir {dst_guest}")
+            #! FIXME: "9p" filesystem is not supported on RHEL!
+            out, err, ret = self.run_command(f"sudo mount -t 9p -o trans=virtio {mount_tag} {dst_guest} -oversion=9p2000.L")
+            if ret != 0:
+                log.error("Mounting '%s' to '%s' failed with retcode: %d\nstdout: %s\nstderr: %s", mount_tag, dst_guest,
+                ret, out, err)
+                raise subprocess.CalledProcessError(
+                    ret,
+                    f"sudo mount -t 9p -o trans=virtio {mount_tag} {dst_guest} -oversion=9p2000.L")
+
+    def __enter__(self):
+        self.start()
+        return self
+
+    def __exit__(self, *exc_details):
+        self.stop()
+
+    @staticmethod
+    def prepare_cloud_init_cdrom(userdata, workdir):
+        """
+        Generates a CDROM ISO used as a data source for cloud-init.
+
+        Returns path to the generated CDROM ISO image.
+        """
+        iso_path = os.path.join(workdir, "cloudinit.iso")
+        cidatadir = os.path.join(workdir, "cidata")
+        user_data_path = os.path.join(cidatadir, "user-data")
+        meta_data_path = os.path.join(cidatadir, "meta-data")
+
+        os.mkdir(cidatadir)
+        if os.path.isdir(userdata):
+            with open(user_data_path, "w") as f:
+                script_dir = os.path.dirname(__file__)
+                subprocess.check_call(
+                    [os.path.abspath(f"{script_dir}/../gen-user-data"), userdata], stdout=f)
+        else:
+            shutil.copy(userdata, user_data_path)
+
+        with open(meta_data_path, "w") as f:
+            f.write("instance-id: nocloud\nlocal-hostname: vm\n")
+
+        sysname = os.uname().sysname
+        log.debug("Generating CDROM ISO image for cloud-init user data: %s", iso_path)
+        if sysname == "Linux":
+            subprocess.check_call(
+                [
+                    "genisoimage",
+                    "-input-charset", "utf-8",
+                    "-output", iso_path,
+                    "-volid", "cidata",
+                    "-joliet",
+                    "-rock",
+                    "-quiet",
+                    "-graft-points",
+                    user_data_path,
+                    meta_data_path
+                ],
+                stdout=subprocess.DEVNULL,
+                stderr=subprocess.DEVNULL
+            )
+        elif sysname == "Darwin":
+            subprocess.check_call(
+                [
+                    "hdiutil",
+                    "makehybrid",
+                    "-iso",
+                    "-joliet",
+                    "-o", iso_path,
+                    "{cidatadir}"
+                ],
+                stdout=subprocess.DEVNULL,
+                stderr=subprocess.DEVNULL
+            )
+        else:
+            raise NotImplementedError(f"Unsupported system '{sysname}' for generating cdrom iso")
+
+        return iso_path
+
+
+class X86_64Runner(BaseRunner):
+    """
+    VM Runner for x86_64 architecture
+    """
+
+    QEMU_BIN = "qemu-system-x86_64"
+    QEMU_CMD = [
+        QEMU_BIN,
+        "-M", "accel=kvm:hvf",
+        "-m", "1024",
+        "-object", "rng-random,filename=/dev/urandom,id=rng0",
+        "-device", "virtio-rng-pci,rng=rng0",
+        "-snapshot",
+        "-cpu", "max",
+        "-net", "nic,model=virtio",
+    ]
+
+
+class Ppc64Runner(BaseRunner):
+    """
+    VM Runner for ppc64le architecture
+    """
+
+    QEMU_BIN = "qemu-system-ppc64"
+    QEMU_CMD = [
+        QEMU_BIN,
+        "-m", "2048", # RAM
+        "-smp", "2", # CPUs
+        "-object", "rng-random,filename=/dev/urandom,id=rng0",
+        "-device", "virtio-rng-pci,rng=rng0",
+        "-snapshot",
+        "-net", "nic,model=virtio",
+    ]
+
+
+class Aarch64Runner(BaseRunner):
+    """
+    VM Runner for aarch64 architecture
+    """
+
+    # aarch64 requires UEFI build for QEMU
+    # https://rwmj.wordpress.com/2015/02/27/how-to-boot-a-fedora-21-aarch64-uefi-guest-on-x86_64/
+    # https://fedoraproject.org/wiki/Architectures/AArch64/Install_with_QEMU
+    QEMU_BIN = "qemu-system-aarch64"
+    QEMU_CMD = [
+        QEMU_BIN,
+        "-m", "2048", # RAM
+        "-smp", "2", # CPUs
+        "-object", "rng-random,filename=/dev/urandom,id=rng0",
+        "-device", "virtio-rng-pci,rng=rng0",
+        "-snapshot",
+        "-monitor", "none",
+        "-machine", "virt",
+        "-cpu", "cortex-a57",
+        "-bios", "/usr/share/edk2/aarch64/QEMU_EFI.fd", # provided by 'edk2-aarch64' Fedora package
+        "-net", "nic,model=virtio",
+    ]
+
+
+class S390xRunner(BaseRunner):
+    """
+    VM Runner for s390x architecture
+    """
+
+    QEMU_BIN = "qemu-system-s390x"
+    QEMU_CMD = [
+        QEMU_BIN,
+        "-m", "2048", # RAM
+        "-smp", "2", # CPUs
+        "-machine", "s390-ccw-virtio",
+        # disable msa5-base to suppress errors:
+        # qemu-system-s390x: warning: 'msa5-base' requires 'kimd-sha-512'
+        # qemu-system-s390x: warning: 'msa5-base' requires 'klmd-sha-512'
+        "-cpu", "max,msa5-base=no",
+        "-object", "rng-random,filename=/dev/urandom,id=rng0",
+        "-device", "virtio-rng-ccw,rng=rng0",
+        "-monitor", "none",
+        "-snapshot",
+        "-net", "nic,model=virtio",
+    ]
+
+    def _get_qemu_cdrom_option(self):
+        """
+        Get the appropriate options for attaching CDROM device to the VM, if the path to ISO has been provided.
+
+        s390x tries to boot from the CDROM if attached the way as BaseRunner does it.
+        """
+        if self.cdrom_iso:
+            iso_path = self.cdrom_iso
+            return list(["-drive", f"file={iso_path},media=cdrom"])
+        else:
+            return list()
+
+    def _get_qemu_boot_image_option(self):
+        """
+        Get the appropriate options for specifying the image to boot from.
+
+        s390x needs to have an explicit 'bootindex' specified.
+        https://qemu.readthedocs.io/en/latest/system/s390x/bootdevices.html
+        """
+        image_path = self.image
+        return [
+            "-drive", f"if=none,id=dr1,file={image_path}",
+            "-device", "virtio-blk,drive=dr1,bootindex=1"
+        ]
+
+
+class TestCaseMatrixGenerator(contextlib.AbstractContextManager):
+    """
+    Class representing generation of all test cases based on provided test
+    cases matrix.
+
+    The class should be used as a context manager to ensure that cleanup
+    of all resources is done (mainly VMs and processes running them).
+
+    VM for each architecture is run in a separate process to ensure that
+    generation is done in parallel.
+    """
+
+    ARCH_RUNNER_MAP = {
+        "x86_64": X86_64Runner,
+        "aarch64": Aarch64Runner,
+        "ppc64le": Ppc64Runner,
+        "s390x": S390xRunner
+    }
+
+    def __init__(self, images, ci_userdata, arch_gen_matrix, output, keep_image_info):
+        """
+        'images' is a dict of qcow2 image paths for each supported architecture,
+        that should be used for VMs:
+        {
+            "arch1": "<image path>",
+            "arch2": "<image path>",
+            ...
+        }
+        'ci_userdata' is path to file / directory containing cloud-init user-data used
+        for generating CDROM ISO image, that is attached to each VM as a cloud-init data source.
+        'arch_get_matrix' is a dict of requested distro-image_type matrix per architecture:
+        {
+            "arch1": {
+                "distro1": [
+                    "image-type1",
+                    "image-type2"
+                ],
+                "distro2": [
+                    "image-type2",
+                    "image-type3"
+                ]
+            },
+            "arch2": {
+                "distro2": [
+                    "image-type2"
+                ]
+            },
+            ...
+        }
+        'output' is a directory path, where the generated test case manifests should be stored.
+        'keep_image_info' specifies whether to pass the '--keep-image-info' option to the 'generate-test-cases' script.
+        """
+        self._processes = list()
+        self.images = images
+        self.ci_userdata = ci_userdata
+        self.arch_gen_matrix = arch_gen_matrix
+        self.output = output
+        self.keep_image_info = keep_image_info
+
+        # check that we have image for each needed architecture
+        for arch in self.arch_gen_matrix.keys():
+            if self.images.get(arch, None) is None:
+                raise RuntimeError(f"architecture '{arch}' is in requested test matrix, but no image was provided")
+
+    @staticmethod
+    def runner_function(arch, runner_cls, image, user, passwd, cdrom_iso, generation_matrix, output, keep_image_info):
+        """
+        Generate test cases using VM with appropriate architecture.
+
+        'generation_matrix' is expected to be already architecture-specific
+        dict of 'distro' x 'image-type' matrix.
+
+        {
+            "fedora-32": [
+                "qcow2",
+                "vmdk"
+            ],
+            "rhel-84": [
+                "qcow2",
+                "tar"
+            ],
+            ...
+        }
+        """
+
+        mount_points = RunnerMountPoint.get_default_runner_mount_points(output)
+        go_tls_timeout_retries = 3
+
+        # spin up appropriate VM represented by 'runner'
+        with runner_cls(image, user, passwd, cdrom_iso, mount_points=mount_points) as runner:
+            log.info("Waiting for the '%s' runner to be configured by cloud-init", arch)
+            runner.wait_until_ready()
+            runner.mount_mount_points()
+
+            # don't use /var/tmp for osbuild's store directory to prevent systemd from possibly
+            # removing some of the downloaded RPMs due to "ageing"
+            guest_osbuild_store_dir = "/root/osbuild-store"
+            runner.run_command_check_call(f"sudo mkdir {guest_osbuild_store_dir}")
+
+            # Log installed versions of important RPMs
+            rpm_versions, _, _ = runner.run_command("rpm -q osbuild osbuild-composer")
+            log.info("Installed packages: %s", " ".join(rpm_versions.split("\n")))
+
+            for distro, img_type_list in generation_matrix.items():
+                for image_type in img_type_list:
+                    log.info("Generating test case for '%s' '%s' image on '%s'", distro, image_type, arch)
+
+                    # is the image with customizations?
+                    if image_type.endswith("-customize"):
+                        with_customizations = True
+                        image_type = image_type.rstrip("-customize")
+                    else:
+                        with_customizations = False
+
+                    gen_test_cases_cmd = "cd /mnt/sources; sudo tools/test-case-generators/generate-test-cases" + \
+                        f" --distro {distro} --arch {arch} --image-types {image_type}" + \
+                        f" --store {guest_osbuild_store_dir} --output /mnt/output/"
+                    if with_customizations:
+                        gen_test_cases_cmd += " --with-customizations"
+                    if keep_image_info:
+                        gen_test_cases_cmd += " --keep-image-info"
+
+                    # allow fixed number of retries if the command fails for a specific reason
+                    for i in range(1, go_tls_timeout_retries+1):
+                        if i > 1:
+                            log.info("Retrying image test case generation (%d of %d)", i, go_tls_timeout_retries)
+
+                        stdout, stderr, retcode = runner.run_command(gen_test_cases_cmd)
+                        # clean up the osbuild-store dir
+                        runner.run_command_check_call(f"sudo rm -rf {guest_osbuild_store_dir}/*")
+
+                        if retcode != 0:
+                            log.error("'%s' retcode: %d\nstdout: %s\nstderr: %s", gen_test_cases_cmd, retcode,
+                                      stdout, stderr)
+
+                            # Retry the command, if there was an error due to TLS handshake timeout
+                            # This is happening on all runners using other than host's arch from time to time.
+                            if stderr.find("net/http: TLS handshake timeout") != -1:
+                                continue
+                        else:
+                            log.info("Generating test case for %s-%s-%s - SUCCEEDED", distro, arch, image_type)
+
+                        # don't retry if the process ended successfully or if there was a different error
+                        break
+
+            log.info("'%s' runner finished its work", arch)
+
+            # TODO: Possibly remove after testing / fine tuning the script
+            log.info("Waiting for 1 hour, before terminating the runner (CTRL + c will terminate all VMs)")
+            time.sleep(3600)
+            runner.stop()
+
+    def generate(self):
+        """
+        Generates all test cases based on provided data
+        """
+        # use the same CDROM ISO image for all VMs
+        with tempfile.TemporaryDirectory(prefix="osbuild-composer-test-gen-") as tmpdir:
+            cdrom_iso = BaseRunner.prepare_cloud_init_cdrom(self.ci_userdata, tmpdir)
+
+            # Load user / password from the cloud-init user-data
+            if os.path.isdir(self.ci_userdata):
+                user_data_path = os.path.join(self.ci_userdata, "user-data.yml")
+            else:
+                user_data_path = self.ci_userdata
+            with open(user_data_path, "r") as ud:
+                user_data = yaml.safe_load(ud)
+                vm_user = user_data["user"]
+                vm_pass = user_data["password"]
+
+            # Start a separate runner VM for each required architecture
+            for arch, generation_matrix in self.arch_gen_matrix.items():
+                process = multiprocessing.Process(
+                    target=self.runner_function,
+                    args=(arch, self.ARCH_RUNNER_MAP[arch], self.images[arch], vm_user, vm_pass, cdrom_iso,
+                          generation_matrix, self.output, self.keep_image_info))
+                self._processes.append(process)
+                process.start()
+                log.info("Started '%s' runner - %s", arch, process.name)
+
+            # wait for all processes to finish
+            log.info("Waiting for all runner processes to finish")
+            for process in self._processes:
+                process.join()
+            self._processes.clear()
+
+    def cleanup(self):
+        """
+        Terminates all running processes of VM runners.
+        """
+        # ensure that all processes running VMs are stopped
+        for process in self._processes:
+            process.terminate()
+            process.join(5)
+            # kill the process if it didn't terminate yet
+            if process.exitcode is None:
+                process.kill()
+            process.close()
+        self._processes.clear()
+
+    def __exit__(self, *exc_details):
+        self.cleanup()
+
+
+def get_args():
+    """
+    Returns ArgumentParser instance specific to this script.
+    """
+    parser = argparse.ArgumentParser(description="(re)generate image all test cases")
+    parser.add_argument(
+        "--image-x86_64",
+        help="Path to x86_64 image to use for QEMU VM",
+        required=False
+    )
+    parser.add_argument(
+        "--image-ppc64le",
+        help="Path to ppc64le image to use for QEMU VM",
+        required=False
+    )
+    parser.add_argument(
+        "--image-aarch64",
+        help="Path to aarch64 image to use for QEMU VM",
+        required=False
+    )
+    parser.add_argument(
+        "--image-s390x",
+        help="Path to s390x image to use for QEMU VM",
+        required=False
+    )
+    parser.add_argument(
+        "--distro",
+        help="Filters the matrix for generation only to specified distro",
+        nargs='*',
+        required=False
+    )
+    parser.add_argument(
+        "--arch",
+        help="Filters the matrix for generation only to specified architecture",
+        nargs='*',
+        required=False
+    )
+    parser.add_argument(
+        "--image-types",
+        help="Filters the matrix for generation only to specified image types",
+        nargs='*',
+        required=False
+    )
+    parser.add_argument(
+        "--keep-image-info",
+        action='store_true',
+        help="Skip image info (re)generation, but keep the one found in the existing test case"
+    )
+    parser.add_argument(
+        "--output",
+        metavar="OUTPUT_DIRECTORY",
+        type=os.path.abspath,
+        help="Path to the output directory, where to store resulting manifests for image test cases",
+        required=True
+    )
+    parser.add_argument(
+        "--gen-matrix-file",
+        help="Path to JSON file from which to read the test case generation matrix (distro x arch x image type)." + \
+            " If not provided, '<script_location_dir>/distro-arch-imagetype-map.json' is read.",
+        type=os.path.abspath
+    )
+    parser.add_argument(
+        "--ci-userdata",
+        help="Path to a file/directory with cloud-init user-data, which should be used to configure runner VMs",
+        type=os.path.abspath
+    )
+    parser.add_argument(
+        "-d", "--debug",
+        action='store_true',
+        default=False,
+        help="turn on debug logging"
+    )
+    return parser.parse_args()
+
+# pylint: disable=too-many-arguments,too-many-locals
+def main(vm_images, distros, arches, image_types, ci_userdata, gen_matrix_file, output, keep_image_info):
+    if not os.path.isdir(output):
+        raise RuntimeError(f"output directory {output} does not exist")
+
+    script_dir = os.path.dirname(__file__)
+    gen_matrix_path = gen_matrix_file if gen_matrix_file else f"{script_dir}/distro-arch-imagetype-map.json"
+    log.info("Loading generation matrix from file: '%s'", gen_matrix_path)
+    with open(gen_matrix_path, "r") as gen_matrix_json:
+        gen_matrix_dict = json.load(gen_matrix_json)
+
+    # Filter generation matrix based on passed arguments
+    for distro in list(gen_matrix_dict.keys()):
+        # filter the distros list
+        if distros and distro not in distros:
+            del gen_matrix_dict[distro]
+            continue
+        for arch in list(gen_matrix_dict[distro].keys()):
+            # filter the arches list of a distro
+            if arches and arch not in arches:
+                del gen_matrix_dict[distro][arch]
+                continue
+            # filter the image types of a distro and arch
+            if image_types:
+                gen_matrix_dict[distro][arch] = list(filter(lambda x: x in image_types, gen_matrix_dict[distro][arch]))
+                # delete the whole arch if there is no image type left after filtering
+                if len(gen_matrix_dict[distro][arch]) == 0:
+                    del gen_matrix_dict[distro][arch]
+
+    log.debug("gen_matrix_dict:\n%s", json.dumps(gen_matrix_dict, indent=2, sort_keys=True))
+
+    # Construct per-architecture matrix dictionary of distro x image type
+    arch_gen_matrix_dict = dict()
+    for distro, arches in gen_matrix_dict.items():
+        for arch, image_types in arches.items():
+            try:
+                arch_dict = arch_gen_matrix_dict[arch]
+            except KeyError as _:
+                arch_dict = arch_gen_matrix_dict[arch] = dict()
+            arch_dict[distro] = image_types.copy()
+
+    log.debug("arch_gen_matrix_dict:\n%s", json.dumps(arch_gen_matrix_dict, indent=2, sort_keys=True))
+
+    ci_userdata_path = ci_userdata if ci_userdata else os.path.abspath(f"{script_dir}/../deploy/gen-test-data")
+    log.debug("Using cloud-init user-data from '%s'", ci_userdata_path)
+
+    with TestCaseMatrixGenerator(vm_images, ci_userdata_path, arch_gen_matrix_dict, output, keep_image_info) as generator:
+        generator.generate()
+
+
+if __name__ == '__main__':
+    args = get_args()
+
+    if  args.debug:
+        log.setLevel(logging.DEBUG)
+
+    vm_images = {
+        "x86_64": args.image_x86_64,
+        "aarch64": args.image_aarch64,
+        "ppc64le": args.image_ppc64le,
+        "s390x": args.image_s390x
+    }
+
+    try:
+        main(
+            vm_images,
+            args.distro,
+            args.arch,
+            args.image_types,
+            args.ci_userdata,
+            args.gen_matrix_file,
+            args.output,
+            args.keep_image_info
+        )
+    except KeyboardInterrupt as _:
+        log.info("Interrupted by user")