Repository: osbuild/osbuild-composer
Pull request #2095 (2021-12-03 18:14:29)

composer: Only set queue and artifact dir for fsqueue
 When backed by a DB, composer has no need of a queue or artifact directory.
    
This also addresses "Error moving artifacts for job" logging noise when using a dbqueue.


Gundersanne: > Makes sense, but feels like a layering violation to make it conditional on no DB. For the jobs dir this obviously makes sense if we have the fs backend, but for artefacts, how is the logic done that disables them for the db backend?

Yea you're right, it really is service vs on prem. at least the tests pass so that's ok, i'll figure out how to make this better
Gundersanne: > this will disable artifacts dir by default, that's not backwards compatible, is it?

should default to https://github.com/osbuild/osbuild-composer/pull/2095/files#diff-93349275b33e6f3c25d6ae24f48c50991dd6ca9db01e5c5c7106f76cfe4a1582R96
teg: > > this will disable artifacts dir by default, that's not backwards compatible, is it?
> 
> should default to https://github.com/osbuild/osbuild-composer/pull/2095/files#diff-93349275b33e6f3c25d6ae24f48c50991dd6ca9db01e5c5c7106f76cfe4a1582R96

Ah, I can't read. Indeed, that's ok.

> But not sure if I like specifying a relative directory as a config option :|

Make it a boolean? But use it to enable/disable artifacts handling as a whole? Now it is conditional on the weldr api.

ondrejbudai: Needs a rebase

Patch:
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 37f9c65e1..871c6067a 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -17,6 +17,7 @@ stages:
     paths:
       - journal-log.gpg
       - ci-artifacts
+      - iostats.json
     when: always
 
 .terraform:
@@ -44,7 +45,9 @@ RPM:
   rules:
     - if: '$CI_PIPELINE_SOURCE != "schedule"'
   script:
+    - schutzbot/start_iostats.sh
     - sh "schutzbot/mockbuild.sh"
+    - schutzbot/stop_iostats.sh
   interruptible: true
   after_script:
     - schutzbot/update_github_status.sh update
@@ -76,7 +79,9 @@ Container:
   rules:
     - if: '$CI_PIPELINE_SOURCE != "schedule"'
   script:
+    - schutzbot/start_iostats.sh
     - sh "schutzbot/containerbuild.sh"
+    - schutzbot/stop_iostats.sh
   interruptible: true
   parallel:
     matrix:
@@ -100,7 +105,9 @@ Prepare-rhel-internal:
     - if: '$CI_PIPELINE_SOURCE == "schedule" && $RUNNER =~ /[\S]+rhel-9.0-[^ga][\S]+/ && $NIGHTLY == "true" && $RHEL_MAJOR == "9"'
     - if: '$CI_PIPELINE_SOURCE == "schedule" && $RUNNER =~ /[\S]+rhel-8.6-[^ga][\S]+/ && $NIGHTLY == "true" && $RHEL_MAJOR == "8"'
   script:
+    - schutzbot/start_iostats.sh
     - schutzbot/prepare-rhel-internal.sh
+    - schutzbot/stop_iostats.sh
   interruptible: true
   artifacts:
     paths:
@@ -125,8 +132,10 @@ Base:
     - if: '$CI_PIPELINE_SOURCE == "schedule" && $RUNNER =~ /[\S]+rhel-9.0-[^ga][\S]+/ && $NIGHTLY == "true" && $RHEL_MAJOR == "9"'
     - if: '$CI_PIPELINE_SOURCE == "schedule" && $RUNNER =~ /[\S]+rhel-8.6-[^ga][\S]+/ && $NIGHTLY == "true" && $RHEL_MAJOR == "8"'
   script:
+    - schutzbot/start_iostats.sh
     - schutzbot/deploy.sh
     - /usr/libexec/tests/osbuild-composer/base_tests.sh
+    - schutzbot/stop_iostats.sh
   interruptible: true
   parallel:
     matrix:
@@ -181,8 +190,10 @@ Regression:
     - if: '$CI_PIPELINE_SOURCE == "schedule" && $RUNNER =~ /[\S]+rhel-9.0-[^ga][\S]+/ && $NIGHTLY == "true" && $RHEL_MAJOR == "9"'
     - if: '$CI_PIPELINE_SOURCE == "schedule" && $RUNNER =~ /[\S]+rhel-8.6-[^ga][\S]+/ && $NIGHTLY == "true" && $RHEL_MAJOR == "8"'
   script:
+    - schutzbot/start_iostats.sh
     - schutzbot/deploy.sh
     - /usr/libexec/tests/osbuild-composer/regression.sh
+    - schutzbot/stop_iostats.sh
   interruptible: true
   parallel:
     matrix:
@@ -206,6 +217,26 @@ Regression:
       - "*.repo"
     when: always
 
+Test Case Generation:
+  stage: test
+  extends: .terraform
+  rules:
+    - if: '$CI_PIPELINE_SOURCE != "schedule"'
+  script:
+    - schutzbot/deploy.sh
+    - /usr/libexec/tests/osbuild-composer/generation.sh
+  interruptible: true
+  parallel:
+    matrix:
+      - RUNNER:
+          - aws/fedora-34-x86_64
+        INTERNAL_NETWORK: ["true"]
+  artifacts:
+    paths:
+      - journal-log.gpg
+      - "*.repo"
+    when: always
+
 OSTree:
   stage: test
   extends: .terraform/openstack
@@ -214,8 +245,10 @@ OSTree:
     - if: '$CI_PIPELINE_SOURCE == "schedule" && $RUNNER =~ /[\S]+rhel-9.0-[^ga][\S]+/ && $NIGHTLY == "true" && $RHEL_MAJOR == "9"'
     - if: '$CI_PIPELINE_SOURCE == "schedule" && $RUNNER =~ /[\S]+rhel-8.6-[^ga][\S]+/ && $NIGHTLY == "true" && $RHEL_MAJOR == "8"'
   script:
+    - schutzbot/start_iostats.sh
     - schutzbot/deploy.sh
     - /usr/libexec/tests/osbuild-composer/ostree.sh
+    - schutzbot/stop_iostats.sh
   interruptible: true
   parallel:
     matrix:
@@ -232,8 +265,10 @@ New OSTree:
   stage: test
   extends: OSTree
   script:
+    - schutzbot/start_iostats.sh
     - schutzbot/deploy.sh
     - /usr/libexec/tests/osbuild-composer/ostree-ng.sh
+    - schutzbot/stop_iostats.sh
   parallel:
     matrix:
       - RUNNER:
@@ -245,8 +280,10 @@ OSTree simplified installer:
   stage: test
   extends: OSTree
   script:
+    - schutzbot/start_iostats.sh
     - schutzbot/deploy.sh
     - /usr/libexec/tests/osbuild-composer/ostree-simplified-installer.sh
+    - schutzbot/stop_iostats.sh
   parallel:
     matrix:
       - RUNNER:
@@ -258,8 +295,10 @@ OSTree raw image:
   stage: test
   extends: OSTree
   script:
+    - schutzbot/start_iostats.sh
     - schutzbot/deploy.sh
     - /usr/libexec/tests/osbuild-composer/ostree-raw-image.sh
+    - schutzbot/stop_iostats.sh
   parallel:
     matrix:
       - RUNNER:
@@ -284,8 +323,10 @@ Integration:
     - if: '$CI_PIPELINE_SOURCE == "schedule" && $RUNNER =~ /[\S]+rhel-9.0-[^ga][\S]+/ && $NIGHTLY == "true" && $RHEL_MAJOR == "9"'
     - if: '$CI_PIPELINE_SOURCE == "schedule" && $RUNNER =~ /[\S]+rhel-8.6-[^ga][\S]+/ && $NIGHTLY == "true" && $RHEL_MAJOR == "8"'
   script:
+    - schutzbot/start_iostats.sh
     - schutzbot/deploy.sh
     - /usr/libexec/tests/osbuild-composer/${SCRIPT}
+    - schutzbot/stop_iostats.sh
   parallel:
     matrix:
       - <<: *INTEGRATION_TESTS
@@ -327,8 +368,10 @@ API:
     - if: '$CI_PIPELINE_SOURCE == "schedule" && $RUNNER =~ /[\S]+rhel-9.0-[^ga][\S]+/ && $NIGHTLY == "true" && $RHEL_MAJOR == "9"'
     - if: '$CI_PIPELINE_SOURCE == "schedule" && $RUNNER =~ /[\S]+rhel-8.6-[^ga][\S]+/ && $NIGHTLY == "true" && $RHEL_MAJOR == "8"'
   script:
+    - schutzbot/start_iostats.sh
     - schutzbot/deploy.sh
     - /usr/libexec/tests/osbuild-composer/api.sh ${IMAGE_TYPE}
+    - schutzbot/stop_iostats.sh
   parallel:
     matrix:
       - <<: *API_TESTS
@@ -346,8 +389,10 @@ libvirt:
     - if: '$CI_PIPELINE_SOURCE == "schedule" && $RUNNER =~ /[\S]+rhel-9.0-[^ga][\S]+/ && $NIGHTLY == "true" && $RHEL_MAJOR == "9"'
     - if: '$CI_PIPELINE_SOURCE == "schedule" && $RUNNER =~ /[\S]+rhel-8.6-[^ga][\S]+/ && $NIGHTLY == "true" && $RHEL_MAJOR == "8"'
   script:
+    - schutzbot/start_iostats.sh
     - schutzbot/deploy.sh
     - /usr/libexec/tests/osbuild-composer/libvirt.sh
+    - schutzbot/stop_iostats.sh
   interruptible: true
   parallel:
     matrix:
@@ -366,8 +411,10 @@ RHEL 9 on 8:
   rules:
     - if: '$CI_PIPELINE_SOURCE != "schedule"'
   script:
+    - schutzbot/start_iostats.sh
     - schutzbot/deploy.sh
     - /usr/libexec/tests/osbuild-composer/koji.sh
+    - schutzbot/stop_iostats.sh
   interruptible: true
   variables:
     RUNNER: aws/rhel-8.5-ga-x86_64
@@ -401,8 +448,10 @@ Installer:
     - if: '$CI_PIPELINE_SOURCE == "schedule" && $RUNNER =~ /[\S]+rhel-9\.[0-9]-[^ga][\S]+/ && $NIGHTLY == "true" && $RHEL_MAJOR == "9"'
     - if: '$CI_PIPELINE_SOURCE == "schedule" && $RUNNER =~ /[\S]+rhel-8\.[0-9]-[^ga][\S]+/ && $NIGHTLY == "true" && $RHEL_MAJOR == "8"'
   script:
+    - schutzbot/start_iostats.sh
     - schutzbot/deploy.sh
     - /usr/libexec/tests/osbuild-composer/installers.sh
+    - schutzbot/stop_iostats.sh
   interruptible: true
   parallel:
     matrix:
diff --git a/cmd/osbuild-composer/composer.go b/cmd/osbuild-composer/composer.go
index 051c51383..29e200be3 100644
--- a/cmd/osbuild-composer/composer.go
+++ b/cmd/osbuild-composer/composer.go
@@ -53,20 +53,19 @@ func NewComposer(config *ComposerConfigFile, stateDir, cacheDir string) (*Compos
 		cacheDir: cacheDir,
 	}
 
-	queueDir, err := c.ensureStateDirectory("jobs", 0700)
-	if err != nil {
-		return nil, err
-	}
-
-	artifactsDir, err := c.ensureStateDirectory("artifacts", 0755)
-	if err != nil {
-		return nil, err
+	var err error
+	artifactsDir := ""
+	if config.Worker.EnableArtifacts {
+		artifactsDir, err = c.ensureStateDirectory("artifacts", 0755)
+		if err != nil {
+			return nil, err
+		}
 	}
 
 	c.distros = distroregistry.NewDefault()
 	logrus.Infof("Loaded %d distros", len(c.distros.List()))
 
-	c.rpm = rpmmd.NewRPMMD(path.Join(c.cacheDir, "rpmmd"), "/usr/libexec/osbuild-composer/dnf-json")
+	c.rpm = rpmmd.NewRPMMD(path.Join(c.cacheDir, "rpmmd"))
 
 	var jobs jobqueue.JobQueue
 	if config.Worker.PGDatabase != "" {
@@ -88,6 +87,10 @@ func NewComposer(config *ComposerConfigFile, stateDir, cacheDir string) (*Compos
 			return nil, fmt.Errorf("cannot create jobqueue: %v", err)
 		}
 	} else {
+		queueDir, err := c.ensureStateDirectory("jobs", 0700)
+		if err != nil {
+			return nil, err
+		}
 		jobs, err = fsjobqueue.New(queueDir)
 		if err != nil {
 			return nil, fmt.Errorf("cannot create jobqueue: %v", err)
@@ -192,11 +195,11 @@ func (c *Composer) InitRemoteWorkers(cert, key string, enableTLS bool, enableMTL
 func (c *Composer) Start() error {
 	// sanity checks
 	if c.localWorkerListener == nil && c.workerListener == nil {
-		log.Fatal("neither the local worker socket nor the remote worker socket is enabled, osbuild-composer is useless without workers")
+		logrus.Fatal("neither the local worker socket nor the remote worker socket is enabled, osbuild-composer is useless without workers")
 	}
 
 	if c.apiListener == nil && c.weldrListener == nil {
-		log.Fatal("neither the weldr API socket nor the composer API socket is enabled, osbuild-composer is useless without one of these APIs enabled")
+		logrus.Fatal("neither the weldr API socket nor the composer API socket is enabled, osbuild-composer is useless without one of these APIs enabled")
 	}
 
 	if c.localWorkerListener != nil {
diff --git a/cmd/osbuild-composer/config.go b/cmd/osbuild-composer/config.go
index e37a6de07..0881d8d58 100644
--- a/cmd/osbuild-composer/config.go
+++ b/cmd/osbuild-composer/config.go
@@ -39,6 +39,7 @@ type WorkerAPIConfig struct {
 	CA                string   `toml:"ca"`
 	RequestJobTimeout string   `toml:"request_job_timeout"`
 	BasePath          string   `toml:"base_path"`
+	EnableArtifacts   bool     `toml:"enable_artifacts"`
 	PGHost            string   `toml:"pg_host" env:"PGHOST"`
 	PGPort            string   `toml:"pg_port" env:"PGPORT"`
 	PGDatabase        string   `toml:"pg_database" env:"PGDATABASE"`
@@ -92,6 +93,7 @@ func GetDefaultConfig() *ComposerConfigFile {
 		Worker: WorkerAPIConfig{
 			RequestJobTimeout: "0",
 			BasePath:          "/api/worker/v1",
+			EnableArtifacts:   true,
 			EnableTLS:         true,
 			EnableMTLS:        true,
 			EnableJWT:         false,
diff --git a/cmd/osbuild-koji/main.go b/cmd/osbuild-koji/main.go
index a51346e58..d6299ac92 100644
--- a/cmd/osbuild-koji/main.go
+++ b/cmd/osbuild-koji/main.go
@@ -3,7 +3,6 @@ package main
 import (
 	"flag"
 	"fmt"
-	"log"
 	"net/http"
 	"os"
 	"path"
@@ -12,6 +11,7 @@ import (
 	"github.com/google/uuid"
 	"github.com/osbuild/osbuild-composer/internal/rpmmd"
 	"github.com/osbuild/osbuild-composer/internal/upload/koji"
+	"github.com/sirupsen/logrus"
 )
 
 func main() {
@@ -50,7 +50,7 @@ func main() {
 	defer func() {
 		err := k.Logout()
 		if err != nil {
-			log.Print("logging out of koji failed ", err)
+			logrus.Warn("logging out of koji failed ", err)
 		}
 	}()
 
diff --git a/cmd/osbuild-pipeline/main.go b/cmd/osbuild-pipeline/main.go
index 7496028f8..d80493de2 100644
--- a/cmd/osbuild-pipeline/main.go
+++ b/cmd/osbuild-pipeline/main.go
@@ -118,7 +118,7 @@ func main() {
 		panic("os.UserHomeDir(): " + err.Error())
 	}
 
-	rpm_md := rpmmd.NewRPMMD(path.Join(home, ".cache/osbuild-composer/rpmmd"), "/usr/libexec/osbuild-composer/dnf-json")
+	rpm_md := rpmmd.NewRPMMD(path.Join(home, ".cache/osbuild-composer/rpmmd"))
 
 	packageSpecSets := make(map[string][]rpmmd.PackageSpec)
 	for name, packages := range packageSets {
diff --git a/cmd/osbuild-store-dump/main.go b/cmd/osbuild-store-dump/main.go
index 4f2599268..bf23daece 100644
--- a/cmd/osbuild-store-dump/main.go
+++ b/cmd/osbuild-store-dump/main.go
@@ -138,7 +138,7 @@ func main() {
 	if err != nil {
 		panic("os.UserHomeDir(): " + err.Error())
 	}
-	rpmmd := rpmmd.NewRPMMD(path.Join(homeDir, ".cache/osbuild-composer/rpmmd"), "/usr/libexec/osbuild-composer/dnf-json")
+	rpmmd := rpmmd.NewRPMMD(path.Join(homeDir, ".cache/osbuild-composer/rpmmd"))
 
 	s := store.New(&cwd, a, nil)
 	if s == nil {
diff --git a/cmd/osbuild-upload-gcp/main.go b/cmd/osbuild-upload-gcp/main.go
index d276f496c..abaae9644 100644
--- a/cmd/osbuild-upload-gcp/main.go
+++ b/cmd/osbuild-upload-gcp/main.go
@@ -5,9 +5,9 @@ import (
 	"flag"
 	"fmt"
 	"io/ioutil"
-	"log"
 
 	"github.com/osbuild/osbuild-composer/internal/cloud/gcp"
+	"github.com/sirupsen/logrus"
 )
 
 type strArrayFlag []string
@@ -52,64 +52,64 @@ func main() {
 		var err error
 		credentials, err = ioutil.ReadFile(credentialsPath)
 		if err != nil {
-			log.Fatalf("[GCP] Error while reading credentials: %v", err)
+			logrus.Fatalf("[GCP] Error while reading credentials: %v", err)
 		}
 	}
 
 	g, err := gcp.New(credentials)
 	if err != nil {
-		log.Fatalf("[GCP] Failed to create new GCP object: %v", err)
+		logrus.Fatalf("[GCP] Failed to create new GCP object: %v", err)
 	}
 
 	ctx := context.Background()
 
 	// Upload image to the Storage
 	if !skipUpload {
-		log.Printf("[GCP] 🚀 Uploading image to: %s/%s", bucketName, objectName)
+		logrus.Infof("[GCP] 🚀 Uploading image to: %s/%s", bucketName, objectName)
 		_, err := g.StorageObjectUpload(ctx, imageFile, bucketName, objectName,
 			map[string]string{gcp.MetadataKeyImageName: imageName})
 		if err != nil {
-			log.Fatalf("[GCP] Uploading image failed: %v", err)
+			logrus.Fatalf("[GCP] Uploading image failed: %v", err)
 		}
 	}
 
 	// Import Image to Compute Engine
 	if !skipImport {
-		log.Printf("[GCP] 📥 Importing image into Compute Engine as '%s'", imageName)
+		logrus.Infof("[GCP] 📥 Importing image into Compute Engine as '%s'", imageName)
 		imageBuild, importErr := g.ComputeImageImport(ctx, bucketName, objectName, imageName, osFamily, region)
 		if imageBuild != nil {
-			log.Printf("[GCP] 📜 Image import log URL: %s", imageBuild.LogUrl)
-			log.Printf("[GCP] 🎉 Image import finished with status: %s", imageBuild.Status)
+			logrus.Infof("[GCP] 📜 Image import log URL: %s", imageBuild.LogUrl)
+			logrus.Infof("[GCP] 🎉 Image import finished with status: %s", imageBuild.Status)
 
 			// Cleanup all resources potentially left after the image import job
 			deleted, err := g.CloudbuildBuildCleanup(ctx, imageBuild.Id)
 			for _, d := range deleted {
-				log.Printf("[GCP] 🧹 Deleted resource after image import job: %s", d)
+				logrus.Infof("[GCP] 🧹 Deleted resource after image import job: %s", d)
 			}
 			if err != nil {
-				log.Printf("[GCP] Encountered error during image import cleanup: %v", err)
+				logrus.Warnf("[GCP] Encountered error during image import cleanup: %v", err)
 			}
 		}
 
 		// Cleanup storage before checking for errors
-		log.Printf("[GCP] 🧹 Deleting uploaded image file: %s/%s", bucketName, objectName)
+		logrus.Infof("[GCP] 🧹 Deleting uploaded image file: %s/%s", bucketName, objectName)
 		if err = g.StorageObjectDelete(ctx, bucketName, objectName); err != nil {
-			log.Printf("[GCP] Encountered error while deleting object: %v", err)
+			logrus.Warnf("[GCP] Encountered error while deleting object: %v", err)
 		}
 
 		// check error from ComputeImageImport()
 		if importErr != nil {
-			log.Fatalf("[GCP] Importing image failed: %v", importErr)
+			logrus.Fatalf("[GCP] Importing image failed: %v", importErr)
 		}
-		log.Printf("[GCP] 💿 Image URL: %s", g.ComputeImageURL(imageName))
+		logrus.Infof("[GCP] 💿 Image URL: %s", g.ComputeImageURL(imageName))
 	}
 
 	// Share the imported Image with specified accounts using IAM policy
 	if len(shareWith) > 0 {
-		log.Printf("[GCP] 🔗 Sharing the image with: %+v", shareWith)
+		logrus.Infof("[GCP] 🔗 Sharing the image with: %+v", shareWith)
 		err = g.ComputeImageShare(ctx, imageName, []string(shareWith))
 		if err != nil {
-			log.Fatalf("[GCP] Sharing image failed: %s", err)
+			logrus.Fatalf("[GCP] Sharing image failed: %s", err)
 		}
 	}
 }
diff --git a/cmd/osbuild-worker/jobimpl-depsolve.go b/cmd/osbuild-worker/jobimpl-depsolve.go
index 091d7d4f2..85018974b 100644
--- a/cmd/osbuild-worker/jobimpl-depsolve.go
+++ b/cmd/osbuild-worker/jobimpl-depsolve.go
@@ -11,12 +11,17 @@ type DepsolveJobImpl struct {
 	RPMMDCache string
 }
 
-func (impl *DepsolveJobImpl) depsolve(packageSets map[string]rpmmd.PackageSet, repos []rpmmd.RepoConfig, modulePlatformID, arch, releasever string) (map[string][]rpmmd.PackageSpec, error) {
-	rpmMD := rpmmd.NewRPMMD(impl.RPMMDCache, "/usr/libexec/osbuild-composer/dnf-json")
+func (impl *DepsolveJobImpl) depsolve(packageSets map[string]rpmmd.PackageSet, repos []rpmmd.RepoConfig, modulePlatformID, arch, releasever string, packageSetsRepositories map[string][]rpmmd.RepoConfig) (map[string][]rpmmd.PackageSpec, error) {
+	rpmMD := rpmmd.NewRPMMD(impl.RPMMDCache)
 
 	packageSpecs := make(map[string][]rpmmd.PackageSpec)
 	for name, packageSet := range packageSets {
-		packageSpec, _, err := rpmMD.Depsolve(packageSet, repos, modulePlatformID, arch, releasever)
+		repositories := make([]rpmmd.RepoConfig, len(repos))
+		copy(repositories, repos)
+		if packageSetRepositories, ok := packageSetsRepositories[name]; ok {
+			repositories = append(repositories, packageSetRepositories...)
+		}
+		packageSpec, _, err := rpmMD.Depsolve(packageSet, repositories, modulePlatformID, arch, releasever)
 		if err != nil {
 			return nil, err
 		}
@@ -33,7 +38,7 @@ func (impl *DepsolveJobImpl) Run(job worker.Job) error {
 	}
 
 	var result worker.DepsolveJobResult
-	result.PackageSpecs, err = impl.depsolve(args.PackageSets, args.Repos, args.ModulePlatformID, args.Arch, args.Releasever)
+	result.PackageSpecs, err = impl.depsolve(args.PackageSets, args.Repos, args.ModulePlatformID, args.Arch, args.Releasever, args.PackageSetsRepositories)
 	if err != nil {
 		switch err.(type) {
 		case *rpmmd.DNFError:
diff --git a/cmd/osbuild-worker/jobimpl-koji-finalize.go b/cmd/osbuild-worker/jobimpl-koji-finalize.go
index 41a5808f7..baf39bafb 100644
--- a/cmd/osbuild-worker/jobimpl-koji-finalize.go
+++ b/cmd/osbuild-worker/jobimpl-koji-finalize.go
@@ -3,11 +3,12 @@ package main
 import (
 	"crypto/tls"
 	"fmt"
-	"log"
 	"net/http"
 	"net/url"
 	"time"
 
+	"github.com/sirupsen/logrus"
+
 	"github.com/osbuild/osbuild-composer/internal/rpmmd"
 	"github.com/osbuild/osbuild-composer/internal/upload/koji"
 	"github.com/osbuild/osbuild-composer/internal/worker"
@@ -48,7 +49,7 @@ func (impl *KojiFinalizeJobImpl) kojiImport(
 	defer func() {
 		err := k.Logout()
 		if err != nil {
-			log.Printf("koji logout failed: %v", err)
+			logrus.Warnf("koji logout failed: %v", err)
 		}
 	}()
 
@@ -86,7 +87,7 @@ func (impl *KojiFinalizeJobImpl) kojiFail(server string, buildID int, token stri
 	defer func() {
 		err := k.Logout()
 		if err != nil {
-			log.Printf("koji logout failed: %v", err)
+			logrus.Warnf("koji logout failed: %v", err)
 		}
 	}()
 
diff --git a/cmd/osbuild-worker/jobimpl-koji-init.go b/cmd/osbuild-worker/jobimpl-koji-init.go
index f85696e44..4df408587 100644
--- a/cmd/osbuild-worker/jobimpl-koji-init.go
+++ b/cmd/osbuild-worker/jobimpl-koji-init.go
@@ -3,10 +3,11 @@ package main
 import (
 	"crypto/tls"
 	"fmt"
-	"log"
 	"net/http"
 	"net/url"
 
+	"github.com/sirupsen/logrus"
+
 	"github.com/osbuild/osbuild-composer/internal/upload/koji"
 	"github.com/osbuild/osbuild-composer/internal/worker"
 )
@@ -41,7 +42,7 @@ func (impl *KojiInitJobImpl) kojiInit(server, name, version, release string) (st
 	defer func() {
 		err := k.Logout()
 		if err != nil {
-			log.Printf("koji logout failed: %v", err)
+			logrus.Warnf("koji logout failed: %v", err)
 		}
 	}()
 
diff --git a/cmd/osbuild-worker/jobimpl-osbuild-koji.go b/cmd/osbuild-worker/jobimpl-osbuild-koji.go
index 7f60650ce..e24c7554b 100644
--- a/cmd/osbuild-worker/jobimpl-osbuild-koji.go
+++ b/cmd/osbuild-worker/jobimpl-osbuild-koji.go
@@ -4,7 +4,6 @@ import (
 	"crypto/tls"
 	"fmt"
 	"io/ioutil"
-	"log"
 	"net/http"
 	"net/url"
 	"os"
@@ -14,6 +13,7 @@ import (
 	"github.com/osbuild/osbuild-composer/internal/distro"
 	"github.com/osbuild/osbuild-composer/internal/upload/koji"
 	"github.com/osbuild/osbuild-composer/internal/worker"
+	"github.com/sirupsen/logrus"
 )
 
 type OSBuildKojiJobImpl struct {
@@ -48,7 +48,7 @@ func (impl *OSBuildKojiJobImpl) kojiUpload(file *os.File, server, directory, fil
 	defer func() {
 		err := k.Logout()
 		if err != nil {
-			log.Printf("koji logout failed: %v", err)
+			logrus.Warnf("koji logout failed: %v", err)
 		}
 	}()
 
@@ -63,7 +63,7 @@ func (impl *OSBuildKojiJobImpl) Run(job worker.Job) error {
 	defer func() {
 		err := os.RemoveAll(outputDirectory)
 		if err != nil {
-			log.Printf("Error removing temporary output directory (%s): %v", outputDirectory, err)
+			logrus.Warnf("Error removing temporary output directory (%s): %v", outputDirectory, err)
 		}
 	}()
 
diff --git a/cmd/osbuild-worker/jobimpl-osbuild.go b/cmd/osbuild-worker/jobimpl-osbuild.go
index 1a89e0c2b..255558e66 100644
--- a/cmd/osbuild-worker/jobimpl-osbuild.go
+++ b/cmd/osbuild-worker/jobimpl-osbuild.go
@@ -4,12 +4,12 @@ import (
 	"context"
 	"fmt"
 	"io/ioutil"
-	"log"
 	"os"
 	"path"
 	"strings"
 
 	"github.com/google/uuid"
+	"github.com/sirupsen/logrus"
 
 	"github.com/osbuild/osbuild-composer/internal/cloud/awscloud"
 	"github.com/osbuild/osbuild-composer/internal/cloud/gcp"
@@ -33,7 +33,7 @@ type OSBuildJobImpl struct {
 
 func appendTargetError(res *worker.OSBuildJobResult, err error) {
 	errStr := err.Error()
-	log.Printf("target failed: %s", errStr)
+	logrus.Errorf("target failed: %s", errStr)
 	res.TargetErrors = append(res.TargetErrors, errStr)
 }
 
@@ -49,6 +49,7 @@ func (impl *OSBuildJobImpl) getAWS(region string, accessId string, secret string
 }
 
 func (impl *OSBuildJobImpl) Run(job worker.Job) error {
+	logWithId := logrus.WithField("jobId", job.Id().String())
 	// Initialize variable needed for reporting back to osbuild-composer.
 	var osbuildJobResult *worker.OSBuildJobResult = &worker.OSBuildJobResult{
 		Success: false,
@@ -64,12 +65,12 @@ func (impl *OSBuildJobImpl) Run(job worker.Job) error {
 	defer func() {
 		err := job.Update(osbuildJobResult)
 		if err != nil {
-			log.Printf("Error reporting job result: %v", err)
+			logWithId.Errorf("Error reporting job result: %v", err)
 		}
 
 		err = os.RemoveAll(outputDirectory)
 		if err != nil {
-			log.Printf("Error removing temporary output directory (%s): %v", outputDirectory, err)
+			logWithId.Errorf("Error removing temporary output directory (%s): %v", outputDirectory, err)
 		}
 	}()
 
@@ -105,7 +106,7 @@ func (impl *OSBuildJobImpl) Run(job worker.Job) error {
 	// The specification allows multiple upload targets because it is an array, but we don't support it.
 	// Return an error to osbuild-composer.
 	if len(args.Targets) > 1 {
-		log.Printf("The job specification contains more than one upload target. This is not supported any more. " +
+		logrus.Warnf("The job specification contains more than one upload target. This is not supported any more. " +
 			"This might indicate a deployment of incompatible osbuild-worker and osbuild-composer versions.")
 		return nil
 	}
@@ -135,15 +136,15 @@ func (impl *OSBuildJobImpl) Run(job worker.Job) error {
 			// no pipeline output
 			continue
 		}
-		log.Printf("%s pipeline results:\n", pipelineName)
+		logWithId.Infof("%s pipeline results:\n", pipelineName)
 		for _, stageResult := range pipelineLog {
 			if stageResult.Success {
-				log.Printf("  %s success", stageResult.Type)
+				logWithId.Infof("  %s success", stageResult.Type)
 			} else {
-				log.Printf("  %s failure:", stageResult.Type)
+				logWithId.Infof("  %s failure:", stageResult.Type)
 				stageOutput := strings.Split(stageResult.Output, "\n")
 				for _, line := range stageOutput {
-					log.Printf("    %s", line)
+					logWithId.Infof("    %s", line)
 				}
 			}
 		}
@@ -206,7 +207,7 @@ func (impl *OSBuildJobImpl) Run(job worker.Job) error {
 			defer func() {
 				err := os.RemoveAll(tempDirectory)
 				if err != nil {
-					log.Printf("Error removing temporary directory for vmware symlink(%s): %v", tempDirectory, err)
+					logWithId.Errorf("Error removing temporary directory for vmware symlink(%s): %v", tempDirectory, err)
 				}
 			}()
 
@@ -327,7 +328,7 @@ func (impl *OSBuildJobImpl) Run(job worker.Job) error {
 				return nil
 			}
 
-			log.Printf("[GCP] 🚀 Uploading image to: %s/%s", options.Bucket, options.Object)
+			logWithId.Infof("[GCP] 🚀 Uploading image to: %s/%s", options.Bucket, options.Object)
 			_, err = g.StorageObjectUpload(ctx, path.Join(outputDirectory, exportPath, options.Filename),
 				options.Bucket, options.Object, map[string]string{gcp.MetadataKeyImageName: args.Targets[0].ImageName})
 			if err != nil {
@@ -335,26 +336,26 @@ func (impl *OSBuildJobImpl) Run(job worker.Job) error {
 				return nil
 			}
 
-			log.Printf("[GCP] 📥 Importing image into Compute Engine as '%s'", args.Targets[0].ImageName)
+			logWithId.Infof("[GCP] 📥 Importing image into Compute Engine as '%s'", args.Targets[0].ImageName)
 			imageBuild, importErr := g.ComputeImageImport(ctx, options.Bucket, options.Object, args.Targets[0].ImageName, options.Os, options.Region)
 			if imageBuild != nil {
-				log.Printf("[GCP] 📜 Image import log URL: %s", imageBuild.LogUrl)
-				log.Printf("[GCP] 🎉 Image import finished with status: %s", imageBuild.Status)
+				logWithId.Infof("[GCP] 📜 Image import log URL: %s", imageBuild.LogUrl)
+				logWithId.Infof("[GCP] 🎉 Image import finished with status: %s", imageBuild.Status)
 
 				// Cleanup all resources potentially left after the image import job
 				deleted, err := g.CloudbuildBuildCleanup(ctx, imageBuild.Id)
 				for _, d := range deleted {
-					log.Printf("[GCP] 🧹 Deleted resource after image import job: %s", d)
+					logWithId.Infof("[GCP] 🧹 Deleted resource after image import job: %s", d)
 				}
 				if err != nil {
-					log.Printf("[GCP] Encountered error during image import cleanup: %v", err)
+					logWithId.Errorf("[GCP] Encountered error during image import cleanup: %v", err)
 				}
 			}
 
 			// Cleanup storage before checking for errors
-			log.Printf("[GCP] 🧹 Deleting uploaded image file: %s/%s", options.Bucket, options.Object)
+			logWithId.Infof("[GCP] 🧹 Deleting uploaded image file: %s/%s", options.Bucket, options.Object)
 			if err = g.StorageObjectDelete(ctx, options.Bucket, options.Object); err != nil {
-				log.Printf("[GCP] Encountered error while deleting object: %v", err)
+				logWithId.Errorf("[GCP] Encountered error while deleting object: %v", err)
 			}
 
 			// check error from ComputeImageImport()
@@ -362,10 +363,10 @@ func (impl *OSBuildJobImpl) Run(job worker.Job) error {
 				appendTargetError(osbuildJobResult, importErr)
 				return nil
 			}
-			log.Printf("[GCP] 💿 Image URL: %s", g.ComputeImageURL(args.Targets[0].ImageName))
+			logWithId.Infof("[GCP] 💿 Image URL: %s", g.ComputeImageURL(args.Targets[0].ImageName))
 
 			if len(options.ShareWithAccounts) > 0 {
-				log.Printf("[GCP] 🔗 Sharing the image with: %+v", options.ShareWithAccounts)
+				logWithId.Infof("[GCP] 🔗 Sharing the image with: %+v", options.ShareWithAccounts)
 				err = g.ComputeImageShare(ctx, args.Targets[0].ImageName, options.ShareWithAccounts)
 				if err != nil {
 					appendTargetError(osbuildJobResult, err)
@@ -393,7 +394,7 @@ func (impl *OSBuildJobImpl) Run(job worker.Job) error {
 				appendTargetError(osbuildJobResult, err)
 				return nil
 			}
-			log.Print("[Azure] 🔑 Logged in Azure")
+			logWithId.Info("[Azure] 🔑 Logged in Azure")
 
 			storageAccountTag := azure.Tag{
 				Name:  "imageBuilderStorageAccount",
@@ -412,7 +413,7 @@ func (impl *OSBuildJobImpl) Run(job worker.Job) error {
 			}
 
 			if storageAccount == "" {
-				log.Print("[Azure] 📦 Creating a new storage account")
+				logWithId.Info("[Azure] 📦 Creating a new storage account")
 				const storageAccountPrefix = "ib"
 				storageAccount = azure.RandomStorageAccountName(storageAccountPrefix)
 
@@ -430,7 +431,7 @@ func (impl *OSBuildJobImpl) Run(job worker.Job) error {
 				}
 			}
 
-			log.Print("[Azure] 🔑📦 Retrieving a storage account key")
+			logWithId.Info("[Azure] 🔑📦 Retrieving a storage account key")
 			storageAccessKey, err := c.GetStorageAccountKey(
 				ctx,
 				options.SubscriptionID,
@@ -450,7 +451,7 @@ func (impl *OSBuildJobImpl) Run(job worker.Job) error {
 
 			storageContainer := "imagebuilder"
 
-			log.Print("[Azure] 📦 Ensuring that we have a storage container")
+			logWithId.Info("[Azure] 📦 Ensuring that we have a storage container")
 			err = azureStorageClient.CreateStorageContainerIfNotExist(ctx, storageAccount, storageContainer)
 			if err != nil {
 				appendTargetError(osbuildJobResult, fmt.Errorf("cannot create a storage container: %v", err))
@@ -462,7 +463,7 @@ func (impl *OSBuildJobImpl) Run(job worker.Job) error {
 				blobName += ".vhd"
 			}
 
-			log.Print("[Azure] ⬆ Uploading the image")
+			logWithId.Info("[Azure] ⬆ Uploading the image")
 			err = azureStorageClient.UploadPageBlob(
 				azure.BlobMetadata{
 					StorageAccount: storageAccount,
@@ -477,7 +478,7 @@ func (impl *OSBuildJobImpl) Run(job worker.Job) error {
 				return nil
 			}
 
-			log.Print("[Azure] 📝 Registering the image")
+			logWithId.Info("[Azure] 📝 Registering the image")
 			err = c.RegisterImage(
 				ctx,
 				options.SubscriptionID,
@@ -493,7 +494,7 @@ func (impl *OSBuildJobImpl) Run(job worker.Job) error {
 				return nil
 			}
 
-			log.Print("[Azure] 🎉 Image uploaded and registered!")
+			logWithId.Info("[Azure] 🎉 Image uploaded and registered!")
 
 			osbuildJobResult.TargetResults = append(osbuildJobResult.TargetResults, target.NewAzureImageTargetResult(&target.AzureImageTargetResultOptions{
 				ImageName: args.Targets[0].ImageName,
diff --git a/containers/osbuild-composer/entrypoint.py b/containers/osbuild-composer/entrypoint.py
index 89090428d..685378e90 100644
--- a/containers/osbuild-composer/entrypoint.py
+++ b/containers/osbuild-composer/entrypoint.py
@@ -95,12 +95,27 @@ class Cli(contextlib.AbstractContextManager):
             help="Disable the weldr-API",
         )
 
+        # --[no-]dnf-json
+        self._parser.add_argument(
+            "--dnf-json",
+            action="store_true",
+            dest="dnf_json",
+            help="Enable  dnf-json",
+        )
+        self._parser.add_argument(
+            "--no-dnf-json",
+            action="store_false",
+            dest="dnf_json",
+            help="Disable dnf-json",
+        )
+
         self._parser.set_defaults(
             builtin_worker=False,
             composer_api=False,
             local_worker_api=False,
             remote_worker_api=False,
             weldr_api=False,
+            dnf_json=False,
         )
 
         return self._parser.parse_args(self._argv[1:])
@@ -238,11 +253,29 @@ class Cli(contextlib.AbstractContextManager):
             preexec_fn=preexec_setenv,
         )
 
+    def _spawn_dnf_json(self):
+        cmd = [
+            "/usr/libexec/osbuild-composer/dnf-json",
+        ]
+
+        dnfenv = os.environ.copy()
+        dnfenv["LISTEN_FDS"] = "0"
+        dnfenv["LISTEN_FDNAMES"] = ""
+
+        return subprocess.Popen(
+            cmd,
+            cwd="/usr/libexec/osbuild-composer",
+            stdin=subprocess.DEVNULL,
+            stderr=subprocess.STDOUT,
+            env=dnfenv,
+        )
+
     def run(self):
         """Program Runtime"""
 
         proc_composer = None
         proc_worker = None
+        proc_dnf_json = None
         res = 0
         sockets = self._prepare_sockets()
 
@@ -250,24 +283,35 @@ class Cli(contextlib.AbstractContextManager):
             if self.args.builtin_worker:
                 proc_worker = self._spawn_worker()
 
+            if self.args.dnf_json:
+                proc_dnf_json = self._spawn_dnf_json()
+
             proc_composer = self._spawn_composer(sockets)
 
             res = proc_composer.wait()
             if proc_worker:
                 proc_worker.terminate()
                 proc_worker.wait()
+            if proc_dnf_json:
+                proc_dnf_json.terminate()
+                proc_dnf_json.wait()
 
             return res
         except KeyboardInterrupt:
             if proc_worker:
                 proc_worker.terminate()
                 proc_worker.wait()
+            if proc_dnf_json:
+                proc_dnf_json.terminate()
+                proc_dnf_json.wait()
             if proc_composer:
                 proc_composer.terminate()
                 res = proc_composer.wait()
         except:
             if proc_worker:
                 proc_worker.kill()
+            if proc_dnf_json:
+                proc_dnf_json.kill()
             if proc_composer:
                 proc_composer.kill()
             raise
diff --git a/distribution/osbuild-composer.service b/distribution/osbuild-composer.service
index a277c8604..9d88b0d81 100644
--- a/distribution/osbuild-composer.service
+++ b/distribution/osbuild-composer.service
@@ -1,5 +1,6 @@
 [Unit]
 Description=OSBuild Composer
+Requires=osbuild-dnf-json.socket
 After=network.target
 
 # Weldr API needs a local worker by default.
diff --git a/distribution/osbuild-dnf-json.service b/distribution/osbuild-dnf-json.service
new file mode 100644
index 000000000..1f012fb46
--- /dev/null
+++ b/distribution/osbuild-dnf-json.service
@@ -0,0 +1,13 @@
+[Unit]
+Description=OSbuild dnf-json service
+Requires=osbuild-dnf-json.socket
+
+[Service]
+Type=simple
+PrivateTmp=true
+ExecStart=/usr/libexec/osbuild-composer/dnf-json
+Restart=always
+RestartSec=1s
+CacheDirectory=osbuild-dnf-json
+# systemd >= 240 sets this, but osbuild-worker runs on earlier versions
+Environment="CACHE_DIRECTORY=/var/cache/osbuild-dnf-json"
diff --git a/distribution/osbuild-dnf-json.socket b/distribution/osbuild-dnf-json.socket
new file mode 100644
index 000000000..231f75e1a
--- /dev/null
+++ b/distribution/osbuild-dnf-json.socket
@@ -0,0 +1,9 @@
+[Unit]
+Description=OSbuild dnf-json socket
+
+[Socket]
+Service=osbuild-dnf-json.service
+ListenStream=/run/osbuild-dnf-json/api.sock
+
+[Install]
+WantedBy=sockets.target
diff --git a/distribution/osbuild-remote-worker@.service b/distribution/osbuild-remote-worker@.service
index ab19e95c3..965c5b7ed 100644
--- a/distribution/osbuild-remote-worker@.service
+++ b/distribution/osbuild-remote-worker@.service
@@ -1,5 +1,6 @@
 [Unit]
 Description=OSBuild Composer Remote Worker (%i)
+Requires=osbuild-dnf-json.socket
 After=network.target
 
 [Service]
diff --git a/distribution/osbuild-worker@.service b/distribution/osbuild-worker@.service
index 4423085d8..335122304 100644
--- a/distribution/osbuild-worker@.service
+++ b/distribution/osbuild-worker@.service
@@ -1,6 +1,6 @@
 [Unit]
 Description=OSBuild Composer Worker (%i)
-Requires=osbuild-local-worker.socket
+Requires=osbuild-local-worker.socket osbuild-dnf-json.socket
 After=network.target osbuild-local-worker.socket
 
 [Service]
diff --git a/dnf-json b/dnf-json
index cc7cfac3b..9f16dd1f8 100755
--- a/dnf-json
+++ b/dnf-json
@@ -1,155 +1,138 @@
 #!/usr/bin/python3
 
 import datetime
-import dnf
 import hashlib
-import hawkey
 import json
-import sys
 import tempfile
+import os
+import socket
+import socketserver
+import logging
+import sys
+from http.server import BaseHTTPRequestHandler
+import pathlib
 
-DNF_ERROR_EXIT_CODE = 10
-
-
-def timestamp_to_rfc3339(timestamp):
-    d = datetime.datetime.utcfromtimestamp(package.buildtime)
-    return d.strftime('%Y-%m-%dT%H:%M:%SZ')
-
-
-def dnfrepo(desc, parent_conf=None):
-    """Makes a dnf.repo.Repo out of a JSON repository description"""
-
-    repo = dnf.repo.Repo(desc["id"], parent_conf)
-
-    if "baseurl" in desc:
-        repo.baseurl = desc["baseurl"]
-    elif "metalink" in desc:
-        repo.metalink = desc["metalink"]
-    elif "mirrorlist" in desc:
-        repo.mirrorlist = desc["mirrorlist"]
-    else:
-        assert False
-
-    if desc.get("ignoressl", False):
-        repo.sslverify = False
-    if "sslcacert" in desc:
-        repo.sslcacert = desc["sslcacert"]
-    if "sslclientkey" in desc:
-        repo.sslclientkey = desc["sslclientkey"]
-    if "sslclientcert" in desc:
-        repo.sslclientcert = desc["sslclientcert"]
-
-    # In dnf, the default metadata expiration time is 48 hours. However,
-    # some repositories never expire the metadata, and others expire it much
-    # sooner than that. We therefore allow this to be configured. If nothing
-    # is provided we error on the side of checking if we should invalidate
-    # the cache. If cache invalidation is not necessary, the overhead of
-    # checking is in the hundreds of milliseconds. In order to avoid this
-    # overhead accumulating for API calls that consist of several dnf calls,
-    # we set the expiration to a short time period, rather than 0.
-    repo.metadata_expire = desc.get("metadata_expire", "20s")
-
-    return repo
-
-
-def create_base(repos, module_platform_id, persistdir, cachedir, arch):
-    base = dnf.Base()
-
-    # Enable fastestmirror to ensure we choose the fastest mirrors for
-    # downloading metadata (when depsolving) and downloading packages.
-    base.conf.fastestmirror = True
-
-    # We use the same cachedir for multiple architectures. Unfortunately,
-    # this is something that doesn't work well in certain situations
-    # with zchunk:
-    # Imagine that we already have cache for arch1. Then, we use dnf-json
-    # to depsolve for arch2. If ZChunk is enabled and available (that's
-    # the case for Fedora), dnf will try to download only differences
-    # between arch1 and arch2 metadata. But, as these are completely
-    # different, dnf must basically redownload everything.
-    # For downloding deltas, zchunk uses HTTP range requests. Unfortunately,
-    # if the mirror doesn't support multi range requests, then zchunk will
-    # download one small segment per a request. Because we need to update
-    # the whole metadata (10s of MB), this can be extremely slow in some cases.
-    # I think that we can come up with a better fix but let's just disable
-    # zchunk for now. As we are already downloading a lot of data when
-    # building images, I don't care if we download even more.
-    base.conf.zchunk = False
-
-    # Try another mirror if it takes longer than 5 seconds to connect.
-    base.conf.timeout = 5
-
-    # Set the rest of the dnf configuration.
-    base.conf.module_platform_id = module_platform_id
-    base.conf.config_file_path = "/dev/null"
-    base.conf.persistdir = persistdir
-    base.conf.cachedir = cachedir
-    base.conf.substitutions['arch'] = arch
-    base.conf.substitutions['basearch'] = dnf.rpm.basearch(arch)
-
-    for repo in repos:
-        base.repos.add(dnfrepo(repo, base.conf))
-
-    base.fill_sack(load_system_repo=False)
-    return base
-
-
-def exit_with_dnf_error(kind: str, reason: str):
-    json.dump({"kind": kind, "reason": reason}, sys.stdout)
-    sys.exit(DNF_ERROR_EXIT_CODE)
-
-
-def repo_checksums(base):
-    checksums = {}
-    for repo in base.repos.iter_enabled():
-        # Uses the same algorithm as libdnf to find cache dir:
-        #   https://github.com/rpm-software-management/libdnf/blob/master/libdnf/repo/Repo.cpp#L1288
-        if repo.metalink:
-            url = repo.metalink
-        elif repo.mirrorlist:
-            url = repo.mirrorlist
-        elif repo.baseurl:
-            url = repo.baseurl[0]
-        else:
-            assert False
-
-        digest = hashlib.sha256(url.encode()).hexdigest()[:16]
-
-        repomd_file = f"{repo.id}-{digest}/repodata/repomd.xml"
-        with open(f"{base.conf.cachedir}/{repomd_file}", "rb") as f:
-            repomd = f.read()
-
-        checksums[repo.id] = "sha256:" + hashlib.sha256(repomd).hexdigest()
-
-    return checksums
-
-
-call = json.load(sys.stdin)
-command = call["command"]
-arguments = call["arguments"]
-repos = arguments.get("repos", {})
-arch = arguments["arch"]
-cachedir = arguments["cachedir"]
-module_platform_id = arguments["module_platform_id"]
+import dnf
+import hawkey
 
-with tempfile.TemporaryDirectory() as persistdir:
-    try:
-        base = create_base(
-            repos,
-            module_platform_id,
-            persistdir,
-            cachedir,
-            arch
-        )
-    except dnf.exceptions.Error as e:
-        exit_with_dnf_error(
-            type(e).__name__,
-            f"Error occurred when setting up repo: {e}"
+formatter = logging.Formatter(
+        fmt="%(asctime)s %(name)s.%(levelname)s: %(message)s",
+        datefmt="%Y.%m.%d %H:%M:%S"
         )
+handler = logging.StreamHandler(stream=sys.stdout)
+handler.setFormatter(formatter)
+
+log = logging.getLogger('dnf-json')
+log.addHandler(handler)
+log.setLevel(logging.INFO)
+
+class Solver():
+
+    def __init__(self, repos, module_platform_id, persistdir, cachedir, arch):
+        self.base = dnf.Base()
+
+        # Enable fastestmirror to ensure we choose the fastest mirrors for
+        # downloading metadata (when depsolving) and downloading packages.
+        self.base.conf.fastestmirror = True
+
+        # We use the same cachedir for multiple architectures. Unfortunately,
+        # this is something that doesn't work well in certain situations
+        # with zchunk:
+        # Imagine that we already have cache for arch1. Then, we use dnf-json
+        # to depsolve for arch2. If ZChunk is enabled and available (that's
+        # the case for Fedora), dnf will try to download only differences
+        # between arch1 and arch2 metadata. But, as these are completely
+        # different, dnf must basically redownload everything.
+        # For downloding deltas, zchunk uses HTTP range requests. Unfortunately,
+        # if the mirror doesn't support multi range requests, then zchunk will
+        # download one small segment per a request. Because we need to update
+        # the whole metadata (10s of MB), this can be extremely slow in some cases.
+        # I think that we can come up with a better fix but let's just disable
+        # zchunk for now. As we are already downloading a lot of data when
+        # building images, I don't care if we download even more.
+        self.base.conf.zchunk = False
+
+        # Try another mirror if it takes longer than 5 seconds to connect.
+        self.base.conf.timeout = 5
+
+        # Set the rest of the dnf configuration.
+        self.base.conf.module_platform_id = module_platform_id
+        self.base.conf.config_file_path = "/dev/null"
+        self.base.conf.persistdir = persistdir
+        self.base.conf.cachedir = cachedir
+        self.base.conf.substitutions['arch'] = arch
+        self.base.conf.substitutions['basearch'] = dnf.rpm.basearch(arch)
+
+        for repo in repos:
+            self.base.repos.add(self._dnfrepo(repo, self.base.conf))
+        self.base.fill_sack(load_system_repo=False)
+
+    def _dnfrepo(self, desc, parent_conf=None):
+        """Makes a dnf.repo.Repo out of a JSON repository description"""
+
+        repo = dnf.repo.Repo(desc["id"], parent_conf)
+
+        if "baseurl" in desc:
+            repo.baseurl = desc["baseurl"]
+        elif "metalink" in desc:
+            repo.metalink = desc["metalink"]
+        elif "mirrorlist" in desc:
+            repo.mirrorlist = desc["mirrorlist"]
+        else:
+            assert False
 
-    if command == "dump":
+        if desc.get("ignoressl", False):
+            repo.sslverify = False
+        if "sslcacert" in desc:
+            repo.sslcacert = desc["sslcacert"]
+        if "sslclientkey" in desc:
+            repo.sslclientkey = desc["sslclientkey"]
+        if "sslclientcert" in desc:
+            repo.sslclientcert = desc["sslclientcert"]
+
+        # In dnf, the default metadata expiration time is 48 hours. However,
+        # some repositories never expire the metadata, and others expire it much
+        # sooner than that. We therefore allow this to be configured. If nothing
+        # is provided we error on the side of checking if we should invalidate
+        # the cache. If cache invalidation is not necessary, the overhead of
+        # checking is in the hundreds of milliseconds. In order to avoid this
+        # overhead accumulating for API calls that consist of several dnf calls,
+        # we set the expiration to a short time period, rather than 0.
+        repo.metadata_expire = desc.get("metadata_expire", "20s")
+
+        return repo
+
+    def _repo_checksums(self):
+        checksums = {}
+        for repo in self.base.repos.iter_enabled():
+            # Uses the same algorithm as libdnf to find cache dir:
+            #   https://github.com/rpm-software-management/libdnf/blob/master/libdnf/repo/Repo.cpp#L1288
+            if repo.metalink:
+                url = repo.metalink
+            elif repo.mirrorlist:
+                url = repo.mirrorlist
+            elif repo.baseurl:
+                url = repo.baseurl[0]
+            else:
+                assert False
+
+            digest = hashlib.sha256(url.encode()).hexdigest()[:16]
+
+            repomd_file = f"{repo.id}-{digest}/repodata/repomd.xml"
+            with open(f"{self.base.conf.cachedir}/{repomd_file}", "rb") as f:
+                repomd = f.read()
+
+            checksums[repo.id] = "sha256:" + hashlib.sha256(repomd).hexdigest()
+
+        return checksums
+
+    def _timestamp_to_rfc3339(self, timestamp):
+        d = datetime.datetime.utcfromtimestamp(timestamp)
+        return d.strftime('%Y-%m-%dT%H:%M:%SZ')
+
+    def dump(self):
         packages = []
-        for package in base.sack.query().available():
+        for package in self.base.sack.query().available():
             packages.append({
                 "name": package.name,
                 "summary": package.summary,
@@ -159,41 +142,19 @@ with tempfile.TemporaryDirectory() as persistdir:
                 "version": package.version,
                 "release": package.release,
                 "arch": package.arch,
-                "buildtime": timestamp_to_rfc3339(package.buildtime),
+                "buildtime": self._timestamp_to_rfc3339(package.buildtime),
                 "license": package.license
             })
-        json.dump({
-            "checksums": repo_checksums(base),
+        return {
+            "checksums": self._repo_checksums(),
             "packages": packages
-        }, sys.stdout)
-
-    elif command == "depsolve":
-        errors = []
-
-        try:
-            base.install_specs(
-                arguments["package-specs"],
-                exclude=arguments.get("exclude-specs", [])
-            )
-        except dnf.exceptions.MarkingErrors as e:
-            exit_with_dnf_error(
-                "MarkingErrors",
-                f"Error occurred when marking packages for installation: {e}"
-            )
-
-        try:
-            base.resolve()
-        except dnf.exceptions.DepsolveError as e:
-            exit_with_dnf_error(
-                "DepsolveError",
-                (
-                    "There was a problem depsolving "
-                    f"{arguments['package-specs']}: {e}"
-                )
-            )
+        }
 
+    def depsolve(self, package_spec, exclude_spec):
+        self.base.install_specs(package_spec, exclude_spec)
+        self.base.resolve()
         dependencies = []
-        for tsi in base.transaction:
+        for tsi in self.base.transaction:
             # Avoid using the install_set() helper, as it does not guarantee
             # a stable order
             if tsi.action not in dnf.transaction.FORWARD_ACTIONS:
@@ -214,7 +175,101 @@ with tempfile.TemporaryDirectory() as persistdir:
                     f"{package.chksum[1].hex()}"
                 )
             })
-        json.dump({
-            "checksums": repo_checksums(base),
+        return {
+            "checksums": self._repo_checksums(),
             "dependencies": dependencies
-        }, sys.stdout)
+        }
+
+class DnfJsonRequestHandler(BaseHTTPRequestHandler):
+
+    def _send(self):
+        self.client_address=('',)
+
+    def response_with_dnf_error(self, kind: str, reason: str):
+        self._send()
+        self.send_response(500)
+        self.send_header("Content-Type", "application/json")
+        self.end_headers()
+        self.wfile.write(json.dumps({"kind": kind, "reason":
+            reason}).encode("utf-8"))
+
+    def response_success(self, json_object):
+        self._send()
+        self.send_response(200)
+        self.send_header("Content-Type", "application/json")
+        self.end_headers()
+        self.wfile.write(json.dumps(json_object).encode("utf-8"))
+
+    def do_POST(self):
+        content_len = int(self.headers.get('Content-Length'))
+        data = self.rfile.read(content_len)
+        call = json.loads(data.decode("utf-8"))
+        command = call["command"]
+        arguments = call["arguments"]
+        repos = arguments.get("repos", {})
+        arch = arguments["arch"]
+        cachedir = arguments["cachedir"]
+        module_platform_id = arguments["module_platform_id"]
+
+        with tempfile.TemporaryDirectory() as persistdir:
+            try:
+                solver = Solver(
+                    repos,
+                    module_platform_id,
+                    persistdir,
+                    cachedir,
+                    arch
+                )
+                if command == "dump":
+                    self.response_success(solver.dump())
+                    log.info("dump success")
+                elif command == "depsolve":
+                    self.response_success(
+                            solver.depsolve(
+                                arguments["package-specs"],
+                                arguments.get("exclude-specs", [])
+                                )
+                            )
+                    log.info("depsolve success")
+
+            except dnf.exceptions.MarkingErrors as e:
+                log.info("error install_specs")
+                self.response_with_dnf_error(
+                    "MarkingErrors",
+                    f"Error occurred when marking packages for installation: {e}"
+                )
+            except dnf.exceptions.DepsolveError as e:
+                log.info("error depsolve")
+                self.response_with_dnf_error(
+                    "DepsolveError",
+                    (
+                        "There was a problem depsolving "
+                        f"{arguments['package-specs']}: {e}"
+                    )
+                )
+            except dnf.exceptions.Error as e:
+                self.response_with_dnf_error(
+                    type(e).__name__,
+                    f"Error occurred when setting up repo: {e}")
+
+log.info("Starting the dnf-json server")
+
+LISTEN_FDS = int(os.environ.get("LISTEN_FDS", 0))
+SOCK_PATH = "/run/osbuild-dnf-json/"
+SOCK_NAME = "api.sock"
+
+class SystemDActivationSocketServer(socketserver.ForkingMixIn, socketserver.UnixStreamServer):
+    def server_bind(self):
+        log.debug("service bind")
+        if LISTEN_FDS == 0:
+            log.debug("create new socket")
+            socketserver.UnixStreamServer.server_bind(self)
+        else:
+            log.debug("rebind socket")
+            log.debug("address_family: %d ", self.address_family)
+            log.debug("socket_type: %d ", self.socket_type)
+            self.socket = socket.fromfd(3, self.address_family, self.socket_type)
+
+pathlib.Path(SOCK_PATH).mkdir(parents=True, exist_ok=True)
+server = SystemDActivationSocketServer(f"{SOCK_PATH}{SOCK_NAME}", DnfJsonRequestHandler)
+server.serve_forever()
diff --git a/internal/cloud/awscloud/awscloud.go b/internal/cloud/awscloud/awscloud.go
index 48748bf95..2a294e197 100644
--- a/internal/cloud/awscloud/awscloud.go
+++ b/internal/cloud/awscloud/awscloud.go
@@ -2,7 +2,6 @@ package awscloud
 
 import (
 	"fmt"
-	"log"
 	"os"
 	"time"
 
@@ -13,6 +12,7 @@ import (
 	"github.com/aws/aws-sdk-go/service/ec2"
 	"github.com/aws/aws-sdk-go/service/s3"
 	"github.com/aws/aws-sdk-go/service/s3/s3manager"
+	"github.com/sirupsen/logrus"
 )
 
 type AWS struct {
@@ -63,7 +63,7 @@ func (a *AWS) Upload(filename, bucket, key string) (*s3manager.UploadOutput, err
 		return nil, err
 	}
 
-	log.Printf("[AWS] 🚀 Uploading image to S3: %s/%s", bucket, key)
+	logrus.Infof("[AWS] 🚀 Uploading image to S3: %s/%s", bucket, key)
 	return a.uploader.Upload(
 		&s3manager.UploadInput{
 			Bucket: aws.String(bucket),
@@ -144,7 +144,7 @@ func (a *AWS) Register(name, bucket, key string, shareWith []string, rpmArch str
 		return nil, fmt.Errorf("ec2 doesn't support the following arch: %s", rpmArch)
 	}
 
-	log.Printf("[AWS] 📥 Importing snapshot from image: %s/%s", bucket, key)
+	logrus.Infof("[AWS] 📥 Importing snapshot from image: %s/%s", bucket, key)
 	snapshotDescription := fmt.Sprintf("Image Builder AWS Import of %s", name)
 	importTaskOutput, err := a.ec2.ImportSnapshot(
 		&ec2.ImportSnapshotInput{
@@ -158,11 +158,11 @@ func (a *AWS) Register(name, bucket, key string, shareWith []string, rpmArch str
 		},
 	)
 	if err != nil {
-		log.Printf("[AWS] error importing snapshot: %s", err)
+		logrus.Warnf("[AWS] error importing snapshot: %s", err)
 		return nil, err
 	}
 
-	log.Printf("[AWS] 🚚 Waiting for snapshot to finish importing: %s", *importTaskOutput.ImportTaskId)
+	logrus.Infof("[AWS] 🚚 Waiting for snapshot to finish importing: %s", *importTaskOutput.ImportTaskId)
 	err = WaitUntilImportSnapshotTaskCompleted(
 		a.ec2,
 		&ec2.DescribeImportSnapshotTasksInput{
@@ -176,7 +176,7 @@ func (a *AWS) Register(name, bucket, key string, shareWith []string, rpmArch str
 	}
 
 	// we no longer need the object in s3, let's just delete it
-	log.Printf("[AWS] 🧹 Deleting image from S3: %s/%s", bucket, key)
+	logrus.Infof("[AWS] 🧹 Deleting image from S3: %s/%s", bucket, key)
 	_, err = a.s3.DeleteObject(&s3.DeleteObjectInput{
 		Bucket: aws.String(bucket),
 		Key:    aws.String(key),
@@ -199,7 +199,7 @@ func (a *AWS) Register(name, bucket, key string, shareWith []string, rpmArch str
 	snapshotID := importOutput.ImportSnapshotTasks[0].SnapshotTaskDetail.SnapshotId
 
 	if len(shareWith) > 0 {
-		log.Printf("[AWS] 🎥 Sharing ec2 snapshot")
+		logrus.Info("[AWS] 🎥 Sharing ec2 snapshot")
 		var userIds []*string
 		for _, v := range shareWith {
 			// Implicit memory alasing doesn't couse any bug in this case
@@ -215,9 +215,10 @@ func (a *AWS) Register(name, bucket, key string, shareWith []string, rpmArch str
 			},
 		)
 		if err != nil {
+			logrus.Warnf("[AWS] 📨 Error sharing ec2 snapshot: %v", err)
 			return nil, err
 		}
-		log.Println("[AWS] 📨 Shared ec2 snapshot")
+		logrus.Info("[AWS] 📨 Shared ec2 snapshot")
 	}
 
 	// Tag the snapshot with the image name.
@@ -237,7 +238,7 @@ func (a *AWS) Register(name, bucket, key string, shareWith []string, rpmArch str
 		return nil, err
 	}
 
-	log.Printf("[AWS] 📋 Registering AMI from imported snapshot: %s", *snapshotID)
+	logrus.Infof("[AWS] 📋 Registering AMI from imported snapshot: %s", *snapshotID)
 	registerOutput, err := a.ec2.RegisterImage(
 		&ec2.RegisterImageInput{
 			Architecture:       aws.String(ec2Arch),
@@ -259,7 +260,7 @@ func (a *AWS) Register(name, bucket, key string, shareWith []string, rpmArch str
 		return nil, err
 	}
 
-	log.Printf("[AWS] 🎉 AMI registered: %s", *registerOutput.ImageId)
+	logrus.Infof("[AWS] 🎉 AMI registered: %s", *registerOutput.ImageId)
 
 	// Tag the image with the image name.
 	req, _ = a.ec2.CreateTagsRequest(
@@ -279,7 +280,7 @@ func (a *AWS) Register(name, bucket, key string, shareWith []string, rpmArch str
 	}
 
 	if len(shareWith) > 0 {
-		log.Println("[AWS] 💿 Sharing ec2 AMI")
+		logrus.Info("[AWS] 💿 Sharing ec2 AMI")
 		var launchPerms []*ec2.LaunchPermission
 		for _, id := range shareWith {
 			launchPerms = append(launchPerms, &ec2.LaunchPermission{
@@ -297,9 +298,10 @@ func (a *AWS) Register(name, bucket, key string, shareWith []string, rpmArch str
 			},
 		)
 		if err != nil {
+			logrus.Warnf("[AWS] 📨 Error sharing AMI: %v", err)
 			return nil, err
 		}
-		log.Println("[AWS] 💿 Shared AMI")
+		logrus.Info("[AWS] 💿 Shared AMI")
 	}
 
 	return registerOutput.ImageId, nil
@@ -332,7 +334,7 @@ func (a *AWS) RemoveSnapshotAndDeregisterImage(image *ec2.Image) error {
 		)
 		if err != nil {
 			// TODO return err?
-			log.Println("Unable to remove snapshot", s)
+			logrus.Warn("Unable to remove snapshot", s)
 		}
 	}
 	return err
@@ -356,7 +358,7 @@ func (a *AWS) DescribeImagesByTag(tagKey, tagValue string) ([]*ec2.Image, error)
 }
 
 func (a *AWS) S3ObjectPresignedURL(bucket, objectKey string) (string, error) {
-	log.Printf("[AWS] 📋 Generating Presigned URL for S3 object %s/%s", bucket, objectKey)
+	logrus.Infof("[AWS] 📋 Generating Presigned URL for S3 object %s/%s", bucket, objectKey)
 	req, _ := a.s3.GetObjectRequest(&s3.GetObjectInput{
 		Bucket: aws.String(bucket),
 		Key:    aws.String(objectKey),
@@ -365,6 +367,6 @@ func (a *AWS) S3ObjectPresignedURL(bucket, objectKey string) (string, error) {
 	if err != nil {
 		return "", err
 	}
-	log.Print("[AWS] 🎉 S3 Presigned URL ready")
+	logrus.Info("[AWS] 🎉 S3 Presigned URL ready")
 	return url, nil
 }
diff --git a/internal/cloudapi/v2/errors.go b/internal/cloudapi/v2/errors.go
index e4ace4ddf..d3ce3da8b 100644
--- a/internal/cloudapi/v2/errors.go
+++ b/internal/cloudapi/v2/errors.go
@@ -16,26 +16,27 @@ const (
 	// ocm-sdk sends ErrorUnauthenticated with id 401 & code COMPOSER-401
 	ErrorUnauthenticated ServiceErrorCode = 401
 
-	ErrorUnauthorized            ServiceErrorCode = 2
-	ErrorUnsupportedMediaType    ServiceErrorCode = 3
-	ErrorUnsupportedDistribution ServiceErrorCode = 4
-	ErrorUnsupportedArchitecture ServiceErrorCode = 5
-	ErrorUnsupportedImageType    ServiceErrorCode = 6
-	ErrorInvalidRepository       ServiceErrorCode = 7
-	ErrorDNFError                ServiceErrorCode = 8
-	ErrorInvalidOSTreeRef        ServiceErrorCode = 9
-	ErrorInvalidOSTreeRepo       ServiceErrorCode = 10
-	ErrorFailedToMakeManifest    ServiceErrorCode = 11
-	ErrorInvalidComposeId        ServiceErrorCode = 14
-	ErrorComposeNotFound         ServiceErrorCode = 15
-	ErrorInvalidErrorId          ServiceErrorCode = 16
-	ErrorErrorNotFound           ServiceErrorCode = 17
-	ErrorInvalidPageParam        ServiceErrorCode = 18
-	ErrorInvalidSizeParam        ServiceErrorCode = 19
-	ErrorBodyDecodingError       ServiceErrorCode = 20
-	ErrorResourceNotFound        ServiceErrorCode = 21
-	ErrorMethodNotAllowed        ServiceErrorCode = 22
-	ErrorNotAcceptable           ServiceErrorCode = 23
+	ErrorUnauthorized                 ServiceErrorCode = 2
+	ErrorUnsupportedMediaType         ServiceErrorCode = 3
+	ErrorUnsupportedDistribution      ServiceErrorCode = 4
+	ErrorUnsupportedArchitecture      ServiceErrorCode = 5
+	ErrorUnsupportedImageType         ServiceErrorCode = 6
+	ErrorInvalidRepository            ServiceErrorCode = 7
+	ErrorDNFError                     ServiceErrorCode = 8
+	ErrorInvalidOSTreeRef             ServiceErrorCode = 9
+	ErrorInvalidOSTreeRepo            ServiceErrorCode = 10
+	ErrorFailedToMakeManifest         ServiceErrorCode = 11
+	ErrorInvalidComposeId             ServiceErrorCode = 14
+	ErrorComposeNotFound              ServiceErrorCode = 15
+	ErrorInvalidErrorId               ServiceErrorCode = 16
+	ErrorErrorNotFound                ServiceErrorCode = 17
+	ErrorInvalidPageParam             ServiceErrorCode = 18
+	ErrorInvalidSizeParam             ServiceErrorCode = 19
+	ErrorBodyDecodingError            ServiceErrorCode = 20
+	ErrorResourceNotFound             ServiceErrorCode = 21
+	ErrorMethodNotAllowed             ServiceErrorCode = 22
+	ErrorNotAcceptable                ServiceErrorCode = 23
+	ErrorNoBaseURLInPayloadRepository ServiceErrorCode = 24
 
 	// Internal errors, these are bugs
 	ErrorFailedToInitializeBlueprint              ServiceErrorCode = 1000
@@ -95,6 +96,7 @@ func getServiceErrors() serviceErrors {
 		serviceError{ErrorResourceNotFound, http.StatusNotFound, "Requested resource doesn't exist"},
 		serviceError{ErrorMethodNotAllowed, http.StatusMethodNotAllowed, "Requested method isn't supported for resource"},
 		serviceError{ErrorNotAcceptable, http.StatusNotAcceptable, "Only 'application/json' content is supported"},
+		serviceError{ErrorNoBaseURLInPayloadRepository, http.StatusBadRequest, "BaseURL must be specified for payload repositories"},
 
 		serviceError{ErrorFailedToInitializeBlueprint, http.StatusInternalServerError, "Failed to initialize blueprint"},
 		serviceError{ErrorFailedToGenerateManifestSeed, http.StatusInternalServerError, "Failed to generate manifest seed"},
diff --git a/internal/cloudapi/v2/openapi.v2.gen.go b/internal/cloudapi/v2/openapi.v2.gen.go
index ebdb8dbc5..bcbf598e1 100644
--- a/internal/cloudapi/v2/openapi.v2.gen.go
+++ b/internal/cloudapi/v2/openapi.v2.gen.go
@@ -106,9 +106,10 @@ type ComposeStatus struct {
 
 // Customizations defines model for Customizations.
 type Customizations struct {
-	Packages     *[]string     `json:"packages,omitempty"`
-	Subscription *Subscription `json:"subscription,omitempty"`
-	Users        *[]User       `json:"users,omitempty"`
+	Packages            *[]string     `json:"packages,omitempty"`
+	PayloadRepositories *[]Repository `json:"payload_repositories,omitempty"`
+	Subscription        *Subscription `json:"subscription,omitempty"`
+	Users               *[]User       `json:"users,omitempty"`
 }
 
 // Error defines model for Error.
@@ -248,6 +249,9 @@ type PackageMetadata struct {
 // Repository defines model for Repository.
 type Repository struct {
 	Baseurl    *string `json:"baseurl,omitempty"`
+	CheckGpg   *bool   `json:"check_gpg,omitempty"`
+	GpgKey     *string `json:"gpg_key,omitempty"`
+	IgnoreSsl  *bool   `json:"ignore_ssl,omitempty"`
 	Metalink   *string `json:"metalink,omitempty"`
 	Mirrorlist *string `json:"mirrorlist,omitempty"`
 	Rhsm       bool    `json:"rhsm"`
@@ -479,66 +483,66 @@ func RegisterHandlers(router EchoRouter, si ServerInterface) {
 // Base64 encoded, gzipped, json marshaled Swagger object
 var swaggerSpec = []string{
 
-	"H4sIAAAAAAAC/+xbe2/buLL/KoTOBbKLK/ntNDFQ7EnTnJ7sbZsiTndxbxMEtDS2uJVIlaTiOIW/+wUf",
-	"kvWgY+dsdhcL5J/GNsmZ3wxnhjND9rsXsjRjFKgU3uS7l2GOU5DA7bcFqL8RiJCTTBJGvYn3CS8AERrB",
-	"ved7cI/TLIHa9Duc5OBNvL63XvseUWu+5cBXnu9RnKoRPdP3RBhDitUSucrU70JyQhd6mSAPDt4f83QG",
-	"HLE5IhJSgQhFgMMYWYJVNAWBEk2vtxWPnvsYnnUxqEmf/Do9Ox18zhKGowsNzcjPWQZcEsOfw0Jj/l6g",
-	"8iYe5MEShAz6nt9k4Xsixhxul0TGtzgMWW63pFz9xesPhqPx4auj415/4N34ntaBA25JHHOOV5o2xZmI",
-	"mbw1AlcxpaugGG2jWvseh2854RApAFYmN9abcjWb/QahVHyrmppKLHOHonBK6ohwSoJeeDTsvToevno1",
-	"Hh+Po9HMpbEnqrghjOJb0tgCfjp83l1263MH822Ky3ni9p0qCzXJSf8h57BDOJLiBZQm0/BEnILyQxkD",
-	"yjUZiJBe0EHnEqW5kGgGKKfkW67ChZ64IHdAEQfBch4CWnCWZ51rej5HigkiArGUSAkRmnOW6iVKFhDS",
-	"RxhxTCOWIkYBzbCACDGKMPr8+fwtIuKaLoACxxKizjXdxAJj4RqYy4QSFmJpd7Au4Hs7gpYxcNBYNBUk",
-	"YpYnkRaukBvTCKm9FBK45v9vtkSSoYQIiXCSoIKNmFzTWMpMTLrdiIWik5KQM8HmshOytAs0yEU3TEgX",
-	"q+3pWt/66Y7A8rX+KQgTEiRYgpD/wA+F890qRrclk4OGApQ1Qq621u1FZjtu9XY8vtP1rdtDNc29uGJ5",
-	"iOmlJfNOc3TFwnxWQrglURvU+VsFqTrtPwAzgnF0NBuEAZ4NRsFo1B8Gx71wHBz2B8PeIRz1jmHgQieB",
-	"YiofwaVAmEn7obLmMic0QkQW3qJdFH1iXOJkH7spbEaSOwgiwiGUjK+685xGOAUqcSJao0HMloFkgWId",
-	"GMgNJY3DVzAfzw6DfjicB6MI9wJ8OBgEvVnvsDcYHkevolc7A91GY+29bVlgxSt3RK5tkbEeuPaJBA28",
-	"FQIuCKcqaRJwrg0AJ8nF3Jt8+e79F4e5N/H+0d0kVV2bNnQv9OJLmAMHGoK39lugozrY/mAI6rgP4Oh4",
-	"FvQH0TDAo/FhMBocHo7Ho1Gv1+t5vjdnPMXSm3h5rpW5Q7DIIdDNRqQPIHGEJX5OwZiQHOA2ZGlKpNNl",
-	"foixiH8sPGeWk0QiO93hfhkOv+KFod1MTfWIibuEhkkeEbpAH89+uTzxKvnSY/JYGqUiWtnU+jH9XZrj",
-	"qm2SYS4kS8kDLs/ax0Cc1mevfS8iSgGzXLbSDR5DEhy5FGWsmG8gPcbyXE0u4DfNpsa9Sbgi/sYhn80r",
-	"NCtR0t0pgoXg9mhLZ4sLtLaoDqVqd5WsPGNCLjiIJ2bklSi4S65pde7a93JhC7S9DPqzAL6PFfveGeeM",
-	"P+fmhSwCpzbUJFw53B1JCRZGMY/HM82hnN4g7N5mLeV7YjxiP0n1bIdtFurfax+Mdl0bUbNTTcqN/N3p",
-	"px0Z+ywPv4LcnsNhiuCeCKmi4vTq5OPbk8u3aCoZV1EzTLAQ6I0m0Wlm0PZLYDlsjTbuauEqBpPiS4Zy",
-	"AWjOuM2JMsalzaB1URkhFUpyCeiMLgi1aVPnml6VKZQm1CgwVClq06Z3p59QxplSm4+WMQljVVjkAqJr",
-	"WvC9mFpaJgnT7A2WDlLVCJNIZBCSOVHYbOVxTQ9CE+Z4gDMSXOe93jBUx67+BAfIKKNgh7CoJH4K9VMq",
-	"k01l2ValEtGMV/LLUqYlSRKlmlK5klX1q0orq0/dGylVidV3EmnqRQbWQVMAVKSeYcLyqLNgbJGATjyF",
-	"MR2dk3bL+sOWdFUl+hpimieSBBZ5MR2FCRMgpIKpJplc8Jr+YEuNwjyNYZbLflRqDmMmgCKcS5ZiSUKc",
-	"JKumkiF/QrelUQOqPILNC71ouVExXeHVVOqW7DJfbZ6da3qGw7gwEq31kFGJiSpjC03xIguybJBC3kG/",
-	"aAQm1xMIc5hcU4QCdKDOgsl3SDFJSLQ+mKATivQ3hKOIg1AmiCXikHEQKh5teIWKBGqI1UH/YhxZ7fno",
-	"ACckhH/a72rPDzqWswB+R0I4MeueiMGwtiS28U5XAZOx9rbsnzjLRMZkZ2EXFWuqkHT98FRtWPmLZoTC",
-	"1VBBlBIqnDqIWIoJnXw3fxVD7Z5omhMJyPyKfsg4STFf/dhmniSGoe6iqFPd7D6Wdm1TIxvXO0CMo4MG",
-	"JrfXPW6aRJg1JjgoQ0WYrq5pod+6N33RycekZRWqrqvbw76b5/me2ba2mj3fswqu/viENGtb+9IeYq7S",
-	"rjxjn6+29D17HN02SzwsQqARpjKYcUyiYNgbjvvDnUVchZy/q1StpfXt3isPYyIhlDlviHN/dHh7ONp+",
-	"zpuf98jHr1YZ6ArGlIG71lxMr9QsLXHGBJGMN7Otx5ZfFotWrqTbnPa3LNurCKvnWq32cVV1Na00oLfY",
-	"3hTbss3Enlzt/KLvODYC7kegZudN8YpKqY7VMFKGQvNUT8vDEIQSco5JYlSRAVVlt/YzktiPBpn5XLRK",
-	"1bcbh4VV7KbCCi8VG93kUhEpWkBQ9gjsN32YAi9+IFRInCT6h0WYqX+VG5R+qv/WZt2JTOVTTlRFyVDf",
-	"q6+EuiuY4jbMDhAqYWEKseJmqj0imcSJa6ixOZqpX16jmdsrs9jfWkH4nvUtxyXGvN1S6B51TQzoKl26",
-	"AsHW+4c240al2EIQWwjtYONW7hatt5tdfqErzcGllGa/xxkjnSAgY1tGitPBkdQngIV7TJBFGo23DVFc",
-	"xOgtZ55j4A64IPtU0TZsadibZRu4vlFCiVFFhUqkbZehWIC1jo1RlUVERDscohib3rXyWqCyGxEhu8rw",
-	"jjaWp+gw0WWiW2t08sRljilInBD61c01Jar8Fp05RIxje4J2GF90i3U/qaj92owHw4Gq6QaHSu7X5Vm4",
-	"E4JmkthAUQdRYlDDnRCoZELz/8lq+fVRoE5InFY4Y/Xv4cj8ovG9wQIupntg4bFIKzs/YywBTNs5kZrm",
-	"8otpo0HVcIpQkjvTaPkKq/aNMoQcZKCGKkgzLMSS8cgFV231rdNm2iazh/SECrKIGzfokufgtxTie4wv",
-	"MLV9vzr/QW/UGw6caZDKZIG3IVcbex2l3QrynZldDYnf1HKNaUVlFXFdO9nqGTEKezS9XK8c1v7ONc0r",
-	"811LWk2tnTzaN9e6O/Z42s5+j/hFjrS/9HuuaFYbT5C9WKFE3+R7++VlPKd0W/K1T2JvENjM3p04+sWh",
-	"Us16q+vamd1SdMSwTNVMoudCqBvaz9il1uVnvcTYuLMedL7XaRYXrTgoRBxANBiP+8fo5OTk5HT48QGf",
-	"9pP/e3ve/3h1Nla/nX/k7/7njH/4X/LfHz58Xub/xpcnP6eX79n5w+V88O3tIHo7fui9ubrvHt67QLTr",
-	"UFWf7355sqVevNEvnSDMOZGrqdKgUdEbwNwofaY//asIvz//elU8nNJB1cwr6ar4bZ5PETpn7Qbb1DaA",
-	"JNO3frYRaxJy058QHc/3EhICNWmTfbF1kuEwBjTo9DybiJYn/XK57GA9rI9Xu1Z035+fnn2cngWDTq8T",
-	"yzTRe0ikVtrF9I1mb6+xONKdToQzUsmHJt7A3l1QNTDxhp1ep6/zcBlrNXVtf1iHHyYcjfhTDlgCwojC",
-	"EtnZPsqYSoEITpIVChkVtkPP5kjAHXBc6EKrx7as9bs30zIlHEWgltj2a/Ue5DzyJt4nJqQVzTN2AEK+",
-	"YdHKXNLoBEx7VJYlxLRXu7/Z+5fNo7hHLyzr15/rur2pg9e8NMmY2gtFbdDrPzf388gwbqjcDKIYCyQk",
-	"5hIitY2jXu/Z+NurnTbvc2pax3ani9dMhn//j+d/kktlJF+BIiIQMWgM9+Efz/0zxbmMGScP5hIiA67y",
-	"NlQap0Ey+jOQfKVsSct9MEoY/xkm8JnCfQahhAiBmoNYGOZcuUU11upjrIiyX27WN74n8jTFqroqgkYR",
-	"XNS6ItKI7ncSrfUp5rr3ewfS3KnoQ1nfACJ7+CPGNcUEFDRLTt8LaUsJkzwCgZYxyBi4mkyZoVXoUKcY",
-	"EEHUjjfvQNYfBPi1l8Vf3K+mSsIGrGRooW8a9YtdFWM3D3bts6FqfKk+3332RzQ3reDVe+7gVfbhWhZU",
-	"18tfFruKwPEStl7C1l5h66oReLbHL92DKbpvjwayYqKhOCeUiLgRvgDBPQ4lUhmn8mrCKOIgc04hQhGo",
-	"KkggRquvi4uny+ay9ZFwVnYJXwLazoC2eUHXtq6r6lYWjzLM6/BiK1/i3Euc+3vEuVZsUgaNK4as4p0m",
-	"LirxrRViNu/SWsHFJdlmSlffA21rHVXm6YuiP9T1NzK4rN28y2VzZJXx4mZ/jZsZQ//7ORkuDQgnCcqY",
-	"EGSWQGlNGzfbXRRhatpMNCz/b4tBtnn2N1shfXS6HXW/DKCk+3tP/eGffIaXW/nioy8++hQfNWurpLVf",
-	"lk3T7effhZ3ituo6WEtOeysiFCkd2NeRf8fM4VFx1uVlo4kz9W43zkhHLRcxsf8ZDGeka1626JY68KB4",
-	"tdy9G3hNKT7YF4osykPzrNbw0vlEm5WQeAG/i+FU4gWhizabJ9LRuqbFQ0lvfbP+/wAAAP//6r7bvs0+",
-	"AAA=",
+	"H4sIAAAAAAAC/+xbe28bOZL/KkTvAZ7BdUuyZDmOgMGs43iz3ps8YDmzuIsNgWKX1Nx0kx2SbUUJ9N0P",
+	"fHSrH5Ql73hmMYD/iSWRrPpVsapYVWS+B4RnOWfAlAwm34McC5yBAuG+LUH/jUESQXNFOQsmwQe8BERZ",
+	"DF+DMICvOMtTaEy/x2kBwSQ4DjabMKB6zZcCxDoIA4YzPWJmhoEkCWRYL1HrXP8ulaBsaZZJ+s3D+12R",
+	"zUEgvkBUQSYRZQgwSZAjWEdTEqjQDAY78Zi5D+HZlIOG9Pk/p5cXw495ynH83kCz8gueg1DU8hewNJi/",
+	"l6iCSQBFtAKpouMgbLMIA5lgAbMVVckME8ILtyXV6k/B8XB0Mj59cfZycDwM7sLA6MADtyKOhcBrQ5vh",
+	"XCZczazAdUzZOipHu6g2YSDgS0EFxBqAk8mP9a5azef/AqI037qmpgqrwqMonNEmIpzRaEDORoMXL0cv",
+	"XozHL8fxydynsUequCWM5lvR2AF+OnraXfbrcw/zXYorROr3nToLPclL/1shYI9wNMNLqEym5Yk4A+2H",
+	"KgFUGDIQI7Ogh64Uygqp0BxQweiXQocLM3FJ74EhAZIXggBaCl7kvVt2tUCaCaIS8YwqBTFaCJ6ZJVoW",
+	"kCpEGAnMYp4hzgDNsYQYcYYw+vjx6jWi8pYtgYHACuLeLdvGAmvhBpjPhFJOsHI72BTwFzeCVgkIMFgM",
+	"FSQTXqSxEa6UG7MY6b2UCoTh/3e+QoqjlEqFcJqiko2c3LJEqVxO+v2YE9nLKBFc8oXqEZ71gUWF7JOU",
+	"9rHenr7zrZ/vKax+Mj9FJKVRihVI9Rf8rXS+mWY0q5gctRSgrREKvbV+L7LbMTPb8fBON7fuANW09+KG",
+	"FwSza0fmjeHoi4XFvIIwo3EX1NVrDak+7d8AcwLj+Gw+JBGeD0+ik5PjUfRyQMbR6fFwNDiFs8FLGPrQ",
+	"KWCYqQdwaRB20mGonLksKIsRVaW3GBdFH7hQOD3EbkqbUfQeopgKIIqLdX9RsBhnwBROZWc0SvgqUjzS",
+	"rCMLuaWkMXkBi/H8NDomo0V0EuNBhE+Hw2gwH5wOhqOX8Yv4xd5At9VYd287Fljzyj2Ra1dkbAauQyJB",
+	"C2+NgA/ChU6aJFwZA8Bp+n4RTD59D/5LwCKYBH/pb5Oqvksb+u/N4mtYgABGINiEHdBxE+zxcAT6uI/g",
+	"7OU8Oh7GowifjE+jk+Hp6Xh8cjIYDAZBGCy4yLAKJkFRGGXuESz2CHS3FektKBxjhZ9SMC6VAJgRnmVU",
+	"eV3mhwTL5MfSc+YFTRVy0z3ul2PyGS8t7XZqakZs3KWMpEVM2RK9u/z1+jyo5UsPyeNoVIroZFObh/R3",
+	"bY+rrkmSQiqe0W+4OmsfAnHRnL0Jg5hqBcwL1Uk3RAJpdOZTlLVisYX0EMsrPbmE3zabBvc24Zr4W4d8",
+	"Mq8wrGRFd68IDoLfox2dHS7Q2aImlLrd1bLynEu1FCAfl5HneK0j2ExAziVVXJTyHmKj1+WitTfZrwXY",
+	"fZSm9bmbMCikq/0OwvFRgjjEQcLgUgguntIuCI/Bq2g9CdfyBk++g6VVzMOh0nCoprcI+y3ISPkLtc52",
+	"mKRmtsfsS/UftA9Wu76NaLiAIeVH/ubiw55iYF6Qz6B2p4eYIfhKpdIBd3pz/u71+fVrNFVc6IBMUiwl",
+	"emVI9NrJufsSOQ47A5m/ELlJwFYPiqNCAlpw4dKtnAvlknNTr8ZIR6lCAbpkS8pcRta7ZTdVdmYItWoX",
+	"XeW6jOzNxQeUC67VFqJVQkmia5ZCQnzLSr7vp46Wze8Me4ulh3ShwxWSORC6oBqbK2pu2RGxEVREOKfR",
+	"bTEYjIg+0c0nOEJWGSU7hGUtp9SoH1P0bIvWriq1iHa8lrpWMq1ommrVVMpVvK5fXbU5fZq2S6VKrL/T",
+	"2FAvk7semgKgMqslKS/i3pLzZQomp5XWdEy6269KG1ct1pUYGohZkSoaOeTldERSLkEqDVNPsmnmLfvB",
+	"VTGleVrDrJb9qNVMEi6BIVwonmFFCU7TdVvJUDyikdMqL3WKwhelXozcqJyu8RoqTUv2ma8xz94tu8Qk",
+	"KY3EaJ1wpjDVFXKpKVEmWI4N0sh76FeDwKaREmEBk1uGUISO9Fkw+Q4ZpimNN0cTdM6Q+YZwHAuQ2gSx",
+	"QgJyAVLHoy0vokmgllg99DcukNNeiI5wSgn81X3Xe37Uc5wliHtK4NyueyQGy9qR2MU7W0dcJcbb8r/i",
+	"PJc5V72lW1SuqUMypcljteHkL/scGldLBXFGmfTqIOYZpmzy3f7VDI17omlBFSD7K/ohFzTDYv1jl3ma",
+	"WoamQaNPdbv7WLm1bY1sXe8IcYGOWpj8XvewaVJp19jgoA0VYba+ZaV+m970ySQfk45V6JKxaQ+Hbl4Q",
+	"BnbbumoOwsApuP7jIzK4XZ1Rd4j5qsbqjH26sjUM3HE0a1ePWBJgMWYqmgtM42g0GI2PR3vrwxq5cF8V",
+	"3KgYum1dQRKqgKhCtMT5enY6Oz3Zfc7bnw9I9W/WOZjiyFaY+9a8n97oWUbiJ0+67Wk/4/lB9V0z1+p0",
+	"puuqa2ilBb3D9q7cll0m9uhC6ldzfbIV8DACDTtvi1cWYU2slpE2FFZkZlpBCEgt5ALT1KoiB6YreuNn",
+	"NHUfLTL7uezC6m93Hgur2U2NFV5pNqZ/piNSvISoaj+4b+YwBVH+QJlUOE3ND0uS63+1G1R+av42Zt3L",
+	"XOdTXlRlydDcq8+U+SuY8qLNDVCmYGkLsfLSqzuiuMKpb6i1OYZpWN3Q2YsxuzjcWUGEgfMtz/3Iotut",
+	"6J/1bQzoa136AsHOq40u41al2EGQOAjdYONX7g6td/toYakrw8GnlHYryRsjvSAg5ztGytPBk9SngKV/",
+	"TNJlFo93DTFcxugdZ55n4B6EpIdU0S5sGdjbZVu4oVVChVFHhVqk7ZahWIKzjq1RVUVEzHoC4gTbtrj2",
+	"WmCqH1Op+trwzraWp+lw2eey3+ihitRnjiQB8nm2zJc1eeecp4BN02SZL2efYe23siXjAmZSpv61GSic",
+	"UvbZL1BGdWUvewuIucDucO5xseyX637WB8JPdjwaDXW5ODzVKv2pOmb3SWeZpC4GNUFUGPRwjwBTXBr+",
+	"P7sN/Oks0ocvzmqcsf739MT+YvC9whLeTw/AIhKZ+RTVTrf0NJ/LTVu9r5a/EUXvbQ/H7VfzHhyIABXp",
+	"oRrSHEu54iL2wdVWNPOaY9caD5CeMkmXSeveX4kCQo/lcLHEzHUrm/yHg5PBaOjNsHSSDKILud4z7Gnt",
+	"1pDvTRobSMK2lhtMayqrievbyU47ijM4oJ/me5uxCfeuaV/071vS6Zft5dG9bzeNt4crAv5bxC/Tr8Ol",
+	"P3BFu5B5hOzlCi36NpU8LOUTBWO78rpDagaLwBUN/pw0LM+rekJdX9dNGleyJ0dVFmhzSB9C0yt/wga4",
+	"qWyb1cvWnc2g95VRu27pxEEpkwji4Xh8/BKdn5+fX4zefcMXx+n/vb46fndzOda/Xb0Tb/7nUrz9X/rf",
+	"b99+XBV/x9fn/8iuf+FX364Xwy+vh/Hr8bfBq5uv/dOvPhDdEleX/vvfy+woRe/M+ywghaBqPdUatCp6",
+	"BVhYpc/Np7+V4fcf/7wpn3uZoGrnVXR1/LaPvihb8G7vbup6S4qbu0rX47W5vm19yF4QBiklwGxG5t6Z",
+	"neeYJICGvUHgctzqpF+tVj1shs3x6tbK/i9XF5fvppfRsDfoJSpLzR5SZZT2fvrKsHeXbwKZJirCOa2l",
+	"WpNg6K5FmB6YBKPeoHdsUnyVGDX1XevZhB8uPT3+CwFYAcKIwQq52SHKuc6uKE7TNSKcSdf85wsk4R4E",
+	"LnVh1OO64ea1nu3GUoFi0EtcZ7d+xXIVB5PgA5fKiRZYOwCpXvF4be9/TG5nPCrPU2o7t/1/uaud7VO+",
+	"B69Zm5e2m6a96YPXvo/Jud4LTW04OH5q7lexZdxSuR1ECZZIKiwUxHobTwaDJ+Pvbo26vK+Y7Uq7nS7f",
+	"YFn+x78///NCaSP5DAxRiahFY7mPfn/uHxkuVMIF/WbvN3IQOm9DlXFaJCd/BJLPjK9YtQ9WCeM/wgQ+",
+	"MviaA1EQI9BzECekENot6rHWHGNllP10t7kLA1lkGdaFWxk0yuCi15WRRva/03hjTjHfleIbUPa6xhzK",
+	"5nIRucMfcWEopqChOXLmyslYCkmLGCRaJaASEHoy45ZWqUOTYkAMcTfevAHVfMYQNt5Df/K/9aoIW7CK",
+	"o6W5xDTvjHWM3T4zdo+d6vGl/uj4yZ/+3HWC1+Cpg1fV4utYUFMv/7HYVQaO57D1HLYOCls3rcCzO36Z",
+	"HkzZ2HswkJUTLcUFZVQmrfAFCL5iopDOOLVXU86QAFUIBjGKQVdBEnFWfxNdPri297gPhLOqAfkc0PYG",
+	"tO27v6513dS3snzvYd+0l1v5HOee49yfI851YpM2aFwzZB3vDHFZi2+dELN98tYJLj7JtlP65oppV+uo",
+	"Ns/cQf2urr+VwWft9jUxXyCnjGc3+8+4mTX0P5+T4cqAcJqinEtJ5ylU1rR1s/1FEWa2zcRI9T9yLLLt",
+	"i8L5Gpmj0++oh2UAFd3feuqP/uAzvNrKZx999tHH+KhdWydt/LJqmu4+/967KX6rboJ15Iy3IsqQ1oF7",
+	"ePlnzBweFGdTXTbaONPsduOc9vRymVD3X9hwTvv20YxpqYOIygfR/fth0JbirXv8yOOC2Be7lpfJJ7qs",
+	"pMJL+E0MpwovKVt22TySjtE1K99gBpu7zf8HAAD//xaq+MiDPwAA",
 }
 
 // GetSwagger returns the Swagger specification corresponding to the generated code
diff --git a/internal/cloudapi/v2/openapi.v2.yml b/internal/cloudapi/v2/openapi.v2.yml
index 3df3c8dfa..a68069b13 100644
--- a/internal/cloudapi/v2/openapi.v2.yml
+++ b/internal/cloudapi/v2/openapi.v2.yml
@@ -531,6 +531,12 @@ components:
           type: string
           format: url
           example: 'https://mirrors.fedoraproject.org/metalink?repo=fedora-32&arch=x86_64'
+        gpg_key:
+          type: string
+        check_gpg:
+          type: boolean
+        ignore_ssl:
+          type: boolean
     UploadOptions:
       oneOf:
       - $ref: '#/components/schemas/AWSEC2UploadOptions'
@@ -669,6 +675,10 @@ components:
           type: array
           items:
             $ref: '#/components/schemas/User'
+        payload_repositories:
+          type: array
+          items:
+            $ref: '#/components/schemas/Repository'
     OSTree:
       type: object
       properties:
diff --git a/internal/cloudapi/v2/v2.go b/internal/cloudapi/v2/v2.go
index 45646d3b1..958c85ef4 100644
--- a/internal/cloudapi/v2/v2.go
+++ b/internal/cloudapi/v2/v2.go
@@ -151,6 +151,11 @@ func (h *apiHandlers) PostCompose(ctx echo.Context) error {
 		}
 	}
 
+	var payloadRepositories []Repository
+	if request.Customizations != nil && request.Customizations.PayloadRepositories != nil {
+		payloadRepositories = *request.Customizations.PayloadRepositories
+	}
+
 	// use the same seed for all images so we get the same IDs
 	bigSeed, err := rand.Int(rand.Reader, big.NewInt(math.MaxInt64))
 	if err != nil {
@@ -182,13 +187,37 @@ func (h *apiHandlers) PostCompose(ctx echo.Context) error {
 		}
 	}
 
+	payloadPackageSets := imageType.PayloadPackageSets()
+	packageSetsRepositories := make(map[string][]rpmmd.RepoConfig, len(payloadPackageSets))
+
+	for _, packageSetKey := range payloadPackageSets {
+		packageSetsRepositories[packageSetKey] = make([]rpmmd.RepoConfig, len(payloadRepositories))
+		for j, repo := range payloadRepositories {
+			if repo.Baseurl != nil {
+				packageSetsRepositories[packageSetKey][j].BaseURL = *repo.Baseurl
+			} else {
+				return HTTPError(ErrorNoBaseURLInPayloadRepository)
+			}
+			if repo.GpgKey != nil {
+				packageSetsRepositories[packageSetKey][j].GPGKey = *repo.GpgKey
+			}
+			if repo.CheckGpg != nil {
+				packageSetsRepositories[packageSetKey][j].CheckGPG = *repo.CheckGpg
+			}
+			if repo.IgnoreSsl != nil {
+				packageSetsRepositories[packageSetKey][j].IgnoreSSL = *repo.IgnoreSsl
+			}
+		}
+	}
+
 	packageSets := imageType.PackageSets(bp)
 	depsolveJobID, err := h.server.workers.EnqueueDepsolve(&worker.DepsolveJob{
-		PackageSets:      packageSets,
-		Repos:            repositories,
-		ModulePlatformID: distribution.ModulePlatformID(),
-		Arch:             arch.Name(),
-		Releasever:       distribution.Releasever(),
+		PackageSets:             packageSets,
+		Repos:                   repositories,
+		ModulePlatformID:        distribution.ModulePlatformID(),
+		Arch:                    arch.Name(),
+		Releasever:              distribution.Releasever(),
+		PackageSetsRepositories: packageSetsRepositories,
 	})
 	if err != nil {
 		return HTTPErrorWithInternal(ErrorEnqueueingJob, err)
diff --git a/internal/distro/distro.go b/internal/distro/distro.go
index b2540df3e..622f322ec 100644
--- a/internal/distro/distro.go
+++ b/internal/distro/distro.go
@@ -108,6 +108,9 @@ type ImageType interface {
 	// Returns the names of the pipelines that create the image.
 	PayloadPipelines() []string
 
+	// Returns the package set names safe to install custom packages via custom repositories.
+	PayloadPackageSets() []string
+
 	// Returns the names of the stages that will produce the build output.
 	Exports() []string
 
@@ -267,3 +270,7 @@ func PayloadPipelinesFallback() []string {
 func ExportsFallback() []string {
 	return []string{"assembler"}
 }
+
+func PayloadPackageSets() []string {
+	return []string{}
+}
diff --git a/internal/distro/fedora33/distro.go b/internal/distro/fedora33/distro.go
index cef67b49c..ac9ca372e 100644
--- a/internal/distro/fedora33/distro.go
+++ b/internal/distro/fedora33/distro.go
@@ -247,6 +247,10 @@ func (t *imageType) PayloadPipelines() []string {
 	return distro.PayloadPipelinesFallback()
 }
 
+func (t *imageType) PayloadPackageSets() []string {
+	return []string{"packages"}
+}
+
 func (t *imageType) Exports() []string {
 	return distro.ExportsFallback()
 }
diff --git a/internal/distro/rhel8/distro.go b/internal/distro/rhel8/distro.go
index b9212f240..96909fd8e 100644
--- a/internal/distro/rhel8/distro.go
+++ b/internal/distro/rhel8/distro.go
@@ -237,6 +237,10 @@ func (t *imageType) PayloadPipelines() []string {
 	return distro.PayloadPipelinesFallback()
 }
 
+func (t *imageType) PayloadPackageSets() []string {
+	return []string{"packages"}
+}
+
 func (t *imageType) Exports() []string {
 	return distro.ExportsFallback()
 }
diff --git a/internal/distro/rhel84/distro.go b/internal/distro/rhel84/distro.go
index 9d9860edd..23d5387a1 100644
--- a/internal/distro/rhel84/distro.go
+++ b/internal/distro/rhel84/distro.go
@@ -281,6 +281,10 @@ func (t *imageType) PayloadPipelines() []string {
 	return distro.PayloadPipelinesFallback()
 }
 
+func (t *imageType) PayloadPackageSets() []string {
+	return []string{"packages"}
+}
+
 func (t *imageType) Exports() []string {
 	if len(t.exports) > 0 {
 		return t.exports
diff --git a/internal/distro/rhel84/distro_v2.go b/internal/distro/rhel84/distro_v2.go
index 0c020c1d7..8a8dc1b2c 100644
--- a/internal/distro/rhel84/distro_v2.go
+++ b/internal/distro/rhel84/distro_v2.go
@@ -127,6 +127,10 @@ func (t *imageTypeS2) PayloadPipelines() []string {
 	return t.payloadPipelines
 }
 
+func (t *imageTypeS2) PayloadPackageSets() []string {
+	return []string{"packages"}
+}
+
 func (t *imageTypeS2) Exports() []string {
 	return t.exports
 }
diff --git a/internal/distro/rhel85/distro.go b/internal/distro/rhel85/distro.go
index 1948b94a5..9c861be85 100644
--- a/internal/distro/rhel85/distro.go
+++ b/internal/distro/rhel85/distro.go
@@ -283,6 +283,10 @@ func (t *imageType) PayloadPipelines() []string {
 	return t.payloadPipelines
 }
 
+func (t *imageType) PayloadPackageSets() []string {
+	return []string{blueprintPkgsKey}
+}
+
 func (t *imageType) Exports() []string {
 	return t.exports
 }
diff --git a/internal/distro/rhel86/distro.go b/internal/distro/rhel86/distro.go
index 34801c3df..f80241e2e 100644
--- a/internal/distro/rhel86/distro.go
+++ b/internal/distro/rhel86/distro.go
@@ -314,6 +314,10 @@ func (t *imageType) PayloadPipelines() []string {
 	return t.payloadPipelines
 }
 
+func (t *imageType) PayloadPackageSets() []string {
+	return []string{blueprintPkgsKey}
+}
+
 func (t *imageType) Exports() []string {
 	if len(t.exports) > 0 {
 		return t.exports
diff --git a/internal/distro/rhel90/distro.go b/internal/distro/rhel90/distro.go
index 29ecec37e..9588846c2 100644
--- a/internal/distro/rhel90/distro.go
+++ b/internal/distro/rhel90/distro.go
@@ -314,6 +314,10 @@ func (t *imageType) PayloadPipelines() []string {
 	return t.payloadPipelines
 }
 
+func (t *imageType) PayloadPackageSets() []string {
+	return []string{blueprintPkgsKey}
+}
+
 func (t *imageType) Exports() []string {
 	if len(t.exports) > 0 {
 		return t.exports
diff --git a/internal/distro/rhel90beta/distro.go b/internal/distro/rhel90beta/distro.go
index 4387c9b87..3da062b07 100644
--- a/internal/distro/rhel90beta/distro.go
+++ b/internal/distro/rhel90beta/distro.go
@@ -318,6 +318,10 @@ func (t *imageType) PayloadPipelines() []string {
 	return t.payloadPipelines
 }
 
+func (t *imageType) PayloadPackageSets() []string {
+	return []string{blueprintPkgsKey}
+}
+
 func (t *imageType) Exports() []string {
 	return t.exports
 }
diff --git a/internal/prometheus/job_metrics.go b/internal/prometheus/job_metrics.go
index 3f87f7072..9c6e17b5e 100644
--- a/internal/prometheus/job_metrics.go
+++ b/internal/prometheus/job_metrics.go
@@ -31,7 +31,7 @@ var (
 		Namespace: namespace,
 		Subsystem: workerSubsystem,
 		Help:      "Duration spent by workers on a job.",
-		Buckets:   []float64{.1, .2, .5, 1, 2, 4, 8, 16, 32, 64, 96, 128, 160, 192, 224, 256, 320, 382, 448, 512, 640, 768, 896, 1024, 1280, 1536, 1792, 2049},
+		Buckets:   []float64{.1, .2, .5, 1, 2, 4, 8, 16, 32, 40, 48, 64, 96, 128, 160, 192, 224, 256, 320, 382, 448, 512, 640, 768, 896, 1024, 1280, 1536, 1792, 2049},
 	}, []string{"type"})
 )
 
@@ -41,6 +41,6 @@ var (
 		Namespace: namespace,
 		Subsystem: workerSubsystem,
 		Help:      "Duration a job spends on the queue.",
-		Buckets:   []float64{.1, .2, .5, 1, 2, 4, 8, 16, 32, 64, 96, 128, 160, 192, 224, 256, 320, 382, 448, 512, 640, 768, 896, 1024, 1280, 1536, 1792, 2049},
+		Buckets:   []float64{.1, .2, .5, 1, 2, 4, 8, 16, 32, 40, 48, 64, 96, 128, 160, 192, 224, 256, 320, 382, 448, 512, 640, 768, 896, 1024, 1280, 1536, 1792, 2049},
 	}, []string{"type"})
 )
diff --git a/internal/rpmmd/repository.go b/internal/rpmmd/repository.go
index a338c01b3..1f87ac431 100644
--- a/internal/rpmmd/repository.go
+++ b/internal/rpmmd/repository.go
@@ -1,12 +1,15 @@
 package rpmmd
 
 import (
+	"bytes"
+	"context"
 	"encoding/json"
 	"fmt"
 	"io/ioutil"
 	"log"
+	"net"
+	"net/http"
 	"os"
-	"os/exec"
 	"path/filepath"
 	"sort"
 	"strconv"
@@ -313,7 +316,7 @@ func LoadRepositories(confPaths []string, distro string) (map[string][]RepoConfi
 	return repoConfigs, nil
 }
 
-func runDNF(dnfJsonPath string, command string, arguments interface{}, result interface{}) error {
+func runDNF(command string, arguments interface{}, result interface{}) error {
 	var call = struct {
 		Command   string      `json:"command"`
 		Arguments interface{} `json:"arguments,omitempty"`
@@ -322,39 +325,31 @@ func runDNF(dnfJsonPath string, command string, arguments interface{}, result in
 		arguments,
 	}
 
-	cmd := exec.Command(dnfJsonPath)
-
-	stdin, err := cmd.StdinPipe()
-	if err != nil {
-		return err
-	}
-
-	cmd.Stderr = os.Stderr
-	stdout, err := cmd.StdoutPipe()
-	if err != nil {
-		return err
+	httpc := http.Client{
+		Transport: &http.Transport{
+			DialContext: func(_ context.Context, _, _ string) (net.Conn, error) {
+				return net.Dial("unix", "/run/osbuild-dnf-json/api.sock")
+			},
+		},
 	}
 
-	err = cmd.Start()
+	bpost, err := json.Marshal(call)
 	if err != nil {
 		return err
 	}
 
-	err = json.NewEncoder(stdin).Encode(call)
+	response, err := httpc.Post("http://unix", "application/json", bytes.NewReader(bpost))
 	if err != nil {
 		return err
 	}
-	stdin.Close()
+	defer response.Body.Close()
 
-	output, err := ioutil.ReadAll(stdout)
+	output, err := ioutil.ReadAll(response.Body)
 	if err != nil {
 		return err
 	}
 
-	err = cmd.Wait()
-
-	const DnfErrorExitCode = 10
-	if runError, ok := err.(*exec.ExitError); ok && runError.ExitCode() == DnfErrorExitCode {
+	if response.StatusCode != http.StatusOK {
 		var dnfError DNFError
 		err = json.Unmarshal(output, &dnfError)
 		if err != nil {
@@ -376,10 +371,9 @@ func runDNF(dnfJsonPath string, command string, arguments interface{}, result in
 type rpmmdImpl struct {
 	CacheDir      string
 	subscriptions *rhsm.Subscriptions
-	dnfJsonPath   string
 }
 
-func NewRPMMD(cacheDir, dnfJsonPath string) RPMMD {
+func NewRPMMD(cacheDir string) RPMMD {
 	subscriptions, err := rhsm.LoadSystemSubscriptions()
 	if err != nil {
 		log.Println("Failed to load subscriptions. osbuild-composer will fail to build images if the "+
@@ -391,7 +385,6 @@ func NewRPMMD(cacheDir, dnfJsonPath string) RPMMD {
 	return &rpmmdImpl{
 		CacheDir:      cacheDir,
 		subscriptions: subscriptions,
-		dnfJsonPath:   dnfJsonPath,
 	}
 }
 
@@ -442,7 +435,7 @@ func (r *rpmmdImpl) FetchMetadata(repos []RepoConfig, modulePlatformID, arch, re
 		Packages  PackageList       `json:"packages"`
 	}
 
-	err := runDNF(r.dnfJsonPath, "dump", arguments, &reply)
+	err := runDNF("dump", arguments, &reply)
 
 	sort.Slice(reply.Packages, func(i, j int) bool {
 		return reply.Packages[i].Name < reply.Packages[j].Name
@@ -477,7 +470,7 @@ func (r *rpmmdImpl) Depsolve(packageSet PackageSet, repos []RepoConfig, modulePl
 		Checksums    map[string]string `json:"checksums"`
 		Dependencies []dnfPackageSpec  `json:"dependencies"`
 	}
-	err := runDNF(r.dnfJsonPath, "depsolve", arguments, &reply)
+	err := runDNF("depsolve", arguments, &reply)
 
 	dependencies := make([]PackageSpec, len(reply.Dependencies))
 	for i, pack := range reply.Dependencies {
diff --git a/internal/worker/api/errors.go b/internal/worker/api/errors.go
index fc35af279..844b30104 100644
--- a/internal/worker/api/errors.go
+++ b/internal/worker/api/errors.go
@@ -148,14 +148,17 @@ func apiErrorFromEchoError(echoError *echo.HTTPError) ServiceErrorCode {
 
 // Convert an echo error into an AOC compliant one so we send a correct json error response
 func HTTPErrorHandler(echoError error, c echo.Context) {
-	doResponse := func(code ServiceErrorCode, c echo.Context) {
+	doResponse := func(code ServiceErrorCode, c echo.Context, internal error) {
 		if !c.Response().Committed {
 			var err error
 			sec := find(code)
 			apiErr := APIError(code, sec, c)
 
 			if sec.httpStatus == http.StatusInternalServerError {
-				c.Logger().Errorf("Internal server error. Code: %s, OperationId: %s",
+				c.Logger().Errorf("Internal server error. Internal: %s, Code: %s, OperationId: %s",
+					internal, apiErr.Code, apiErr.OperationId)
+			} else {
+				c.Logger().Infof("%s, Code: %s, OperationId: %s", internal,
 					apiErr.Code, apiErr.OperationId)
 			}
 
@@ -172,15 +175,15 @@ func HTTPErrorHandler(echoError error, c echo.Context) {
 
 	he, ok := echoError.(*echo.HTTPError)
 	if !ok {
-		doResponse(ErrorNotHTTPError, c)
+		doResponse(ErrorNotHTTPError, c, he.Internal)
 		return
 	}
 
 	sec, ok := he.Message.(ServiceErrorCode)
 	if !ok {
 		// No service code was set, so Echo threw this error
-		doResponse(apiErrorFromEchoError(he), c)
+		doResponse(apiErrorFromEchoError(he), c, he.Internal)
 		return
 	}
-	doResponse(sec, c)
+	doResponse(sec, c, nil)
 }
diff --git a/internal/worker/json.go b/internal/worker/json.go
index e194f2e9e..321fb2a69 100644
--- a/internal/worker/json.go
+++ b/internal/worker/json.go
@@ -94,11 +94,12 @@ func (pn *PipelineNames) All() []string {
 }
 
 type DepsolveJob struct {
-	PackageSets      map[string]rpmmd.PackageSet `json:"package_sets"`
-	Repos            []rpmmd.RepoConfig          `json:"repos"`
-	ModulePlatformID string                      `json:"module_platform_id"`
-	Arch             string                      `json:"arch"`
-	Releasever       string                      `json:"releasever"`
+	PackageSets             map[string]rpmmd.PackageSet   `json:"package_sets"`
+	Repos                   []rpmmd.RepoConfig            `json:"repos"`
+	ModulePlatformID        string                        `json:"module_platform_id"`
+	Arch                    string                        `json:"arch"`
+	Releasever              string                        `json:"releasever"`
+	PackageSetsRepositories map[string][]rpmmd.RepoConfig `json:"package_sets_repositories,omitempty"`
 }
 
 type ErrorType string
diff --git a/internal/worker/server.go b/internal/worker/server.go
index 855296b3e..23faf5689 100644
--- a/internal/worker/server.go
+++ b/internal/worker/server.go
@@ -328,7 +328,7 @@ type apiHandlers struct {
 func (h *apiHandlers) GetOpenapi(ctx echo.Context) error {
 	spec, err := api.GetSwagger()
 	if err != nil {
-		return api.HTTPError(api.ErrorFailedLoadingOpenAPISpec)
+		return api.HTTPErrorWithInternal(api.ErrorFailedLoadingOpenAPISpec, err)
 	}
 	return ctx.JSON(http.StatusOK, spec)
 }
@@ -347,7 +347,7 @@ func (h *apiHandlers) GetStatus(ctx echo.Context) error {
 func (h *apiHandlers) GetError(ctx echo.Context, id string) error {
 	errorId, err := strconv.Atoi(id)
 	if err != nil {
-		return api.HTTPError(api.ErrorInvalidErrorId)
+		return api.HTTPErrorWithInternal(api.ErrorInvalidErrorId, err)
 	}
 
 	apiError := api.APIError(api.ServiceErrorCode(errorId), nil, ctx)
@@ -407,7 +407,7 @@ func (h *apiHandlers) RequestJob(ctx echo.Context) error {
 func (h *apiHandlers) GetJob(ctx echo.Context, tokenstr string) error {
 	token, err := uuid.Parse(tokenstr)
 	if err != nil {
-		return api.HTTPError(api.ErrorMalformedJobToken)
+		return api.HTTPErrorWithInternal(api.ErrorMalformedJobToken, err)
 	}
 
 	jobId, err := h.server.jobs.IdFromToken(token)
@@ -416,7 +416,7 @@ func (h *apiHandlers) GetJob(ctx echo.Context, tokenstr string) error {
 		case jobqueue.ErrNotExist:
 			return api.HTTPError(api.ErrorJobNotFound)
 		default:
-			return api.HTTPError(api.ErrorResolvingJobId)
+			return api.HTTPErrorWithInternal(api.ErrorResolvingJobId, err)
 		}
 	}
 
@@ -451,7 +451,7 @@ func (h *apiHandlers) GetJob(ctx echo.Context, tokenstr string) error {
 func (h *apiHandlers) UpdateJob(ctx echo.Context, idstr string) error {
 	token, err := uuid.Parse(idstr)
 	if err != nil {
-		return api.HTTPError(api.ErrorMalformedJobId)
+		return api.HTTPErrorWithInternal(api.ErrorMalformedJobId, err)
 	}
 
 	var body api.UpdateJobRequest
@@ -468,7 +468,7 @@ func (h *apiHandlers) UpdateJob(ctx echo.Context, idstr string) error {
 		case ErrJobNotRunning:
 			return api.HTTPError(api.ErrorJobNotRunning)
 		default:
-			return api.HTTPError(api.ErrorFinishingJob)
+			return api.HTTPErrorWithInternal(api.ErrorFinishingJob, err)
 		}
 	}
 
@@ -482,7 +482,7 @@ func (h *apiHandlers) UpdateJob(ctx echo.Context, idstr string) error {
 func (h *apiHandlers) UploadJobArtifact(ctx echo.Context, tokenstr string, name string) error {
 	token, err := uuid.Parse(tokenstr)
 	if err != nil {
-		return api.HTTPError(api.ErrorMalformedJobId)
+		return api.HTTPErrorWithInternal(api.ErrorMalformedJobId, err)
 	}
 
 	request := ctx.Request()
@@ -490,19 +490,19 @@ func (h *apiHandlers) UploadJobArtifact(ctx echo.Context, tokenstr string, name
 	if h.server.artifactsDir == "" {
 		_, err := io.Copy(ioutil.Discard, request.Body)
 		if err != nil {
-			return api.HTTPError(api.ErrorDiscardingArtifact)
+			return api.HTTPErrorWithInternal(api.ErrorDiscardingArtifact, err)
 		}
 		return ctx.NoContent(http.StatusOK)
 	}
 
 	f, err := os.Create(path.Join(h.server.artifactsDir, "tmp", token.String(), name))
 	if err != nil {
-		return api.HTTPError(api.ErrorDiscardingArtifact)
+		return api.HTTPErrorWithInternal(api.ErrorDiscardingArtifact, err)
 	}
 
 	_, err = io.Copy(f, request.Body)
 	if err != nil {
-		return api.HTTPError(api.ErrorWritingArtifact)
+		return api.HTTPErrorWithInternal(api.ErrorWritingArtifact, err)
 	}
 
 	return ctx.NoContent(http.StatusOK)
@@ -523,7 +523,7 @@ func (b binder) Bind(i interface{}, ctx echo.Context) error {
 
 	err := json.NewDecoder(request.Body).Decode(i)
 	if err != nil {
-		return api.HTTPError(api.ErrorBodyDecodingError)
+		return api.HTTPErrorWithInternal(api.ErrorBodyDecodingError, err)
 	}
 
 	return nil
diff --git a/osbuild-composer.spec b/osbuild-composer.spec
index f34acce21..9c8dbc026 100644
--- a/osbuild-composer.spec
+++ b/osbuild-composer.spec
@@ -167,34 +167,34 @@ go build -tags=integration -ldflags="${TEST_LDFLAGS}" -o _bin/osbuild-mock-openi
 %endif
 
 %install
-install -m 0755 -vd                                             %{buildroot}%{_libexecdir}/osbuild-composer
-install -m 0755 -vp _bin/osbuild-composer                       %{buildroot}%{_libexecdir}/osbuild-composer/
-install -m 0755 -vp _bin/osbuild-worker                         %{buildroot}%{_libexecdir}/osbuild-composer/
-install -m 0755 -vp dnf-json                                    %{buildroot}%{_libexecdir}/osbuild-composer/
+install -m 0755 -vd                                                %{buildroot}%{_libexecdir}/osbuild-composer
+install -m 0755 -vp _bin/osbuild-composer                          %{buildroot}%{_libexecdir}/osbuild-composer/
+install -m 0755 -vp _bin/osbuild-worker                            %{buildroot}%{_libexecdir}/osbuild-composer/
+install -m 0755 -vp dnf-json                                       %{buildroot}%{_libexecdir}/osbuild-composer/
 
 # Only include repositories for the distribution and release
-install -m 0755 -vd                                             %{buildroot}%{_datadir}/osbuild-composer/repositories
+install -m 0755 -vd                                                %{buildroot}%{_datadir}/osbuild-composer/repositories
 # CentOS also defines rhel so we check for centos first
 %if 0%{?centos}
 
 # CentOS 9 supports building for CentOS 8 and later
 %if 0%{?centos} >= 9
-install -m 0644 -vp repositories/centos-*                       %{buildroot}%{_datadir}/osbuild-composer/repositories/
+install -m 0644 -vp repositories/centos-*                          %{buildroot}%{_datadir}/osbuild-composer/repositories/
 %else
 # CentOS 8 only supports building for CentOS 8
-install -m 0644 -vp repositories/centos-%{centos}*              %{buildroot}%{_datadir}/osbuild-composer/repositories/
-install -m 0644 -vp repositories/centos-stream-%{centos}*       %{buildroot}%{_datadir}/osbuild-composer/repositories/
+install -m 0644 -vp repositories/centos-%{centos}*                 %{buildroot}%{_datadir}/osbuild-composer/repositories/
+install -m 0644 -vp repositories/centos-stream-%{centos}*          %{buildroot}%{_datadir}/osbuild-composer/repositories/
 
 %endif
 %else
 %if 0%{?rhel}
 # RHEL 9 supports building for RHEL 8 and later
 %if 0%{?rhel} >= 9
-install -m 0644 -vp repositories/rhel-*                         %{buildroot}%{_datadir}/osbuild-composer/repositories/
+install -m 0644 -vp repositories/rhel-*                            %{buildroot}%{_datadir}/osbuild-composer/repositories/
 
 %else
 # RHEL 8 only supports building for 8
-install -m 0644 -vp repositories/rhel-%{rhel}*                  %{buildroot}%{_datadir}/osbuild-composer/repositories/
+install -m 0644 -vp repositories/rhel-%{rhel}*                     %{buildroot}%{_datadir}/osbuild-composer/repositories/
 
 %endif
 %endif
@@ -202,84 +202,85 @@ install -m 0644 -vp repositories/rhel-%{rhel}*                  %{buildroot}%{_d
 
 # Fedora can build for all included fedora releases
 %if 0%{?fedora}
-install -m 0644 -vp repositories/fedora-*                       %{buildroot}%{_datadir}/osbuild-composer/repositories/
+install -m 0644 -vp repositories/fedora-*                          %{buildroot}%{_datadir}/osbuild-composer/repositories/
 %endif
 
-install -m 0755 -vd                                             %{buildroot}%{_unitdir}
-install -m 0644 -vp distribution/*.{service,socket}             %{buildroot}%{_unitdir}/
+install -m 0755 -vd                                                %{buildroot}%{_unitdir}
+install -m 0644 -vp distribution/*.{service,socket}                %{buildroot}%{_unitdir}/
 
-install -m 0755 -vd                                             %{buildroot}%{_sysusersdir}
-install -m 0644 -vp distribution/osbuild-composer.conf          %{buildroot}%{_sysusersdir}/
+install -m 0755 -vd                                                %{buildroot}%{_sysusersdir}
+install -m 0644 -vp distribution/osbuild-composer.conf             %{buildroot}%{_sysusersdir}/
 
-install -m 0755 -vd                                             %{buildroot}%{_localstatedir}/cache/osbuild-composer/dnf-cache
+install -m 0755 -vd                                                %{buildroot}%{_localstatedir}/cache/osbuild-composer/dnf-cache
 
-install -m 0755 -vd                                             %{buildroot}%{_mandir}/man7
-install -m 0644 -vp docs/*.7                                    %{buildroot}%{_mandir}/man7/
+install -m 0755 -vd                                                %{buildroot}%{_mandir}/man7
+install -m 0644 -vp docs/*.7                                       %{buildroot}%{_mandir}/man7/
 
 %if %{with tests} || 0%{?rhel}
 
-install -m 0755 -vd                                             %{buildroot}%{_libexecdir}/osbuild-composer-test
-install -m 0755 -vp _bin/osbuild-composer-cli-tests             %{buildroot}%{_libexecdir}/osbuild-composer-test/
-install -m 0755 -vp _bin/osbuild-weldr-tests                    %{buildroot}%{_libexecdir}/osbuild-composer-test/
-install -m 0755 -vp _bin/osbuild-dnf-json-tests                 %{buildroot}%{_libexecdir}/osbuild-composer-test/
-install -m 0755 -vp _bin/osbuild-image-tests                    %{buildroot}%{_libexecdir}/osbuild-composer-test/
-install -m 0755 -vp _bin/osbuild-auth-tests                     %{buildroot}%{_libexecdir}/osbuild-composer-test/
-install -m 0755 -vp _bin/osbuild-koji-tests                     %{buildroot}%{_libexecdir}/osbuild-composer-test/
-install -m 0755 -vp _bin/osbuild-composer-dbjobqueue-tests      %{buildroot}%{_libexecdir}/osbuild-composer-test/
-install -m 0755 -vp _bin/osbuild-composer-manifest-tests        %{buildroot}%{_libexecdir}/osbuild-composer-test/
-install -m 0755 -vp _bin/cloud-cleaner                          %{buildroot}%{_libexecdir}/osbuild-composer-test/
-install -m 0755 -vp _bin/osbuild-mock-openid-provider           %{buildroot}%{_libexecdir}/osbuild-composer-test/
-install -m 0755 -vp tools/define-compose-url.sh                 %{buildroot}%{_libexecdir}/osbuild-composer-test/
-install -m 0755 -vp tools/provision.sh                          %{buildroot}%{_libexecdir}/osbuild-composer-test/
-install -m 0755 -vp tools/gen-certs.sh                          %{buildroot}%{_libexecdir}/osbuild-composer-test/
-install -m 0755 -vp tools/gen-ssh.sh                            %{buildroot}%{_libexecdir}/osbuild-composer-test/
-install -m 0755 -vp tools/image-info                            %{buildroot}%{_libexecdir}/osbuild-composer-test/
-install -m 0755 -vp tools/run-koji-container.sh                 %{buildroot}%{_libexecdir}/osbuild-composer-test/
-install -m 0755 -vp tools/koji-compose.py                       %{buildroot}%{_libexecdir}/osbuild-composer-test/
-install -m 0755 -vp tools/libvirt_test.sh                       %{buildroot}%{_libexecdir}/osbuild-composer-test/
-install -m 0755 -vp tools/set-env-variables.sh                  %{buildroot}%{_libexecdir}/osbuild-composer-test/
-install -m 0755 -vd                                             %{buildroot}%{_libexecdir}/tests/osbuild-composer
-install -m 0755 -vp test/cases/*                                %{buildroot}%{_libexecdir}/tests/osbuild-composer/
-
-install -m 0755 -vd                                             %{buildroot}%{_datadir}/tests/osbuild-composer/ansible
-install -m 0644 -vp test/data/ansible/*                         %{buildroot}%{_datadir}/tests/osbuild-composer/ansible/
-
-install -m 0755 -vd                                             %{buildroot}%{_datadir}/tests/osbuild-composer/azure
-install -m 0644 -vp test/data/azure/*                           %{buildroot}%{_datadir}/tests/osbuild-composer/azure/
-
-install -m 0755 -vd                                             %{buildroot}%{_datadir}/tests/osbuild-composer/manifests
-install -m 0644 -vp test/data/manifests/*                       %{buildroot}%{_datadir}/tests/osbuild-composer/manifests/
-
-install -m 0755 -vd                                             %{buildroot}%{_datadir}/tests/osbuild-composer/cloud-init
-install -m 0644 -vp test/data/cloud-init/*                      %{buildroot}%{_datadir}/tests/osbuild-composer/cloud-init/
-
-install -m 0755 -vd                                             %{buildroot}%{_datadir}/tests/osbuild-composer/composer
-install -m 0644 -vp test/data/composer/*                        %{buildroot}%{_datadir}/tests/osbuild-composer/composer/
-
-install -m 0755 -vd                                             %{buildroot}%{_datadir}/tests/osbuild-composer/worker
-install -m 0644 -vp test/data/worker/*                          %{buildroot}%{_datadir}/tests/osbuild-composer/worker/
-
-install -m 0755 -vd                                             %{buildroot}%{_datadir}/tests/osbuild-composer/repositories
-install -m 0644 -vp test/data/repositories/*                    %{buildroot}%{_datadir}/tests/osbuild-composer/repositories/
-
-install -m 0755 -vd                                             %{buildroot}%{_datadir}/tests/osbuild-composer/kerberos
-install -m 0644 -vp test/data/kerberos/*                        %{buildroot}%{_datadir}/tests/osbuild-composer/kerberos/
-
-install -m 0755 -vd                                             %{buildroot}%{_datadir}/tests/osbuild-composer/keyring
-install -m 0644 -vp test/data/keyring/id_rsa.pub                %{buildroot}%{_datadir}/tests/osbuild-composer/keyring/
-install -m 0600 -vp test/data/keyring/id_rsa                    %{buildroot}%{_datadir}/tests/osbuild-composer/keyring/
-
-install -m 0755 -vd                                             %{buildroot}%{_datadir}/tests/osbuild-composer/koji
-install -m 0644 -vp test/data/koji/*                            %{buildroot}%{_datadir}/tests/osbuild-composer/koji/
-
-install -m 0755 -vd                                             %{buildroot}%{_datadir}/tests/osbuild-composer/x509
-install -m 0644 -vp test/data/x509/*                            %{buildroot}%{_datadir}/tests/osbuild-composer/x509/
-
-install -m 0755 -vd                                             %{buildroot}%{_datadir}/tests/osbuild-composer/openshift
-install -m 0644 -vp test/data/openshift/*                       %{buildroot}%{_datadir}/tests/osbuild-composer/openshift/
-
-install -m 0755 -vd                                             %{buildroot}%{_datadir}/tests/osbuild-composer/schemas
-install -m 0644 -vp internal/jobqueue/dbjobqueue/schemas/*      %{buildroot}%{_datadir}/tests/osbuild-composer/schemas/
+install -m 0755 -vd                                                %{buildroot}%{_libexecdir}/osbuild-composer-test
+install -m 0755 -vp _bin/osbuild-composer-cli-tests                %{buildroot}%{_libexecdir}/osbuild-composer-test/
+install -m 0755 -vp _bin/osbuild-weldr-tests                       %{buildroot}%{_libexecdir}/osbuild-composer-test/
+install -m 0755 -vp _bin/osbuild-dnf-json-tests                    %{buildroot}%{_libexecdir}/osbuild-composer-test/
+install -m 0755 -vp _bin/osbuild-image-tests                       %{buildroot}%{_libexecdir}/osbuild-composer-test/
+install -m 0755 -vp _bin/osbuild-auth-tests                        %{buildroot}%{_libexecdir}/osbuild-composer-test/
+install -m 0755 -vp _bin/osbuild-koji-tests                        %{buildroot}%{_libexecdir}/osbuild-composer-test/
+install -m 0755 -vp _bin/osbuild-composer-dbjobqueue-tests         %{buildroot}%{_libexecdir}/osbuild-composer-test/
+install -m 0755 -vp _bin/osbuild-composer-manifest-tests           %{buildroot}%{_libexecdir}/osbuild-composer-test/
+install -m 0755 -vp _bin/cloud-cleaner                             %{buildroot}%{_libexecdir}/osbuild-composer-test/
+install -m 0755 -vp _bin/osbuild-mock-openid-provider              %{buildroot}%{_libexecdir}/osbuild-composer-test/
+install -m 0755 -vp tools/define-compose-url.sh                    %{buildroot}%{_libexecdir}/osbuild-composer-test/
+install -m 0755 -vp tools/provision.sh                             %{buildroot}%{_libexecdir}/osbuild-composer-test/
+install -m 0755 -vp tools/gen-certs.sh                             %{buildroot}%{_libexecdir}/osbuild-composer-test/
+install -m 0755 -vp tools/gen-ssh.sh                               %{buildroot}%{_libexecdir}/osbuild-composer-test/
+install -m 0755 -vp tools/image-info                               %{buildroot}%{_libexecdir}/osbuild-composer-test/
+install -m 0755 -vp tools/run-koji-container.sh                    %{buildroot}%{_libexecdir}/osbuild-composer-test/
+install -m 0755 -vp tools/koji-compose.py                          %{buildroot}%{_libexecdir}/osbuild-composer-test/
+install -m 0755 -vp tools/libvirt_test.sh                          %{buildroot}%{_libexecdir}/osbuild-composer-test/
+install -m 0755 -vp tools/set-env-variables.sh                     %{buildroot}%{_libexecdir}/osbuild-composer-test/
+install -m 0755 -vp tools/test-case-generators/generate-test-cases %{buildroot}%{_libexecdir}/osbuild-composer-test/
+install -m 0755 -vd                                                %{buildroot}%{_libexecdir}/tests/osbuild-composer
+install -m 0755 -vp test/cases/*                                   %{buildroot}%{_libexecdir}/tests/osbuild-composer/
+
+install -m 0755 -vd                                                %{buildroot}%{_datadir}/tests/osbuild-composer/ansible
+install -m 0644 -vp test/data/ansible/*                            %{buildroot}%{_datadir}/tests/osbuild-composer/ansible/
+
+install -m 0755 -vd                                                %{buildroot}%{_datadir}/tests/osbuild-composer/azure
+install -m 0644 -vp test/data/azure/*                              %{buildroot}%{_datadir}/tests/osbuild-composer/azure/
+
+install -m 0755 -vd                                                %{buildroot}%{_datadir}/tests/osbuild-composer/manifests
+install -m 0644 -vp test/data/manifests/*                          %{buildroot}%{_datadir}/tests/osbuild-composer/manifests/
+
+install -m 0755 -vd                                                %{buildroot}%{_datadir}/tests/osbuild-composer/cloud-init
+install -m 0644 -vp test/data/cloud-init/*                         %{buildroot}%{_datadir}/tests/osbuild-composer/cloud-init/
+
+install -m 0755 -vd                                                %{buildroot}%{_datadir}/tests/osbuild-composer/composer
+install -m 0644 -vp test/data/composer/*                           %{buildroot}%{_datadir}/tests/osbuild-composer/composer/
+
+install -m 0755 -vd                                                %{buildroot}%{_datadir}/tests/osbuild-composer/worker
+install -m 0644 -vp test/data/worker/*                             %{buildroot}%{_datadir}/tests/osbuild-composer/worker/
+
+install -m 0755 -vd                                                %{buildroot}%{_datadir}/tests/osbuild-composer/repositories
+install -m 0644 -vp test/data/repositories/*                       %{buildroot}%{_datadir}/tests/osbuild-composer/repositories/
+
+install -m 0755 -vd                                                %{buildroot}%{_datadir}/tests/osbuild-composer/kerberos
+install -m 0644 -vp test/data/kerberos/*                           %{buildroot}%{_datadir}/tests/osbuild-composer/kerberos/
+
+install -m 0755 -vd                                                %{buildroot}%{_datadir}/tests/osbuild-composer/keyring
+install -m 0644 -vp test/data/keyring/id_rsa.pub                   %{buildroot}%{_datadir}/tests/osbuild-composer/keyring/
+install -m 0600 -vp test/data/keyring/id_rsa                       %{buildroot}%{_datadir}/tests/osbuild-composer/keyring/
+
+install -m 0755 -vd                                                %{buildroot}%{_datadir}/tests/osbuild-composer/koji
+install -m 0644 -vp test/data/koji/*                               %{buildroot}%{_datadir}/tests/osbuild-composer/koji/
+
+install -m 0755 -vd                                                %{buildroot}%{_datadir}/tests/osbuild-composer/x509
+install -m 0644 -vp test/data/x509/*                               %{buildroot}%{_datadir}/tests/osbuild-composer/x509/
+
+install -m 0755 -vd                                                %{buildroot}%{_datadir}/tests/osbuild-composer/openshift
+install -m 0644 -vp test/data/openshift/*                          %{buildroot}%{_datadir}/tests/osbuild-composer/openshift/
+
+install -m 0755 -vd                                                %{buildroot}%{_datadir}/tests/osbuild-composer/schemas
+install -m 0644 -vp internal/jobqueue/dbjobqueue/schemas/*         %{buildroot}%{_datadir}/tests/osbuild-composer/schemas/
 
 %endif
 
@@ -375,6 +376,8 @@ The dnf-json binary used by osbuild-composer and the workers.
 
 %files dnf-json
 %{_libexecdir}/osbuild-composer/dnf-json
+%{_unitdir}/osbuild-dnf-json.service
+%{_unitdir}/osbuild-dnf-json.socket
 
 %if %{with tests} || 0%{?rhel}
 
diff --git a/schutzbot/start_iostats.sh b/schutzbot/start_iostats.sh
new file mode 100755
index 000000000..35206c9da
--- /dev/null
+++ b/schutzbot/start_iostats.sh
@@ -0,0 +1,4 @@
+#!/bin/bash
+sudo dnf install -y sysstat
+iostat -y -x -o JSON 5 > iostats.json &
+echo "PERFLOG $(date --rfc-3339=seconds) starting iostat"
diff --git a/schutzbot/stop_iostats.sh b/schutzbot/stop_iostats.sh
new file mode 100755
index 000000000..264f598d7
--- /dev/null
+++ b/schutzbot/stop_iostats.sh
@@ -0,0 +1,5 @@
+#!/bin/bash
+killall -s SIGINT iostat
+sleep 10
+killall iostat || true
+echo "PERFLOG $(date --rfc-3339=seconds) stopping iostat"
diff --git a/templates/composer.yml b/templates/composer.yml
index 295a81692..718936af8 100644
--- a/templates/composer.yml
+++ b/templates/composer.yml
@@ -214,6 +214,7 @@ objects:
       [worker]
       request_job_timeout = "20s"
       base_path = "/api/image-builder-worker/v1"
+      enable_artifacts = false
       enable_tls = false
       enable_mtls = false
       enable_jwt = true
diff --git a/templates/dashboards/grafana-dashboard-image-builder-worker-general.configmap.yml b/templates/dashboards/grafana-dashboard-image-builder-worker-general.configmap.yml
new file mode 100644
index 000000000..23274d555
--- /dev/null
+++ b/templates/dashboards/grafana-dashboard-image-builder-worker-general.configmap.yml
@@ -0,0 +1,1776 @@
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: grafana-dashboard-image-builder-worker-general
+  labels:
+    grafana_dashboard: "true"
+  annotations:
+    grafana-folder: /grafana-dashboard-definitions/Image-Builder
+data:
+  grafana.json: |-
+    {
+      "annotations": {
+        "list": [
+          {
+            "builtIn": 1,
+            "datasource": "-- Grafana --",
+            "enable": true,
+            "hide": true,
+            "iconColor": "rgba(0, 211, 255, 1)",
+            "name": "Annotations & Alerts",
+            "target": {
+              "limit": 100,
+              "matchAny": false,
+              "tags": [],
+              "type": "dashboard"
+            },
+            "type": "dashboard"
+          }
+        ]
+      },
+      "editable": true,
+      "gnetId": null,
+      "graphTooltip": 0,
+      "id": 3,
+      "iteration": 1639488117492,
+      "links": [],
+      "panels": [
+        {
+          "collapsed": false,
+          "datasource": null,
+          "gridPos": {
+            "h": 1,
+            "w": 24,
+            "x": 0,
+            "y": 0
+          },
+          "id": 128,
+          "panels": [],
+          "title": "Depsolve Duration",
+          "type": "row"
+        },
+        {
+          "datasource": "${datasource}",
+          "description": "The duration of 90% of depsolve jobs",
+          "fieldConfig": {
+            "defaults": {
+              "color": {
+                "mode": "thresholds"
+              },
+              "decimals": 0,
+              "mappings": [
+                {
+                  "options": {
+                    "match": "null",
+                    "result": {
+                      "index": 0,
+                      "text": "N/A"
+                    }
+                  },
+                  "type": "special"
+                }
+              ],
+              "thresholds": {
+                "mode": "absolute",
+                "steps": [
+                  {
+                    "color": "green",
+                    "value": null
+                  },
+                  {
+                    "color": "#EAB839",
+                    "value": "25"
+                  },
+                  {
+                    "color": "red",
+                    "value": "30"
+                  }
+                ]
+              },
+              "unit": "s"
+            },
+            "overrides": []
+          },
+          "gridPos": {
+            "h": 8,
+            "w": 5,
+            "x": 0,
+            "y": 1
+          },
+          "id": 197,
+          "mappings": [
+            {
+              "options": {
+                "match": "null",
+                "result": {
+                  "text": "N/A"
+                }
+              },
+              "type": "special"
+            }
+          ],
+          "options": {
+            "colorMode": "value",
+            "graphMode": "none",
+            "justifyMode": "auto",
+            "orientation": "auto",
+            "reduceOptions": {
+              "calcs": [
+                "lastNotNull"
+              ],
+              "fields": "",
+              "values": false
+            },
+            "text": {},
+            "textMode": "auto"
+          },
+          "pluginVersion": "8.1.5",
+          "targets": [
+            {
+              "exemplar": true,
+              "expr": "histogram_quantile(0.9, sum(rate(image_builder_composer_worker_job_duration_seconds_bucket{type=\"depsolve\"}[$__range])) by (le))",
+              "interval": "",
+              "legendFormat": "",
+              "refId": "A"
+            }
+          ],
+          "title": "Depsolve Job Duration",
+          "type": "stat"
+        },
+        {
+          "datasource": "${datasource}",
+          "description": "The request Duration for depsolve jobs over the selected date range",
+          "fieldConfig": {
+            "defaults": {
+              "color": {
+                "mode": "thresholds"
+              },
+              "custom": {
+                "axisLabel": "seconds",
+                "axisPlacement": "auto",
+                "barAlignment": 0,
+                "drawStyle": "line",
+                "fillOpacity": 35,
+                "gradientMode": "scheme",
+                "hideFrom": {
+                  "legend": false,
+                  "tooltip": false,
+                  "viz": false
+                },
+                "lineInterpolation": "linear",
+                "lineWidth": 3,
+                "pointSize": 5,
+                "scaleDistribution": {
+                  "type": "linear"
+                },
+                "showPoints": "never",
+                "spanNulls": true,
+                "stacking": {
+                  "group": "A",
+                  "mode": "none"
+                },
+                "thresholdsStyle": {
+                  "mode": "off"
+                }
+              },
+              "mappings": [],
+              "min": 0,
+              "thresholds": {
+                "mode": "absolute",
+                "steps": [
+                  {
+                    "color": "green",
+                    "value": null
+                  },
+                  {
+                    "color": "#EAB839",
+                    "value": "175"
+                  },
+                  {
+                    "color": "red",
+                    "value": "200"
+                  }
+                ]
+              },
+              "unit": "s"
+            },
+            "overrides": [
+              {
+                "matcher": {
+                  "id": "byName",
+                  "options": "99p"
+                },
+                "properties": [
+                  {
+                    "id": "color",
+                    "value": {
+                      "fixedColor": "red",
+                      "mode": "fixed"
+                    }
+                  }
+                ]
+              },
+              {
+                "matcher": {
+                  "id": "byName",
+                  "options": "95p"
+                },
+                "properties": [
+                  {
+                    "id": "color",
+                    "value": {
+                      "fixedColor": "blue",
+                      "mode": "fixed"
+                    }
+                  }
+                ]
+              },
+              {
+                "matcher": {
+                  "id": "byName",
+                  "options": "90p"
+                },
+                "properties": [
+                  {
+                    "id": "color",
+                    "value": {
+                      "fixedColor": "yellow",
+                      "mode": "fixed"
+                    }
+                  }
+                ]
+              },
+              {
+                "matcher": {
+                  "id": "byName",
+                  "options": "50p"
+                },
+                "properties": [
+                  {
+                    "id": "color",
+                    "value": {
+                      "fixedColor": "blue",
+                      "mode": "fixed"
+                    }
+                  }
+                ]
+              },
+              {
+                "matcher": {
+                  "id": "byName",
+                  "options": "p90"
+                },
+                "properties": [
+                  {
+                    "id": "color",
+                    "value": {
+                      "fixedColor": "yellow",
+                      "mode": "fixed"
+                    }
+                  }
+                ]
+              },
+              {
+                "matcher": {
+                  "id": "byName",
+                  "options": "p99"
+                },
+                "properties": [
+                  {
+                    "id": "color",
+                    "value": {
+                      "fixedColor": "red",
+                      "mode": "fixed"
+                    }
+                  }
+                ]
+              },
+              {
+                "matcher": {
+                  "id": "byName",
+                  "options": "p50"
+                },
+                "properties": [
+                  {
+                    "id": "color",
+                    "value": {
+                      "fixedColor": "blue",
+                      "mode": "fixed"
+                    }
+                  }
+                ]
+              }
+            ]
+          },
+          "gridPos": {
+            "h": 8,
+            "w": 11,
+            "x": 5,
+            "y": 1
+          },
+          "id": 205,
+          "options": {
+            "legend": {
+              "calcs": [],
+              "displayMode": "list",
+              "placement": "bottom"
+            },
+            "tooltip": {
+              "mode": "single"
+            }
+          },
+          "targets": [
+            {
+              "exemplar": true,
+              "expr": "histogram_quantile(0.5, sum(rate(image_builder_composer_worker_job_duration_seconds_bucket{type=\"depsolve\"}[$interval])) by (le))",
+              "interval": "",
+              "legendFormat": "p50",
+              "refId": "A"
+            },
+            {
+              "exemplar": true,
+              "expr": "histogram_quantile(0.9, sum(rate(image_builder_composer_worker_job_duration_seconds_bucket{type=\"depsolve\"}[$interval])) by (le))",
+              "hide": false,
+              "interval": "",
+              "legendFormat": "p90",
+              "refId": "B"
+            },
+            {
+              "exemplar": true,
+              "expr": "histogram_quantile(0.99, sum(rate(image_builder_composer_worker_job_duration_seconds_bucket{type=\"depsolve\"}[$interval])) by (le))",
+              "hide": false,
+              "interval": "",
+              "legendFormat": "p99",
+              "refId": "C"
+            }
+          ],
+          "title": "Depsolve Job Duration",
+          "type": "timeseries"
+        },
+        {
+          "datasource": "${datasource}",
+          "description": "Percent of requests exceeding Duration allowed by SLO",
+          "fieldConfig": {
+            "defaults": {
+              "color": {
+                "mode": "thresholds"
+              },
+              "custom": {
+                "axisLabel": "",
+                "axisPlacement": "auto",
+                "barAlignment": 0,
+                "drawStyle": "bars",
+                "fillOpacity": 0,
+                "gradientMode": "none",
+                "hideFrom": {
+                  "legend": false,
+                  "tooltip": false,
+                  "viz": false
+                },
+                "lineInterpolation": "linear",
+                "lineWidth": 1,
+                "pointSize": 5,
+                "scaleDistribution": {
+                  "type": "linear"
+                },
+                "showPoints": "auto",
+                "spanNulls": false,
+                "stacking": {
+                  "group": "A",
+                  "mode": "none"
+                },
+                "thresholdsStyle": {
+                  "mode": "off"
+                }
+              },
+              "mappings": [],
+              "max": 1,
+              "thresholds": {
+                "mode": "absolute",
+                "steps": [
+                  {
+                    "color": "green",
+                    "value": null
+                  },
+                  {
+                    "color": "red",
+                    "value": 80
+                  }
+                ]
+              },
+              "unit": "percentunit"
+            },
+            "overrides": []
+          },
+          "gridPos": {
+            "h": 8,
+            "w": 8,
+            "x": 16,
+            "y": 1
+          },
+          "id": 194,
+          "options": {
+            "legend": {
+              "calcs": [],
+              "displayMode": "hidden",
+              "placement": "bottom"
+            },
+            "tooltip": {
+              "mode": "single"
+            }
+          },
+          "targets": [
+            {
+              "exemplar": true,
+              "expr": "1 - sum(rate(image_builder_composer_worker_job_duration_seconds_bucket{le=\"32\", type=\"depsolve\"}[$interval]))/sum(rate(image_builder_composer_worker_job_duration_seconds_count{type=\"depsolve\"}[$interval]))",
+              "interval": "",
+              "legendFormat": "",
+              "refId": "A"
+            }
+          ],
+          "title": "Slow Request Rate",
+          "type": "timeseries"
+        },
+        {
+          "cacheTimeout": 1,
+          "datasource": "${datasource}",
+          "description": "How long will it take to consume all our budget if our error consumption remains at the current rate for the selected date range.",
+          "fieldConfig": {
+            "defaults": {
+              "color": {
+                "mode": "thresholds"
+              },
+              "decimals": 2,
+              "mappings": [
+                {
+                  "options": {
+                    "match": "null",
+                    "result": {
+                      "index": 0,
+                      "text": "1.40 days"
+                    }
+                  },
+                  "type": "special"
+                },
+                {
+                  "options": {
+                    "from": 672,
+                    "result": {
+                      "index": 1,
+                      "text": "∞"
+                    },
+                    "to": 3360100
+                  },
+                  "type": "range"
+                }
+              ],
+              "thresholds": {
+                "mode": "absolute",
+                "steps": [
+                  {
+                    "color": "red",
+                    "value": null
+                  },
+                  {
+                    "color": "#EAB839",
+                    "value": 40
+                  },
+                  {
+                    "color": "green",
+                    "value": 50
+                  }
+                ]
+              },
+              "unit": "h"
+            },
+            "overrides": []
+          },
+          "gridPos": {
+            "h": 8,
+            "w": 4,
+            "x": 0,
+            "y": 9
+          },
+          "id": 115,
+          "interval": null,
+          "links": [],
+          "maxDataPoints": 100,
+          "options": {
+            "colorMode": "value",
+            "graphMode": "none",
+            "justifyMode": "auto",
+            "orientation": "horizontal",
+            "reduceOptions": {
+              "calcs": [
+                "lastNotNull"
+              ],
+              "fields": "",
+              "values": false
+            },
+            "text": {
+              "valueSize": 80
+            },
+            "textMode": "auto"
+          },
+          "pluginVersion": "8.1.5",
+          "targets": [
+            {
+              "exemplar": true,
+              "expr": "28 * 24 * (1 - $latency_slo) / (1 - sum(rate(image_builder_composer_worker_job_duration_seconds_bucket{le=\"32\",type=\"depsolve\"}[$__range]))/sum(rate(image_builder_composer_worker_job_duration_seconds_count{type=\"depsolve\"}[$__range])))",
+              "format": "time_series",
+              "interval": "",
+              "intervalFactor": 1,
+              "legendFormat": "",
+              "refId": "A"
+            }
+          ],
+          "timeFrom": null,
+          "timeShift": null,
+          "title": "Error Budget Remaining",
+          "type": "stat"
+        },
+        {
+          "cacheTimeout": null,
+          "datasource": "${datasource}",
+          "description": "The percentage of error budget consumed for the selected time range. ",
+          "fieldConfig": {
+            "defaults": {
+              "color": {
+                "mode": "thresholds"
+              },
+              "custom": {
+                "axisLabel": "",
+                "axisPlacement": "auto",
+                "barAlignment": 0,
+                "drawStyle": "line",
+                "fillOpacity": 100,
+                "gradientMode": "scheme",
+                "hideFrom": {
+                  "legend": false,
+                  "tooltip": false,
+                  "viz": false
+                },
+                "lineInterpolation": "linear",
+                "lineStyle": {
+                  "fill": "solid"
+                },
+                "lineWidth": 0,
+                "pointSize": 5,
+                "scaleDistribution": {
+                  "type": "linear"
+                },
+                "showPoints": "never",
+                "spanNulls": true,
+                "stacking": {
+                  "group": "A",
+                  "mode": "none"
+                },
+                "thresholdsStyle": {
+                  "mode": "off"
+                }
+              },
+              "links": [],
+              "mappings": [],
+              "max": 1,
+              "thresholds": {
+                "mode": "absolute",
+                "steps": [
+                  {
+                    "color": "green",
+                    "value": null
+                  },
+                  {
+                    "color": "#EAB839",
+                    "value": 0.95
+                  },
+                  {
+                    "color": "red",
+                    "value": 1
+                  }
+                ]
+              },
+              "unit": "percentunit"
+            },
+            "overrides": []
+          },
+          "gridPos": {
+            "h": 8,
+            "w": 20,
+            "x": 4,
+            "y": 9
+          },
+          "id": 119,
+          "links": [],
+          "options": {
+            "legend": {
+              "calcs": [],
+              "displayMode": "hidden",
+              "placement": "bottom"
+            },
+            "tooltip": {
+              "mode": "single"
+            }
+          },
+          "pluginVersion": "8.1.5",
+          "targets": [
+            {
+              "exemplar": true,
+              "expr": "1 - ((sum(increase(image_builder_composer_worker_job_duration_seconds_bucket{le=\"32\", type=\"depsolve\"}[28d]))/sum(increase(image_builder_composer_worker_job_duration_seconds_count{type=\"depsolve\"}[28d]))) - $latency_slo)/ (1 - $latency_slo)",
+              "instant": false,
+              "interval": "",
+              "intervalFactor": 10,
+              "legendFormat": "errorbudget",
+              "refId": "A"
+            }
+          ],
+          "timeFrom": null,
+          "timeShift": null,
+          "title": "Error Budget Consumed",
+          "type": "timeseries"
+        },
+        {
+          "collapsed": false,
+          "datasource": null,
+          "gridPos": {
+            "h": 1,
+            "w": 24,
+            "x": 0,
+            "y": 17
+          },
+          "id": 129,
+          "panels": [],
+          "title": "Osbuild Job Duration",
+          "type": "row"
+        },
+        {
+          "datasource": "${datasource}",
+          "description": "The duration of 90% of osbuild jobs",
+          "fieldConfig": {
+            "defaults": {
+              "color": {
+                "mode": "thresholds"
+              },
+              "decimals": 0,
+              "mappings": [
+                {
+                  "options": {
+                    "match": "null",
+                    "result": {
+                      "index": 0,
+                      "text": "N/A"
+                    }
+                  },
+                  "type": "special"
+                }
+              ],
+              "thresholds": {
+                "mode": "absolute",
+                "steps": [
+                  {
+                    "color": "green",
+                    "value": null
+                  },
+                  {
+                    "color": "#EAB839",
+                    "value": "1228"
+                  },
+                  {
+                    "color": "red",
+                    "value": "1536"
+                  }
+                ]
+              },
+              "unit": "s"
+            },
+            "overrides": []
+          },
+          "gridPos": {
+            "h": 8,
+            "w": 5,
+            "x": 0,
+            "y": 18
+          },
+          "id": 200,
+          "mappings": [
+            {
+              "options": {
+                "match": "null",
+                "result": {
+                  "text": "N/A"
+                }
+              },
+              "type": "special"
+            }
+          ],
+          "options": {
+            "colorMode": "value",
+            "graphMode": "none",
+            "justifyMode": "auto",
+            "orientation": "auto",
+            "reduceOptions": {
+              "calcs": [
+                "lastNotNull"
+              ],
+              "fields": "",
+              "values": false
+            },
+            "text": {},
+            "textMode": "auto"
+          },
+          "pluginVersion": "8.1.5",
+          "targets": [
+            {
+              "exemplar": true,
+              "expr": "histogram_quantile(0.9, sum(rate(image_builder_composer_worker_job_duration_seconds_bucket{type=~\"osbuild:.*\"}[$__range])) by (le))",
+              "interval": "",
+              "legendFormat": "",
+              "refId": "A"
+            }
+          ],
+          "title": "Osbuild Job Duration",
+          "type": "stat"
+        },
+        {
+          "datasource": "${datasource}",
+          "description": "The request Duration for osbuild jobs over the selected date range",
+          "fieldConfig": {
+            "defaults": {
+              "color": {
+                "mode": "thresholds"
+              },
+              "custom": {
+                "axisLabel": "seconds",
+                "axisPlacement": "auto",
+                "barAlignment": 0,
+                "drawStyle": "line",
+                "fillOpacity": 35,
+                "gradientMode": "scheme",
+                "hideFrom": {
+                  "legend": false,
+                  "tooltip": false,
+                  "viz": false
+                },
+                "lineInterpolation": "linear",
+                "lineWidth": 3,
+                "pointSize": 5,
+                "scaleDistribution": {
+                  "type": "linear"
+                },
+                "showPoints": "never",
+                "spanNulls": true,
+                "stacking": {
+                  "group": "A",
+                  "mode": "none"
+                },
+                "thresholdsStyle": {
+                  "mode": "off"
+                }
+              },
+              "mappings": [],
+              "min": 0,
+              "thresholds": {
+                "mode": "absolute",
+                "steps": [
+                  {
+                    "color": "green",
+                    "value": null
+                  },
+                  {
+                    "color": "#EAB839",
+                    "value": "175"
+                  },
+                  {
+                    "color": "red",
+                    "value": "200"
+                  }
+                ]
+              },
+              "unit": "s"
+            },
+            "overrides": [
+              {
+                "matcher": {
+                  "id": "byName",
+                  "options": "99p"
+                },
+                "properties": [
+                  {
+                    "id": "color",
+                    "value": {
+                      "fixedColor": "red",
+                      "mode": "fixed"
+                    }
+                  }
+                ]
+              },
+              {
+                "matcher": {
+                  "id": "byName",
+                  "options": "95p"
+                },
+                "properties": [
+                  {
+                    "id": "color",
+                    "value": {
+                      "fixedColor": "blue",
+                      "mode": "fixed"
+                    }
+                  }
+                ]
+              },
+              {
+                "matcher": {
+                  "id": "byName",
+                  "options": "90p"
+                },
+                "properties": [
+                  {
+                    "id": "color",
+                    "value": {
+                      "fixedColor": "yellow",
+                      "mode": "fixed"
+                    }
+                  }
+                ]
+              },
+              {
+                "matcher": {
+                  "id": "byName",
+                  "options": "50p"
+                },
+                "properties": [
+                  {
+                    "id": "color",
+                    "value": {
+                      "fixedColor": "blue",
+                      "mode": "fixed"
+                    }
+                  }
+                ]
+              },
+              {
+                "matcher": {
+                  "id": "byName",
+                  "options": "p90"
+                },
+                "properties": [
+                  {
+                    "id": "color",
+                    "value": {
+                      "fixedColor": "yellow",
+                      "mode": "fixed"
+                    }
+                  }
+                ]
+              },
+              {
+                "matcher": {
+                  "id": "byName",
+                  "options": "p99"
+                },
+                "properties": [
+                  {
+                    "id": "color",
+                    "value": {
+                      "fixedColor": "red",
+                      "mode": "fixed"
+                    }
+                  }
+                ]
+              },
+              {
+                "matcher": {
+                  "id": "byName",
+                  "options": "p50"
+                },
+                "properties": [
+                  {
+                    "id": "color",
+                    "value": {
+                      "fixedColor": "blue",
+                      "mode": "fixed"
+                    }
+                  }
+                ]
+              }
+            ]
+          },
+          "gridPos": {
+            "h": 8,
+            "w": 11,
+            "x": 5,
+            "y": 18
+          },
+          "id": 201,
+          "options": {
+            "legend": {
+              "calcs": [],
+              "displayMode": "list",
+              "placement": "bottom"
+            },
+            "tooltip": {
+              "mode": "single"
+            }
+          },
+          "targets": [
+            {
+              "exemplar": true,
+              "expr": "histogram_quantile(0.5, sum(rate(image_builder_composer_worker_job_duration_seconds_bucket{type=~\"osbuild:.*\"}[$interval])) by (le))",
+              "interval": "",
+              "legendFormat": "p50",
+              "refId": "A"
+            },
+            {
+              "exemplar": true,
+              "expr": "histogram_quantile(0.9, sum(rate(image_builder_composer_worker_job_duration_seconds_bucket{type=~\"osbuild:.*\"}[$interval])) by (le))",
+              "hide": false,
+              "interval": "",
+              "legendFormat": "p90",
+              "refId": "B"
+            },
+            {
+              "exemplar": true,
+              "expr": "histogram_quantile(0.99, sum(rate(image_builder_composer_worker_job_duration_seconds_bucket{type=~\"osbuild:.*\"}[$interval])) by (le))",
+              "hide": false,
+              "interval": "",
+              "legendFormat": "p99",
+              "refId": "C"
+            }
+          ],
+          "title": "Osbuild Job Duration",
+          "type": "timeseries"
+        },
+        {
+          "datasource": "${datasource}",
+          "description": "Percent of requests exceeding Duration allowed by SLO",
+          "fieldConfig": {
+            "defaults": {
+              "color": {
+                "mode": "palette-classic"
+              },
+              "custom": {
+                "axisLabel": "",
+                "axisPlacement": "auto",
+                "barAlignment": 0,
+                "drawStyle": "line",
+                "fillOpacity": 0,
+                "gradientMode": "none",
+                "hideFrom": {
+                  "legend": false,
+                  "tooltip": false,
+                  "viz": false
+                },
+                "lineInterpolation": "linear",
+                "lineWidth": 1,
+                "pointSize": 5,
+                "scaleDistribution": {
+                  "type": "linear"
+                },
+                "showPoints": "auto",
+                "spanNulls": false,
+                "stacking": {
+                  "group": "A",
+                  "mode": "none"
+                },
+                "thresholdsStyle": {
+                  "mode": "off"
+                }
+              },
+              "mappings": [],
+              "max": 1,
+              "min": 0,
+              "thresholds": {
+                "mode": "absolute",
+                "steps": [
+                  {
+                    "color": "green",
+                    "value": null
+                  },
+                  {
+                    "color": "red",
+                    "value": 80
+                  }
+                ]
+              },
+              "unit": "percentunit"
+            },
+            "overrides": []
+          },
+          "gridPos": {
+            "h": 8,
+            "w": 8,
+            "x": 16,
+            "y": 18
+          },
+          "id": 210,
+          "options": {
+            "legend": {
+              "calcs": [],
+              "displayMode": "hidden",
+              "placement": "bottom"
+            },
+            "tooltip": {
+              "mode": "single"
+            }
+          },
+          "targets": [
+            {
+              "exemplar": true,
+              "expr": "1 - sum(rate(image_builder_composer_worker_job_duration_seconds_bucket{le=\"1536\",type=~\"osbuild:.*\"}[$interval]))/sum(rate(image_builder_composer_worker_job_duration_seconds_count{type=~\"osbuild:.*\"}[$interval]))",
+              "interval": "",
+              "legendFormat": "",
+              "refId": "A"
+            }
+          ],
+          "title": "Slow Request Rate",
+          "type": "timeseries"
+        },
+        {
+          "cacheTimeout": 1,
+          "datasource": "${datasource}",
+          "description": "How long will it take to consume all our budget if our error consumption remains at the current rate for the selected date range.",
+          "fieldConfig": {
+            "defaults": {
+              "color": {
+                "mode": "thresholds"
+              },
+              "decimals": 2,
+              "mappings": [
+                {
+                  "options": {
+                    "match": "null",
+                    "result": {
+                      "index": 0,
+                      "text": "1.40 days"
+                    }
+                  },
+                  "type": "special"
+                },
+                {
+                  "options": {
+                    "from": 672,
+                    "result": {
+                      "index": 1,
+                      "text": "∞"
+                    },
+                    "to": 3360100
+                  },
+                  "type": "range"
+                }
+              ],
+              "thresholds": {
+                "mode": "absolute",
+                "steps": [
+                  {
+                    "color": "red",
+                    "value": null
+                  },
+                  {
+                    "color": "#EAB839",
+                    "value": 40
+                  },
+                  {
+                    "color": "green",
+                    "value": 50
+                  }
+                ]
+              },
+              "unit": "h"
+            },
+            "overrides": []
+          },
+          "gridPos": {
+            "h": 8,
+            "w": 4,
+            "x": 0,
+            "y": 26
+          },
+          "id": 198,
+          "interval": null,
+          "links": [],
+          "maxDataPoints": 100,
+          "options": {
+            "colorMode": "value",
+            "graphMode": "none",
+            "justifyMode": "auto",
+            "orientation": "horizontal",
+            "reduceOptions": {
+              "calcs": [
+                "lastNotNull"
+              ],
+              "fields": "",
+              "values": false
+            },
+            "text": {
+              "valueSize": 80
+            },
+            "textMode": "auto"
+          },
+          "pluginVersion": "8.1.5",
+          "targets": [
+            {
+              "exemplar": true,
+              "expr": "28 * 24 * (1 - $latency_slo) / (1 - sum(rate(image_builder_composer_worker_job_duration_seconds_bucket{le=\"1536\",type=~\"osbuild:.*\"}[$__range]))/sum(rate(image_builder_composer_worker_job_duration_seconds_count{type=~\"osbuild:.*\"}[$__range])))",
+              "format": "time_series",
+              "interval": "",
+              "intervalFactor": 1,
+              "legendFormat": "",
+              "refId": "A"
+            }
+          ],
+          "timeFrom": null,
+          "timeShift": null,
+          "title": "Error Budget Remaining",
+          "type": "stat"
+        },
+        {
+          "cacheTimeout": null,
+          "datasource": "${datasource}",
+          "description": "The percentage of error budget consumed for the selected time range. ",
+          "fieldConfig": {
+            "defaults": {
+              "color": {
+                "mode": "thresholds"
+              },
+              "custom": {
+                "axisLabel": "",
+                "axisPlacement": "auto",
+                "barAlignment": 0,
+                "drawStyle": "line",
+                "fillOpacity": 100,
+                "gradientMode": "scheme",
+                "hideFrom": {
+                  "legend": false,
+                  "tooltip": false,
+                  "viz": false
+                },
+                "lineInterpolation": "linear",
+                "lineStyle": {
+                  "fill": "solid"
+                },
+                "lineWidth": 0,
+                "pointSize": 5,
+                "scaleDistribution": {
+                  "type": "linear"
+                },
+                "showPoints": "never",
+                "spanNulls": true,
+                "stacking": {
+                  "group": "A",
+                  "mode": "none"
+                },
+                "thresholdsStyle": {
+                  "mode": "off"
+                }
+              },
+              "links": [],
+              "mappings": [],
+              "thresholds": {
+                "mode": "absolute",
+                "steps": [
+                  {
+                    "color": "green",
+                    "value": null
+                  },
+                  {
+                    "color": "#EAB839",
+                    "value": 0.95
+                  },
+                  {
+                    "color": "red",
+                    "value": 1
+                  }
+                ]
+              },
+              "unit": "percentunit"
+            },
+            "overrides": []
+          },
+          "gridPos": {
+            "h": 8,
+            "w": 20,
+            "x": 4,
+            "y": 26
+          },
+          "id": 199,
+          "links": [],
+          "options": {
+            "legend": {
+              "calcs": [],
+              "displayMode": "hidden",
+              "placement": "bottom"
+            },
+            "tooltip": {
+              "mode": "single"
+            }
+          },
+          "pluginVersion": "8.1.5",
+          "targets": [
+            {
+              "exemplar": true,
+              "expr": "1 - ((sum(increase(image_builder_composer_worker_job_duration_seconds_bucket{le=\"1536\",type=~\"osbuild:.*\"}[28d]))/sum(increase(image_builder_composer_worker_job_duration_seconds_count{type=~\"osbuild:.*\"}[28d]))) - $latency_slo)/ (1 - $latency_slo)",
+              "instant": false,
+              "interval": "",
+              "intervalFactor": 10,
+              "legendFormat": "errorbudget",
+              "refId": "A"
+            }
+          ],
+          "timeFrom": null,
+          "timeShift": null,
+          "title": "Error Budget Consumed",
+          "type": "timeseries"
+        },
+        {
+          "collapsed": false,
+          "datasource": null,
+          "gridPos": {
+            "h": 1,
+            "w": 24,
+            "x": 0,
+            "y": 34
+          },
+          "id": 207,
+          "panels": [],
+          "title": "Job Wait Duration",
+          "type": "row"
+        },
+        {
+          "datasource": "${datasource}",
+          "description": "The duration of 90% of jobs waiting for execution in the job queue",
+          "fieldConfig": {
+            "defaults": {
+              "color": {
+                "mode": "thresholds"
+              },
+              "decimals": 0,
+              "mappings": [
+                {
+                  "options": {
+                    "match": "null",
+                    "result": {
+                      "index": 0,
+                      "text": "N/A"
+                    }
+                  },
+                  "type": "special"
+                }
+              ],
+              "thresholds": {
+                "mode": "absolute",
+                "steps": [
+                  {
+                    "color": "green",
+                    "value": null
+                  },
+                  {
+                    "color": "#EAB839",
+                    "value": "1228"
+                  },
+                  {
+                    "color": "red",
+                    "value": "1536"
+                  }
+                ]
+              },
+              "unit": "s"
+            },
+            "overrides": []
+          },
+          "gridPos": {
+            "h": 8,
+            "w": 5,
+            "x": 0,
+            "y": 35
+          },
+          "id": 208,
+          "mappings": [
+            {
+              "options": {
+                "match": "null",
+                "result": {
+                  "text": "N/A"
+                }
+              },
+              "type": "special"
+            }
+          ],
+          "options": {
+            "colorMode": "value",
+            "graphMode": "none",
+            "justifyMode": "auto",
+            "orientation": "auto",
+            "reduceOptions": {
+              "calcs": [
+                "lastNotNull"
+              ],
+              "fields": "",
+              "values": false
+            },
+            "text": {},
+            "textMode": "auto"
+          },
+          "pluginVersion": "8.1.5",
+          "targets": [
+            {
+              "exemplar": true,
+              "expr": "histogram_quantile(0.9, sum(rate(image_builder_composer_worker_job_wait_duration_seconds_bucket[$__range])) by (le))",
+              "interval": "",
+              "legendFormat": "",
+              "refId": "A"
+            }
+          ],
+          "title": "Job Wait Duration",
+          "type": "stat"
+        },
+        {
+          "datasource": "${datasource}",
+          "description": "The  duration for jobs waiting in the job queue over the selected date range",
+          "fieldConfig": {
+            "defaults": {
+              "color": {
+                "mode": "thresholds"
+              },
+              "custom": {
+                "axisLabel": "seconds",
+                "axisPlacement": "auto",
+                "barAlignment": 0,
+                "drawStyle": "line",
+                "fillOpacity": 35,
+                "gradientMode": "scheme",
+                "hideFrom": {
+                  "legend": false,
+                  "tooltip": false,
+                  "viz": false
+                },
+                "lineInterpolation": "linear",
+                "lineWidth": 3,
+                "pointSize": 5,
+                "scaleDistribution": {
+                  "type": "linear"
+                },
+                "showPoints": "never",
+                "spanNulls": true,
+                "stacking": {
+                  "group": "A",
+                  "mode": "none"
+                },
+                "thresholdsStyle": {
+                  "mode": "off"
+                }
+              },
+              "mappings": [],
+              "min": 0,
+              "thresholds": {
+                "mode": "absolute",
+                "steps": [
+                  {
+                    "color": "green",
+                    "value": null
+                  },
+                  {
+                    "color": "#EAB839",
+                    "value": "175"
+                  },
+                  {
+                    "color": "red",
+                    "value": "200"
+                  }
+                ]
+              },
+              "unit": "s"
+            },
+            "overrides": [
+              {
+                "matcher": {
+                  "id": "byName",
+                  "options": "99p"
+                },
+                "properties": [
+                  {
+                    "id": "color",
+                    "value": {
+                      "fixedColor": "red",
+                      "mode": "fixed"
+                    }
+                  }
+                ]
+              },
+              {
+                "matcher": {
+                  "id": "byName",
+                  "options": "95p"
+                },
+                "properties": [
+                  {
+                    "id": "color",
+                    "value": {
+                      "fixedColor": "blue",
+                      "mode": "fixed"
+                    }
+                  }
+                ]
+              },
+              {
+                "matcher": {
+                  "id": "byName",
+                  "options": "90p"
+                },
+                "properties": [
+                  {
+                    "id": "color",
+                    "value": {
+                      "fixedColor": "yellow",
+                      "mode": "fixed"
+                    }
+                  }
+                ]
+              },
+              {
+                "matcher": {
+                  "id": "byName",
+                  "options": "50p"
+                },
+                "properties": [
+                  {
+                    "id": "color",
+                    "value": {
+                      "fixedColor": "blue",
+                      "mode": "fixed"
+                    }
+                  }
+                ]
+              },
+              {
+                "matcher": {
+                  "id": "byName",
+                  "options": "p90"
+                },
+                "properties": [
+                  {
+                    "id": "color",
+                    "value": {
+                      "fixedColor": "yellow",
+                      "mode": "fixed"
+                    }
+                  }
+                ]
+              },
+              {
+                "matcher": {
+                  "id": "byName",
+                  "options": "p99"
+                },
+                "properties": [
+                  {
+                    "id": "color",
+                    "value": {
+                      "fixedColor": "red",
+                      "mode": "fixed"
+                    }
+                  }
+                ]
+              },
+              {
+                "matcher": {
+                  "id": "byName",
+                  "options": "p50"
+                },
+                "properties": [
+                  {
+                    "id": "color",
+                    "value": {
+                      "fixedColor": "blue",
+                      "mode": "fixed"
+                    }
+                  }
+                ]
+              }
+            ]
+          },
+          "gridPos": {
+            "h": 8,
+            "w": 11,
+            "x": 5,
+            "y": 35
+          },
+          "id": 209,
+          "options": {
+            "legend": {
+              "calcs": [],
+              "displayMode": "list",
+              "placement": "bottom"
+            },
+            "tooltip": {
+              "mode": "single"
+            }
+          },
+          "targets": [
+            {
+              "exemplar": true,
+              "expr": "histogram_quantile(0.5, sum(rate(image_builder_composer_worker_job_wait_duration_seconds_bucket[$interval])) by (le))",
+              "interval": "",
+              "legendFormat": "p50",
+              "refId": "A"
+            },
+            {
+              "exemplar": true,
+              "expr": "histogram_quantile(0.9, sum(rate(image_builder_composer_worker_job_wait_duration_seconds_bucket[$interval])) by (le))",
+              "hide": false,
+              "interval": "",
+              "legendFormat": "p90",
+              "refId": "B"
+            },
+            {
+              "exemplar": true,
+              "expr": "histogram_quantile(0.99, sum(rate(image_builder_composer_worker_job_wait_duration_seconds_bucket[$interval])) by (le))",
+              "hide": false,
+              "interval": "",
+              "legendFormat": "p99",
+              "refId": "C"
+            }
+          ],
+          "title": "Job Wait Duration",
+          "type": "timeseries"
+        },
+        {
+          "datasource": "${datasource}",
+          "description": "Percent of requests exceeding duration allowed by SLO",
+          "fieldConfig": {
+            "defaults": {
+              "color": {
+                "mode": "palette-classic"
+              },
+              "custom": {
+                "axisLabel": "",
+                "axisPlacement": "auto",
+                "barAlignment": 0,
+                "drawStyle": "line",
+                "fillOpacity": 0,
+                "gradientMode": "none",
+                "hideFrom": {
+                  "legend": false,
+                  "tooltip": false,
+                  "viz": false
+                },
+                "lineInterpolation": "linear",
+                "lineWidth": 1,
+                "pointSize": 5,
+                "scaleDistribution": {
+                  "type": "linear"
+                },
+                "showPoints": "auto",
+                "spanNulls": false,
+                "stacking": {
+                  "group": "A",
+                  "mode": "none"
+                },
+                "thresholdsStyle": {
+                  "mode": "off"
+                }
+              },
+              "mappings": [],
+              "max": 1,
+              "min": 0,
+              "thresholds": {
+                "mode": "absolute",
+                "steps": [
+                  {
+                    "color": "green",
+                    "value": null
+                  },
+                  {
+                    "color": "red",
+                    "value": 80
+                  }
+                ]
+              },
+              "unit": "percentunit"
+            },
+            "overrides": []
+          },
+          "gridPos": {
+            "h": 8,
+            "w": 8,
+            "x": 16,
+            "y": 35
+          },
+          "id": 204,
+          "options": {
+            "legend": {
+              "calcs": [],
+              "displayMode": "hidden",
+              "placement": "bottom"
+            },
+            "tooltip": {
+              "mode": "single"
+            }
+          },
+          "targets": [
+            {
+              "exemplar": true,
+              "expr": "1 - sum(rate(image_builder_composer_worker_job_wait_duration_seconds_bucket{le=\"1536\"}[$interval]))/sum(rate(image_builder_composer_worker_job_wait_duration_seconds_count[$interval]))",
+              "interval": "",
+              "legendFormat": "",
+              "refId": "A"
+            }
+          ],
+          "title": "Slow Request Rate",
+          "type": "timeseries"
+        }
+      ],
+      "refresh": false,
+      "schemaVersion": 30,
+      "style": "dark",
+      "tags": [],
+      "templating": {
+        "list": [
+          {
+            "current": {
+              "selected": false,
+              "text": "app-sre-prod-04-prometheus",
+              "value": "app-sre-prod-04-prometheus"
+            },
+            "description": null,
+            "error": null,
+            "hide": 0,
+            "includeAll": false,
+            "label": null,
+            "multi": false,
+            "name": "datasource",
+            "options": [],
+            "query": "prometheus",
+            "queryValue": "",
+            "refresh": 1,
+            "regex": "/app-sre-(prod-04|stage-01)-prometheus/",
+            "skipUrlSync": false,
+            "type": "datasource"
+          },
+          {
+            "auto": false,
+            "auto_count": 30,
+            "auto_min": "10s",
+            "current": {
+              "selected": false,
+              "text": "28d",
+              "value": "28d"
+            },
+            "description": null,
+            "error": null,
+            "hide": 0,
+            "label": null,
+            "name": "interval",
+            "options": [
+              {
+                "selected": false,
+                "text": "5m",
+                "value": "5m"
+              },
+              {
+                "selected": false,
+                "text": "30m",
+                "value": "30m"
+              },
+              {
+                "selected": false,
+                "text": "1h",
+                "value": "1h"
+              },
+              {
+                "selected": false,
+                "text": "6h",
+                "value": "6h"
+              },
+              {
+                "selected": false,
+                "text": "12h",
+                "value": "12h"
+              },
+              {
+                "selected": false,
+                "text": "1d",
+                "value": "1d"
+              },
+              {
+                "selected": false,
+                "text": "3d",
+                "value": "3d"
+              },
+              {
+                "selected": false,
+                "text": "7d",
+                "value": "7d"
+              },
+              {
+                "selected": false,
+                "text": "14d",
+                "value": "14d"
+              },
+              {
+                "selected": true,
+                "text": "28d",
+                "value": "28d"
+              }
+            ],
+            "query": "5m,30m,1h,6h,12h,1d,3d,7d,14d,28d",
+            "queryValue": "",
+            "refresh": 2,
+            "skipUrlSync": false,
+            "type": "interval"
+          },
+          {
+            "description": "Compose stability SLO target",
+            "error": null,
+            "hide": 2,
+            "label": null,
+            "name": "stability_slo",
+            "query": "0.95",
+            "skipUrlSync": false,
+            "type": "constant"
+          },
+          {
+            "description": "Compose Duration SLO target",
+            "error": null,
+            "hide": 2,
+            "label": null,
+            "name": "latency_slo",
+            "query": "0.9",
+            "skipUrlSync": false,
+            "type": "constant"
+          }
+        ]
+      },
+      "time": {
+        "from": "now-28d",
+        "to": "now"
+      },
+      "timepicker": {
+        "refresh_intervals": [
+          "5s",
+          "10s",
+          "30s",
+          "1m",
+          "5m",
+          "15m",
+          "30m",
+          "1h",
+          "2h",
+          "1d"
+        ],
+        "time_options": [
+          "5m",
+          "15m",
+          "1h",
+          "6h",
+          "12h",
+          "24h",
+          "2d",
+          "7d",
+          "28d"
+        ]
+      },
+      "timezone": "",
+      "title": "Image Builder Worker",
+      "uid": "y8lhL9hnz",
+      "version": 1
+    }


Test patch:
diff --git a/cmd/osbuild-composer/config_test.go b/cmd/osbuild-composer/config_test.go
index 935a083c3..3ccf57dc9 100644
--- a/cmd/osbuild-composer/config_test.go
+++ b/cmd/osbuild-composer/config_test.go
@@ -41,6 +41,7 @@ func TestDefaultConfig(t *testing.T) {
 	require.Equal(t, WorkerAPIConfig{
 		RequestJobTimeout: "0",
 		BasePath:          "/api/worker/v1",
+		EnableArtifacts:   true,
 		EnableTLS:         true,
 		EnableMTLS:        true,
 		EnableJWT:         false,
diff --git a/cmd/osbuild-dnf-json-tests/main_test.go b/cmd/osbuild-dnf-json-tests/main_test.go
index 1b74dcb0d..003c8d6f0 100644
--- a/cmd/osbuild-dnf-json-tests/main_test.go
+++ b/cmd/osbuild-dnf-json-tests/main_test.go
@@ -5,8 +5,8 @@
 package main
 
 import (
-	"fmt"
 	"io/ioutil"
+	"net/http"
 	"os"
 	"path"
 	"testing"
@@ -23,6 +23,11 @@ import (
 
 func TestFetchChecksum(t *testing.T) {
 	dir, err := test.SetUpTemporaryRepository()
+	fs := http.FileServer(http.Dir(dir))
+	go func() {
+		err := http.ListenAndServe(":9000", fs)
+		assert.Nilf(t, err, "Could not start the http server: %v", err)
+	}()
 	defer func(dir string) {
 		err := test.TearDownTemporaryRepository(dir)
 		assert.Nil(t, err, "Failed to clean up temporary repository.")
@@ -31,13 +36,13 @@ func TestFetchChecksum(t *testing.T) {
 
 	repoCfg := rpmmd.RepoConfig{
 		Name:      "repo",
-		BaseURL:   fmt.Sprintf("file://%s", dir),
+		BaseURL:   "http://localhost:9000",
 		IgnoreSSL: true,
 	}
 
 	// use a fullpath to dnf-json, this allows this test to have an arbitrary
 	// working directory
-	rpmMetadata := rpmmd.NewRPMMD(path.Join(dir, "rpmmd"), "/usr/libexec/osbuild-composer/dnf-json")
+	rpmMetadata := rpmmd.NewRPMMD(path.Join(dir, "rpmmd"))
 	_, c, err := rpmMetadata.FetchMetadata([]rpmmd.RepoConfig{repoCfg}, "platform:f31", "x86_64", "31")
 	assert.Nilf(t, err, "Failed to fetch checksum: %v", err)
 	assert.NotEqual(t, "", c["repo"], "The checksum is empty")
@@ -64,7 +69,7 @@ func TestCrossArchDepsolve(t *testing.T) {
 
 			// use a fullpath to dnf-json, this allows this test to have an arbitrary
 			// working directory
-			rpm := rpmmd.NewRPMMD(dir, "/usr/libexec/osbuild-composer/dnf-json")
+			rpm := rpmmd.NewRPMMD(dir)
 
 			repos, err := rpmmd.LoadRepositories([]string{repoDir}, distroStruct.Name())
 			require.NoErrorf(t, err, "Failed to LoadRepositories %v", distroStruct.Name())
diff --git a/internal/cloudapi/v2/v2_test.go b/internal/cloudapi/v2/v2_test.go
index ebb9a55f8..0546a4a70 100644
--- a/internal/cloudapi/v2/v2_test.go
+++ b/internal/cloudapi/v2/v2_test.go
@@ -392,6 +392,12 @@ func TestComposeCustomizations(t *testing.T) {
 				"name": "user1",
 				"groups": [ "wheel" ],
 				"key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINrGKErMYi+MMUwuHaRAJmRLoIzRf2qD2dD5z0BTx/6x"
+			}],
+			"payload_repositories": [{
+				"baseurl": "some-custom-repo.org",
+				"check_gpg": false,
+				"ignore_ssl": false,
+				"gpg_key": "some-gpg-key"
 			}]
 		},
 		"image_request":{
diff --git a/internal/distro/distro_test_common/distro_test_common.go b/internal/distro/distro_test_common/distro_test_common.go
index 6a1437be4..457b0aac9 100644
--- a/internal/distro/distro_test_common/distro_test_common.go
+++ b/internal/distro/distro_test_common/distro_test_common.go
@@ -132,7 +132,7 @@ func TestDistro_Manifest(t *testing.T, pipelinePath string, prefix string, regis
 func getImageTypePkgSpecSets(imageType distro.ImageType, bp blueprint.Blueprint, repos []rpmmd.RepoConfig, cacheDir, dnfJsonPath string) map[string][]rpmmd.PackageSpec {
 	imgPackageSets := imageType.PackageSets(bp)
 
-	rpm_md := rpmmd.NewRPMMD(cacheDir, dnfJsonPath)
+	rpm_md := rpmmd.NewRPMMD(cacheDir)
 
 	imgPackageSpecSets := make(map[string][]rpmmd.PackageSpec)
 	for name, packages := range imgPackageSets {
diff --git a/internal/distro/test_distro/distro.go b/internal/distro/test_distro/distro.go
index 585d846f9..f23f32bd4 100644
--- a/internal/distro/test_distro/distro.go
+++ b/internal/distro/test_distro/distro.go
@@ -187,6 +187,10 @@ func (t *TestImageType) PayloadPipelines() []string {
 	return distro.PayloadPipelinesFallback()
 }
 
+func (t *TestImageType) PayloadPackageSets() []string {
+	return []string{}
+}
+
 func (t *TestImageType) Exports() []string {
 	return distro.ExportsFallback()
 }
diff --git a/test/cases/api.sh b/test/cases/api.sh
index 69d479596..3413f10c5 100755
--- a/test/cases/api.sh
+++ b/test/cases/api.sh
@@ -303,6 +303,44 @@ function cleanup() {
 }
 trap cleanup EXIT
 
+# make a dummy rpm and repo to test payload_repositories
+sudo dnf install -y rpm-build createrepo
+DUMMYRPMDIR=$(mktemp -d)
+DUMMYSPECFILE="$DUMMYRPMDIR/dummy.spec"
+PAYLOAD_REPO_PORT="9999"
+PAYLOAD_REPO_URL="http://localhost:9999"
+pushd "$DUMMYRPMDIR"
+
+cat <<EOF > "$DUMMYSPECFILE"
+#----------- spec file starts ---------------
+Name:                   dummy
+Version:                1.0.0
+Release:                0
+BuildArch:              noarch
+Vendor:                 dummy
+Summary:                Provides %{name}
+License:                BSD
+Provides:               dummy
+
+%description
+%{summary}
+
+%files
+EOF
+
+mkdir -p "DUMMYRPMDIR/rpmbuild"
+rpmbuild --quiet --define "_topdir $DUMMYRPMDIR/rpmbuild" -bb "$DUMMYSPECFILE"
+
+mkdir -p "$DUMMYRPMDIR/repo"
+cp "$DUMMYRPMDIR"/rpmbuild/RPMS/noarch/*rpm "$DUMMYRPMDIR/repo"
+pushd "$DUMMYRPMDIR/repo"
+createrepo .
+sudo python3 -m http.server "$PAYLOAD_REPO_PORT" &
+KILL_PIDS+=("$!")
+popd
+popd
+
+
 #
 # Install the necessary cloud provider client tools
 #
@@ -483,8 +521,14 @@ function createReqFileAWS() {
 {
   "distribution": "$DISTRO",
   "customizations": {
+    "payload_repositories": [
+      {
+        "baseurl": "$PAYLOAD_REPO_URL"
+      }
+    ],
     "packages": [
-      "postgresql"
+      "postgresql",
+      "dummy"
     ]${SUBSCRIPTION_BLOCK},
     "users":[
       {
@@ -522,8 +566,14 @@ function createReqFileAWSS3() {
 {
   "distribution": "$DISTRO",
   "customizations": {
+    "payload_repositories": [
+      {
+        "baseurl": "$PAYLOAD_REPO_URL"
+      }
+    ],
     "packages": [
-      "postgresql"
+      "postgresql",
+      "dummy"
     ],
     "users":[
       {
@@ -566,8 +616,14 @@ function createReqFileGCP() {
 {
   "distribution": "$DISTRO",
   "customizations": {
+    "payload_repositories": [
+      {
+        "baseurl": "$PAYLOAD_REPO_URL"
+      }
+    ],
     "packages": [
-      "postgresql"
+      "postgresql",
+      "dummy"
     ]${SUBSCRIPTION_BLOCK}
   },
   "image_request": {
@@ -592,8 +648,14 @@ function createReqFileAzure() {
 {
   "distribution": "$DISTRO",
   "customizations": {
+    "payload_repositories": [
+      {
+        "baseurl": "$PAYLOAD_REPO_URL"
+      }
+    ],
     "packages": [
-      "postgresql"
+      "postgresql",
+      "dummy"
     ]${SUBSCRIPTION_BLOCK}
   },
   "image_request": {
@@ -821,7 +883,7 @@ function _instanceCheck() {
   local _ssh="$1"
 
   # Check if postgres is installed
-  $_ssh rpm -q postgresql
+  $_ssh rpm -q postgresql dummy
 
   # Verify subscribe status. Loop check since the system may not be registered such early(RHEL only)
   if [[ "$ID" == "rhel" ]]; then
@@ -1197,6 +1259,7 @@ jwt_acl_file = ""
 [worker]
 pg_host = "localhost"
 pg_port = "5432"
+enable_artifacts = false
 pg_database = "osbuildcomposer"
 pg_user = "postgres"
 pg_password = "foobar"
diff --git a/test/cases/generation.sh b/test/cases/generation.sh
new file mode 100755
index 000000000..43668f702
--- /dev/null
+++ b/test/cases/generation.sh
@@ -0,0 +1,33 @@
+#!/usr/bin/bash
+set -euxo pipefail
+
+# Provision the software under test.
+/usr/libexec/osbuild-composer-test/provision.sh
+
+# Get OS data
+source /usr/libexec/osbuild-composer-test/set-env-variables.sh
+
+# Colorful timestamped output.
+function greenprint {
+    echo -e "\033[1;32m[$(date -Isecond)] ${1}\033[0m"
+}
+
+# install requirements
+sudo dnf -y install go
+
+# stop dnf-json socket
+sudo systemctl stop osbuild-dnf-json.socket
+
+# test the test cases generation
+WORKDIR=$(mktemp -d -p /var/tmp)
+
+OSBUILD_LABEL=$(matchpathcon -n "$(which osbuild)")
+chcon "$OSBUILD_LABEL" tools/image-info
+
+# test the test case generation when dnf-json socket is stopped
+sudo ./tools/test-case-generators/generate-test-cases\
+    --output test/data/manifests\
+    --arch x86_64\
+    --distro rhel-8\
+    --image-type qcow2\
+    --store "$WORKDIR"
diff --git a/test/cases/regression-old-worker-new-composer.sh b/test/cases/regression-old-worker-new-composer.sh
index c01f1acf9..486534257 100644
--- a/test/cases/regression-old-worker-new-composer.sh
+++ b/test/cases/regression-old-worker-new-composer.sh
@@ -68,7 +68,7 @@ sudo podman run  \
      -v "$WELDR_DIR:/run/weldr/":Z \
      -p 8700:8700 \
      "quay.io/osbuild/osbuild-composer-ubi-pr:${CI_COMMIT_SHA}" \
-     --weldr-api --remote-worker-api \
+     --weldr-api --dnf-json --remote-worker-api \
      --no-local-worker-api --no-composer-api
 
 # try starting a worker
diff --git a/tools/test-case-generators/generate-test-cases b/tools/test-case-generators/generate-test-cases
index 8a55d74ab..ec6440ef0 100755
--- a/tools/test-case-generators/generate-test-cases
+++ b/tools/test-case-generators/generate-test-cases
@@ -7,6 +7,9 @@ import os
 import sys
 import tempfile
 
+def is_subprocess_succeeding(*args, **kwargs):
+    sp = subprocess.run(*args, **kwargs, stdout=subprocess.PIPE)
+    return sp.returncode == 0
 
 def get_subprocess_stdout(*args, **kwargs):
     sp = subprocess.run(*args, **kwargs, stdout=subprocess.PIPE)
@@ -63,6 +66,14 @@ class TestCaseGenerator:
     def get_test_case(self, no_image_info, store):
         compose_request = json.dumps(self.test_case["compose-request"])
 
+        command = ["systemctl", "is-active", "--quiet", "osbuild-dnf-json.socket"]
+        if not is_subprocess_succeeding(command):
+            print("osbuild-dnf-json.socket is not running, starting it up")
+            command = ["systemctl", "start", "osbuild-dnf-json.socket", "--quiet"]
+            if not is_subprocess_succeeding(command):
+                print("impossible to start osbuild-dnf-json.socket with systemd")
+                sys.exit(1)
+
         pipeline_command = ["go", "run", "./cmd/osbuild-pipeline", "-"]
         self.test_case["manifest"] = json.loads(get_subprocess_stdout(pipeline_command, input=compose_request, encoding="utf-8"))