Repository: osbuild/osbuild-composer
Pull request #2062 (2021-11-24 10:27:19)

dnf json as a service
This pull request includes:

- [ ] adequate testing for the new functionality or fixed issue
- [ ] adequate documentation informing people about the change such as
  - [ ] submit a PR for the [guides](https://github.com/osbuild/guides) repository if this PR changed any behavior described there: https://www.osbuild.org/guides/

<!--
Thanks for proposing a change to osbuild-composer!

Please don't remove the above check list. These are things that each pull
request must have before it is merged. It helps maintainers to not forget
anything.

If the reason for ticking any of the boxes is ambiguous, please add a short
note explaining why.

In addition, if this pull request fixes a downstream issue, please refer to
test/README.md and add these additional items:

- [ ] 1st commit of any `rhbz#` related PR contains bug reproducer; CI reports FAIL or
- [ ] PR contains automated tests for new functionality and
- [ ] QE has approved reproducer/new tests and
- [ ] Subsequent commits provide bug fixes without modifying the reproducer; CI reports PASS and
- [ ] QE approves this PR; RHBZ status is set to `MODIFIED + Verified=Tested`
-->


lavocatt: Unfortunately I discovered that I have memory leaks to take care :/
lavocatt: > The issue is currently noticeable in the failing CI Manifest tests because they use the same approach as generators of image test cases.

The issue in the manifest test case is due to a memory leak in the daemon.

```
Nov 30 08:40:04 ip-10-30-18-53.us-east-1.aws.redhat.com kernel: Out of memory: Killed process 19731 (dnf-json) total-vm:3905420kB, anon-rss:3607436kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:7272kB oom_score_adj:0

```


lavocatt: > Please correct me if I'm wrong, but I think that running `dnf-json` only as a service will break our tooling used to generate image test cases. Specifically `osbuild-pipeline` is not run as a service and it uses RPMMD with `dnf-json` to depsolve image package sets:

@thozza I've corrected the issue. I removed the dnfJsonPath from the api of RPMMD.
It works well also since I solved the memory leak issue (see https://gitlab.com/osbuild/ci/osbuild-composer/-/jobs/1832645625). 

For clients of the Rpmmd api the behaviour is the same as before, because repository.go hides the complexity of calling the daemon through the unix socket.

SystemD is responsible for starting the process and I don't think that we should shut it down. Maybe we can add a timeout that closes the daemon if no communication was made to it for along time.

lavocatt: My main issue now is that `create_repo_c` is creating a repository with access rights that does not allow the `dnf-json` daemon to access the files. I'll probably have to `chmod` stuff, but it seems hacky. I'll try to figure out something better.
ondrejbudai: Isn't this a SELinux issue? IIRC SELinux prevents root processes to access home directories. A solution might be to run an ad-hoc HTTP server, I suggest something like `python -m http.server`, or a simple Go thing.
thozza: > For clients of the Rpmmd api the behaviour is the same as before, because repository.go hides the complexity of calling the daemon through the unix socket.
> 
> SystemD is responsible for starting the process and I don't think that we should shut it down. Maybe we can add a timeout that closes the daemon if no communication was made to it for along time.

Right, this will work really nicely except for the case when generating image test cases...

The approach expects that RPMs which ship the `dnf-json` as a service are installed on the system when the API is used, e.g. when image test cases are being generated. This is in general not the case when generating image test cases, because thanks to `osbuild-pipeline` being run using `go run ...` we just need to generate manifests using the appropriate source code, but we don't need to have the RPMs built from it also installed on the system.

For CI testing, this should not be an issue, since the RPMs built from the tested source code are always installed on the system under test.

If you generate image test cases, e.g. using `./tools/test-case-generators/generate-all-test-cases ... remote ...`, the script by default installs latest versions of `osbuild` and `osbuild-composer` available in the distribution repositories. There is an option to define additional repos and these will have a higher priority. However there is a pretty unfortunate downside with `osbuild-composer`... we don't build RPMs for all architectures in our CI, for which we need to generate image test cases and test them (specifically `s390x` and `ppc64le`). This would be easily solved if we could use COPR repo which would contain the latest upstream builds (e.g. as it is done by PackIt for `osbuild` repository). However we don't have this yet for `osbuild-composer` and until the `osbuild-composer` RPMs with `dnf-json` as a service would get to Fedora 34/35 updates stable repo, generating image test cases for all architectures from the git `main` would be extremely harder.

Therefore if we go this way (having `dnf-json` only as a service), I propose that we first enable RPM builds for all architectures for Fedora using PackIt in osbuild-composer repository.
lavocatt: > Isn't this a SELinux issue? IIRC SELinux prevents root processes to access home directories. A solution might be to run an ad-hoc HTTP server, I suggest something like `python -m http.server`, or a simple Go thing.

In fact, it's because `dnf-json` has a private `/tmp`, so when it tries to read the files at the path generated by the test it fails.
Using an `http.FileServer` in go makes the trick, good idea!
Gundersanne: So dnf-json as a daemon does break composer as a standalone container for anything that expects the depsolving to happen in composer itself (so koji and the weldr api). This is the reason that the old-worker-new-composer regression test is broken as well, since there the composer PR version is running in a container and exposing the weldr api, but of course the dnf-json daemon isn't running in the container because we don't have systemd in the container.

We could expand `containers/osbuild-composer/entrypoint.py` to include dnf-json now, and invoke it the same way as we do weldr etc.... 
Gundersanne: So I finally managed to get the regression test to work https://gitlab.com/osbuild/ci/osbuild-composer/-/jobs/1841343046, but with a hack https://github.com/osbuild/osbuild-composer/pull/2085/commits/3612c14c2579d169ca3da6107df5c19006511c86#diff-0ffbc497a008289d8efbb1d7edc7781d4e0c3a1e72ab964bd3a3a9401dadae86R257 not sure if there's a better way.

The fds are shared between the container entrpoint process and all the child processes (so osbuild-composer which has a variable number of fds, and dnf-json). So we somehow need to get the correct fd into dnf-json, which should of course default to 3. 
thozza: I had a look at https://github.com/osbuild/osbuild-composer/pull/2062/commits/8f39dd5c08291bf1973664647c7c593313fbad44 and think it won’t work with regenerating image test cases using `./tools/test-case-generators/generate-all-test-cases`. The reason is that the `dnf-json` installed on the system won’t be the one which creates a socket (until that version of composer gets to the distribution updates repos) and thus the RPMMD API from the “newer” sources copied over to the host won’t be able to connect to it.

I think that one solution could be to have a fallback in the RPMMD API to execute `dnf-json` as an executable if it is not listening on a socket, but I think this would be a bad approach as the code would be relevant for a limited time. So the alternative is IMO only to install the osbuild-composer version which includes `dnf-json` able to run as a service on the remote hosts when regenerating test cases. There are two possibilities:

1. Use PackIt and RPMs built in COPR when regenerating test cases by passing the COPR repo to the script. Passing the repo would be needed only for a limited time until the osbuild-composer version with `dnf-json` as a service would get to Fedora updates repo. 
2. Build the osbuild-composer RPMs on the remote host from copied sources and install them before regenerating test cases.

I'm in favor of (1), since having PackIt building composer RPMs for all arches would be definitely beneficial. (2) seems to me as adding additional potentially fragile step to generating test cases, but it could be a valid option. (2) would also require more coding as (1) is mostly about enabling PackIt and creating a YAML config for it. 
ondrejbudai: Note that COPR doesn't support s390x for el8 and both ppc64le and s390x for el9. For Fedora it supports all arches we need but that means we are stuck with Fedora for regenerating the test cases.
thozza: > Note that COPR doesn't support s390x for el8 and both ppc64le and s390x for el9. For Fedora it supports all arches we need but that means we are stuck with Fedora for regenerating the test cases.

True.

We can ask COPR devels to enable these architectures, but s390x will be always only emulated 😞 .

But to be honest, I have been regenerating all image test cases on Fedora 34 for the past "forever" and it works great. There are sometimes issues which needed a workaround, but these show up in CI right away when a PR with regenerated test cases is proposed. So from my point of view, using only F34/F35 for regenerating test cases is not an issue...
thozza: I implemented a workaround in the `generate-all-test-cases` script, which will allow the user to build the composer RPMs from sources on the runner and install them. Once merged, we should have a workaround for the transition period until composer RPMs with this change will get to the distribution repositories.

For more information see PR#2098
thozza: I tested generating image test cases with the proposed branch and `dnf-json` service does not get started (I didn't expect it to, since there is no code in `internal/rpmmd/repository.go` which would start it. The generation fails with:
```
panic: Could not depsolve: Post "http://unix": dial unix /run/osbuild-dnf-json/api.sock: connect: no such file or directory

goroutine 1 [running]:
main.main()
        /root/tmp.VhHjpMVJIn/sources/cmd/osbuild-pipeline/main.go:127 +0x11c6
exit status 2
```

I think that the solution could be to simply extend the `tools/test-case-generators/generate-test-cases` script to check if the `osbuild-dnf-json.socket` is active and if not, start it. This should not be an issue, since the script must be run with root privileges as it runs `osbuild` and `image-info`...

Patch:
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 37f9c65e1..d17ff25ec 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -206,6 +206,26 @@ Regression:
       - "*.repo"
     when: always
 
+Test Case Generation:
+  stage: test
+  extends: .terraform
+  rules:
+    - if: '$CI_PIPELINE_SOURCE != "schedule"'
+  script:
+    - schutzbot/deploy.sh
+    - /usr/libexec/tests/osbuild-composer/generation.sh
+  interruptible: true
+  parallel:
+    matrix:
+      - RUNNER:
+          - aws/fedora-34-x86_64
+        INTERNAL_NETWORK: ["true"]
+  artifacts:
+    paths:
+      - journal-log.gpg
+      - "*.repo"
+    when: always
+
 OSTree:
   stage: test
   extends: .terraform/openstack
diff --git a/cmd/osbuild-composer/composer.go b/cmd/osbuild-composer/composer.go
index 051c51383..2ea508a1a 100644
--- a/cmd/osbuild-composer/composer.go
+++ b/cmd/osbuild-composer/composer.go
@@ -66,7 +66,7 @@ func NewComposer(config *ComposerConfigFile, stateDir, cacheDir string) (*Compos
 	c.distros = distroregistry.NewDefault()
 	logrus.Infof("Loaded %d distros", len(c.distros.List()))
 
-	c.rpm = rpmmd.NewRPMMD(path.Join(c.cacheDir, "rpmmd"), "/usr/libexec/osbuild-composer/dnf-json")
+	c.rpm = rpmmd.NewRPMMD(path.Join(c.cacheDir, "rpmmd"))
 
 	var jobs jobqueue.JobQueue
 	if config.Worker.PGDatabase != "" {
diff --git a/cmd/osbuild-pipeline/main.go b/cmd/osbuild-pipeline/main.go
index 7496028f8..d80493de2 100644
--- a/cmd/osbuild-pipeline/main.go
+++ b/cmd/osbuild-pipeline/main.go
@@ -118,7 +118,7 @@ func main() {
 		panic("os.UserHomeDir(): " + err.Error())
 	}
 
-	rpm_md := rpmmd.NewRPMMD(path.Join(home, ".cache/osbuild-composer/rpmmd"), "/usr/libexec/osbuild-composer/dnf-json")
+	rpm_md := rpmmd.NewRPMMD(path.Join(home, ".cache/osbuild-composer/rpmmd"))
 
 	packageSpecSets := make(map[string][]rpmmd.PackageSpec)
 	for name, packages := range packageSets {
diff --git a/cmd/osbuild-store-dump/main.go b/cmd/osbuild-store-dump/main.go
index 4f2599268..bf23daece 100644
--- a/cmd/osbuild-store-dump/main.go
+++ b/cmd/osbuild-store-dump/main.go
@@ -138,7 +138,7 @@ func main() {
 	if err != nil {
 		panic("os.UserHomeDir(): " + err.Error())
 	}
-	rpmmd := rpmmd.NewRPMMD(path.Join(homeDir, ".cache/osbuild-composer/rpmmd"), "/usr/libexec/osbuild-composer/dnf-json")
+	rpmmd := rpmmd.NewRPMMD(path.Join(homeDir, ".cache/osbuild-composer/rpmmd"))
 
 	s := store.New(&cwd, a, nil)
 	if s == nil {
diff --git a/cmd/osbuild-worker/jobimpl-depsolve.go b/cmd/osbuild-worker/jobimpl-depsolve.go
index 091d7d4f2..c42272741 100644
--- a/cmd/osbuild-worker/jobimpl-depsolve.go
+++ b/cmd/osbuild-worker/jobimpl-depsolve.go
@@ -12,7 +12,7 @@ type DepsolveJobImpl struct {
 }
 
 func (impl *DepsolveJobImpl) depsolve(packageSets map[string]rpmmd.PackageSet, repos []rpmmd.RepoConfig, modulePlatformID, arch, releasever string) (map[string][]rpmmd.PackageSpec, error) {
-	rpmMD := rpmmd.NewRPMMD(impl.RPMMDCache, "/usr/libexec/osbuild-composer/dnf-json")
+	rpmMD := rpmmd.NewRPMMD(impl.RPMMDCache)
 
 	packageSpecs := make(map[string][]rpmmd.PackageSpec)
 	for name, packageSet := range packageSets {
diff --git a/containers/osbuild-composer/entrypoint.py b/containers/osbuild-composer/entrypoint.py
index 89090428d..685378e90 100644
--- a/containers/osbuild-composer/entrypoint.py
+++ b/containers/osbuild-composer/entrypoint.py
@@ -95,12 +95,27 @@ class Cli(contextlib.AbstractContextManager):
             help="Disable the weldr-API",
         )
 
+        # --[no-]dnf-json
+        self._parser.add_argument(
+            "--dnf-json",
+            action="store_true",
+            dest="dnf_json",
+            help="Enable  dnf-json",
+        )
+        self._parser.add_argument(
+            "--no-dnf-json",
+            action="store_false",
+            dest="dnf_json",
+            help="Disable dnf-json",
+        )
+
         self._parser.set_defaults(
             builtin_worker=False,
             composer_api=False,
             local_worker_api=False,
             remote_worker_api=False,
             weldr_api=False,
+            dnf_json=False,
         )
 
         return self._parser.parse_args(self._argv[1:])
@@ -238,11 +253,29 @@ class Cli(contextlib.AbstractContextManager):
             preexec_fn=preexec_setenv,
         )
 
+    def _spawn_dnf_json(self):
+        cmd = [
+            "/usr/libexec/osbuild-composer/dnf-json",
+        ]
+
+        dnfenv = os.environ.copy()
+        dnfenv["LISTEN_FDS"] = "0"
+        dnfenv["LISTEN_FDNAMES"] = ""
+
+        return subprocess.Popen(
+            cmd,
+            cwd="/usr/libexec/osbuild-composer",
+            stdin=subprocess.DEVNULL,
+            stderr=subprocess.STDOUT,
+            env=dnfenv,
+        )
+
     def run(self):
         """Program Runtime"""
 
         proc_composer = None
         proc_worker = None
+        proc_dnf_json = None
         res = 0
         sockets = self._prepare_sockets()
 
@@ -250,24 +283,35 @@ class Cli(contextlib.AbstractContextManager):
             if self.args.builtin_worker:
                 proc_worker = self._spawn_worker()
 
+            if self.args.dnf_json:
+                proc_dnf_json = self._spawn_dnf_json()
+
             proc_composer = self._spawn_composer(sockets)
 
             res = proc_composer.wait()
             if proc_worker:
                 proc_worker.terminate()
                 proc_worker.wait()
+            if proc_dnf_json:
+                proc_dnf_json.terminate()
+                proc_dnf_json.wait()
 
             return res
         except KeyboardInterrupt:
             if proc_worker:
                 proc_worker.terminate()
                 proc_worker.wait()
+            if proc_dnf_json:
+                proc_dnf_json.terminate()
+                proc_dnf_json.wait()
             if proc_composer:
                 proc_composer.terminate()
                 res = proc_composer.wait()
         except:
             if proc_worker:
                 proc_worker.kill()
+            if proc_dnf_json:
+                proc_dnf_json.kill()
             if proc_composer:
                 proc_composer.kill()
             raise
diff --git a/distribution/osbuild-composer.service b/distribution/osbuild-composer.service
index a277c8604..9d88b0d81 100644
--- a/distribution/osbuild-composer.service
+++ b/distribution/osbuild-composer.service
@@ -1,5 +1,6 @@
 [Unit]
 Description=OSBuild Composer
+Requires=osbuild-dnf-json.socket
 After=network.target
 
 # Weldr API needs a local worker by default.
diff --git a/distribution/osbuild-dnf-json.service b/distribution/osbuild-dnf-json.service
new file mode 100644
index 000000000..1f012fb46
--- /dev/null
+++ b/distribution/osbuild-dnf-json.service
@@ -0,0 +1,13 @@
+[Unit]
+Description=OSbuild dnf-json service
+Requires=osbuild-dnf-json.socket
+
+[Service]
+Type=simple
+PrivateTmp=true
+ExecStart=/usr/libexec/osbuild-composer/dnf-json
+Restart=always
+RestartSec=1s
+CacheDirectory=osbuild-dnf-json
+# systemd >= 240 sets this, but osbuild-worker runs on earlier versions
+Environment="CACHE_DIRECTORY=/var/cache/osbuild-dnf-json"
diff --git a/distribution/osbuild-dnf-json.socket b/distribution/osbuild-dnf-json.socket
new file mode 100644
index 000000000..231f75e1a
--- /dev/null
+++ b/distribution/osbuild-dnf-json.socket
@@ -0,0 +1,9 @@
+[Unit]
+Description=OSbuild dnf-json socket
+
+[Socket]
+Service=osbuild-dnf-json.service
+ListenStream=/run/osbuild-dnf-json/api.sock
+
+[Install]
+WantedBy=sockets.target
diff --git a/distribution/osbuild-remote-worker@.service b/distribution/osbuild-remote-worker@.service
index ab19e95c3..965c5b7ed 100644
--- a/distribution/osbuild-remote-worker@.service
+++ b/distribution/osbuild-remote-worker@.service
@@ -1,5 +1,6 @@
 [Unit]
 Description=OSBuild Composer Remote Worker (%i)
+Requires=osbuild-dnf-json.socket
 After=network.target
 
 [Service]
diff --git a/distribution/osbuild-worker@.service b/distribution/osbuild-worker@.service
index 4423085d8..335122304 100644
--- a/distribution/osbuild-worker@.service
+++ b/distribution/osbuild-worker@.service
@@ -1,6 +1,6 @@
 [Unit]
 Description=OSBuild Composer Worker (%i)
-Requires=osbuild-local-worker.socket
+Requires=osbuild-local-worker.socket osbuild-dnf-json.socket
 After=network.target osbuild-local-worker.socket
 
 [Service]
diff --git a/dnf-json b/dnf-json
index cc7cfac3b..9f16dd1f8 100755
--- a/dnf-json
+++ b/dnf-json
@@ -1,155 +1,138 @@
 #!/usr/bin/python3
 
 import datetime
-import dnf
 import hashlib
-import hawkey
 import json
-import sys
 import tempfile
+import os
+import socket
+import socketserver
+import logging
+import sys
+from http.server import BaseHTTPRequestHandler
+import pathlib
 
-DNF_ERROR_EXIT_CODE = 10
-
-
-def timestamp_to_rfc3339(timestamp):
-    d = datetime.datetime.utcfromtimestamp(package.buildtime)
-    return d.strftime('%Y-%m-%dT%H:%M:%SZ')
-
-
-def dnfrepo(desc, parent_conf=None):
-    """Makes a dnf.repo.Repo out of a JSON repository description"""
-
-    repo = dnf.repo.Repo(desc["id"], parent_conf)
-
-    if "baseurl" in desc:
-        repo.baseurl = desc["baseurl"]
-    elif "metalink" in desc:
-        repo.metalink = desc["metalink"]
-    elif "mirrorlist" in desc:
-        repo.mirrorlist = desc["mirrorlist"]
-    else:
-        assert False
-
-    if desc.get("ignoressl", False):
-        repo.sslverify = False
-    if "sslcacert" in desc:
-        repo.sslcacert = desc["sslcacert"]
-    if "sslclientkey" in desc:
-        repo.sslclientkey = desc["sslclientkey"]
-    if "sslclientcert" in desc:
-        repo.sslclientcert = desc["sslclientcert"]
-
-    # In dnf, the default metadata expiration time is 48 hours. However,
-    # some repositories never expire the metadata, and others expire it much
-    # sooner than that. We therefore allow this to be configured. If nothing
-    # is provided we error on the side of checking if we should invalidate
-    # the cache. If cache invalidation is not necessary, the overhead of
-    # checking is in the hundreds of milliseconds. In order to avoid this
-    # overhead accumulating for API calls that consist of several dnf calls,
-    # we set the expiration to a short time period, rather than 0.
-    repo.metadata_expire = desc.get("metadata_expire", "20s")
-
-    return repo
-
-
-def create_base(repos, module_platform_id, persistdir, cachedir, arch):
-    base = dnf.Base()
-
-    # Enable fastestmirror to ensure we choose the fastest mirrors for
-    # downloading metadata (when depsolving) and downloading packages.
-    base.conf.fastestmirror = True
-
-    # We use the same cachedir for multiple architectures. Unfortunately,
-    # this is something that doesn't work well in certain situations
-    # with zchunk:
-    # Imagine that we already have cache for arch1. Then, we use dnf-json
-    # to depsolve for arch2. If ZChunk is enabled and available (that's
-    # the case for Fedora), dnf will try to download only differences
-    # between arch1 and arch2 metadata. But, as these are completely
-    # different, dnf must basically redownload everything.
-    # For downloding deltas, zchunk uses HTTP range requests. Unfortunately,
-    # if the mirror doesn't support multi range requests, then zchunk will
-    # download one small segment per a request. Because we need to update
-    # the whole metadata (10s of MB), this can be extremely slow in some cases.
-    # I think that we can come up with a better fix but let's just disable
-    # zchunk for now. As we are already downloading a lot of data when
-    # building images, I don't care if we download even more.
-    base.conf.zchunk = False
-
-    # Try another mirror if it takes longer than 5 seconds to connect.
-    base.conf.timeout = 5
-
-    # Set the rest of the dnf configuration.
-    base.conf.module_platform_id = module_platform_id
-    base.conf.config_file_path = "/dev/null"
-    base.conf.persistdir = persistdir
-    base.conf.cachedir = cachedir
-    base.conf.substitutions['arch'] = arch
-    base.conf.substitutions['basearch'] = dnf.rpm.basearch(arch)
-
-    for repo in repos:
-        base.repos.add(dnfrepo(repo, base.conf))
-
-    base.fill_sack(load_system_repo=False)
-    return base
-
-
-def exit_with_dnf_error(kind: str, reason: str):
-    json.dump({"kind": kind, "reason": reason}, sys.stdout)
-    sys.exit(DNF_ERROR_EXIT_CODE)
-
-
-def repo_checksums(base):
-    checksums = {}
-    for repo in base.repos.iter_enabled():
-        # Uses the same algorithm as libdnf to find cache dir:
-        #   https://github.com/rpm-software-management/libdnf/blob/master/libdnf/repo/Repo.cpp#L1288
-        if repo.metalink:
-            url = repo.metalink
-        elif repo.mirrorlist:
-            url = repo.mirrorlist
-        elif repo.baseurl:
-            url = repo.baseurl[0]
-        else:
-            assert False
-
-        digest = hashlib.sha256(url.encode()).hexdigest()[:16]
-
-        repomd_file = f"{repo.id}-{digest}/repodata/repomd.xml"
-        with open(f"{base.conf.cachedir}/{repomd_file}", "rb") as f:
-            repomd = f.read()
-
-        checksums[repo.id] = "sha256:" + hashlib.sha256(repomd).hexdigest()
-
-    return checksums
-
-
-call = json.load(sys.stdin)
-command = call["command"]
-arguments = call["arguments"]
-repos = arguments.get("repos", {})
-arch = arguments["arch"]
-cachedir = arguments["cachedir"]
-module_platform_id = arguments["module_platform_id"]
+import dnf
+import hawkey
 
-with tempfile.TemporaryDirectory() as persistdir:
-    try:
-        base = create_base(
-            repos,
-            module_platform_id,
-            persistdir,
-            cachedir,
-            arch
-        )
-    except dnf.exceptions.Error as e:
-        exit_with_dnf_error(
-            type(e).__name__,
-            f"Error occurred when setting up repo: {e}"
+formatter = logging.Formatter(
+        fmt="%(asctime)s %(name)s.%(levelname)s: %(message)s",
+        datefmt="%Y.%m.%d %H:%M:%S"
         )
+handler = logging.StreamHandler(stream=sys.stdout)
+handler.setFormatter(formatter)
+
+log = logging.getLogger('dnf-json')
+log.addHandler(handler)
+log.setLevel(logging.INFO)
+
+class Solver():
+
+    def __init__(self, repos, module_platform_id, persistdir, cachedir, arch):
+        self.base = dnf.Base()
+
+        # Enable fastestmirror to ensure we choose the fastest mirrors for
+        # downloading metadata (when depsolving) and downloading packages.
+        self.base.conf.fastestmirror = True
+
+        # We use the same cachedir for multiple architectures. Unfortunately,
+        # this is something that doesn't work well in certain situations
+        # with zchunk:
+        # Imagine that we already have cache for arch1. Then, we use dnf-json
+        # to depsolve for arch2. If ZChunk is enabled and available (that's
+        # the case for Fedora), dnf will try to download only differences
+        # between arch1 and arch2 metadata. But, as these are completely
+        # different, dnf must basically redownload everything.
+        # For downloding deltas, zchunk uses HTTP range requests. Unfortunately,
+        # if the mirror doesn't support multi range requests, then zchunk will
+        # download one small segment per a request. Because we need to update
+        # the whole metadata (10s of MB), this can be extremely slow in some cases.
+        # I think that we can come up with a better fix but let's just disable
+        # zchunk for now. As we are already downloading a lot of data when
+        # building images, I don't care if we download even more.
+        self.base.conf.zchunk = False
+
+        # Try another mirror if it takes longer than 5 seconds to connect.
+        self.base.conf.timeout = 5
+
+        # Set the rest of the dnf configuration.
+        self.base.conf.module_platform_id = module_platform_id
+        self.base.conf.config_file_path = "/dev/null"
+        self.base.conf.persistdir = persistdir
+        self.base.conf.cachedir = cachedir
+        self.base.conf.substitutions['arch'] = arch
+        self.base.conf.substitutions['basearch'] = dnf.rpm.basearch(arch)
+
+        for repo in repos:
+            self.base.repos.add(self._dnfrepo(repo, self.base.conf))
+        self.base.fill_sack(load_system_repo=False)
+
+    def _dnfrepo(self, desc, parent_conf=None):
+        """Makes a dnf.repo.Repo out of a JSON repository description"""
+
+        repo = dnf.repo.Repo(desc["id"], parent_conf)
+
+        if "baseurl" in desc:
+            repo.baseurl = desc["baseurl"]
+        elif "metalink" in desc:
+            repo.metalink = desc["metalink"]
+        elif "mirrorlist" in desc:
+            repo.mirrorlist = desc["mirrorlist"]
+        else:
+            assert False
 
-    if command == "dump":
+        if desc.get("ignoressl", False):
+            repo.sslverify = False
+        if "sslcacert" in desc:
+            repo.sslcacert = desc["sslcacert"]
+        if "sslclientkey" in desc:
+            repo.sslclientkey = desc["sslclientkey"]
+        if "sslclientcert" in desc:
+            repo.sslclientcert = desc["sslclientcert"]
+
+        # In dnf, the default metadata expiration time is 48 hours. However,
+        # some repositories never expire the metadata, and others expire it much
+        # sooner than that. We therefore allow this to be configured. If nothing
+        # is provided we error on the side of checking if we should invalidate
+        # the cache. If cache invalidation is not necessary, the overhead of
+        # checking is in the hundreds of milliseconds. In order to avoid this
+        # overhead accumulating for API calls that consist of several dnf calls,
+        # we set the expiration to a short time period, rather than 0.
+        repo.metadata_expire = desc.get("metadata_expire", "20s")
+
+        return repo
+
+    def _repo_checksums(self):
+        checksums = {}
+        for repo in self.base.repos.iter_enabled():
+            # Uses the same algorithm as libdnf to find cache dir:
+            #   https://github.com/rpm-software-management/libdnf/blob/master/libdnf/repo/Repo.cpp#L1288
+            if repo.metalink:
+                url = repo.metalink
+            elif repo.mirrorlist:
+                url = repo.mirrorlist
+            elif repo.baseurl:
+                url = repo.baseurl[0]
+            else:
+                assert False
+
+            digest = hashlib.sha256(url.encode()).hexdigest()[:16]
+
+            repomd_file = f"{repo.id}-{digest}/repodata/repomd.xml"
+            with open(f"{self.base.conf.cachedir}/{repomd_file}", "rb") as f:
+                repomd = f.read()
+
+            checksums[repo.id] = "sha256:" + hashlib.sha256(repomd).hexdigest()
+
+        return checksums
+
+    def _timestamp_to_rfc3339(self, timestamp):
+        d = datetime.datetime.utcfromtimestamp(timestamp)
+        return d.strftime('%Y-%m-%dT%H:%M:%SZ')
+
+    def dump(self):
         packages = []
-        for package in base.sack.query().available():
+        for package in self.base.sack.query().available():
             packages.append({
                 "name": package.name,
                 "summary": package.summary,
@@ -159,41 +142,19 @@ with tempfile.TemporaryDirectory() as persistdir:
                 "version": package.version,
                 "release": package.release,
                 "arch": package.arch,
-                "buildtime": timestamp_to_rfc3339(package.buildtime),
+                "buildtime": self._timestamp_to_rfc3339(package.buildtime),
                 "license": package.license
             })
-        json.dump({
-            "checksums": repo_checksums(base),
+        return {
+            "checksums": self._repo_checksums(),
             "packages": packages
-        }, sys.stdout)
-
-    elif command == "depsolve":
-        errors = []
-
-        try:
-            base.install_specs(
-                arguments["package-specs"],
-                exclude=arguments.get("exclude-specs", [])
-            )
-        except dnf.exceptions.MarkingErrors as e:
-            exit_with_dnf_error(
-                "MarkingErrors",
-                f"Error occurred when marking packages for installation: {e}"
-            )
-
-        try:
-            base.resolve()
-        except dnf.exceptions.DepsolveError as e:
-            exit_with_dnf_error(
-                "DepsolveError",
-                (
-                    "There was a problem depsolving "
-                    f"{arguments['package-specs']}: {e}"
-                )
-            )
+        }
 
+    def depsolve(self, package_spec, exclude_spec):
+        self.base.install_specs(package_spec, exclude_spec)
+        self.base.resolve()
         dependencies = []
-        for tsi in base.transaction:
+        for tsi in self.base.transaction:
             # Avoid using the install_set() helper, as it does not guarantee
             # a stable order
             if tsi.action not in dnf.transaction.FORWARD_ACTIONS:
@@ -214,7 +175,101 @@ with tempfile.TemporaryDirectory() as persistdir:
                     f"{package.chksum[1].hex()}"
                 )
             })
-        json.dump({
-            "checksums": repo_checksums(base),
+        return {
+            "checksums": self._repo_checksums(),
             "dependencies": dependencies
-        }, sys.stdout)
+        }
+
+class DnfJsonRequestHandler(BaseHTTPRequestHandler):
+
+    def _send(self):
+        self.client_address=('',)
+
+    def response_with_dnf_error(self, kind: str, reason: str):
+        self._send()
+        self.send_response(500)
+        self.send_header("Content-Type", "application/json")
+        self.end_headers()
+        self.wfile.write(json.dumps({"kind": kind, "reason":
+            reason}).encode("utf-8"))
+
+    def response_success(self, json_object):
+        self._send()
+        self.send_response(200)
+        self.send_header("Content-Type", "application/json")
+        self.end_headers()
+        self.wfile.write(json.dumps(json_object).encode("utf-8"))
+
+    def do_POST(self):
+        content_len = int(self.headers.get('Content-Length'))
+        data = self.rfile.read(content_len)
+        call = json.loads(data.decode("utf-8"))
+        command = call["command"]
+        arguments = call["arguments"]
+        repos = arguments.get("repos", {})
+        arch = arguments["arch"]
+        cachedir = arguments["cachedir"]
+        module_platform_id = arguments["module_platform_id"]
+
+        with tempfile.TemporaryDirectory() as persistdir:
+            try:
+                solver = Solver(
+                    repos,
+                    module_platform_id,
+                    persistdir,
+                    cachedir,
+                    arch
+                )
+                if command == "dump":
+                    self.response_success(solver.dump())
+                    log.info("dump success")
+                elif command == "depsolve":
+                    self.response_success(
+                            solver.depsolve(
+                                arguments["package-specs"],
+                                arguments.get("exclude-specs", [])
+                                )
+                            )
+                    log.info("depsolve success")
+
+            except dnf.exceptions.MarkingErrors as e:
+                log.info("error install_specs")
+                self.response_with_dnf_error(
+                    "MarkingErrors",
+                    f"Error occurred when marking packages for installation: {e}"
+                )
+            except dnf.exceptions.DepsolveError as e:
+                log.info("error depsolve")
+                self.response_with_dnf_error(
+                    "DepsolveError",
+                    (
+                        "There was a problem depsolving "
+                        f"{arguments['package-specs']}: {e}"
+                    )
+                )
+            except dnf.exceptions.Error as e:
+                self.response_with_dnf_error(
+                    type(e).__name__,
+                    f"Error occurred when setting up repo: {e}")
+
+log.info("Starting the dnf-json server")
+
+LISTEN_FDS = int(os.environ.get("LISTEN_FDS", 0))
+SOCK_PATH = "/run/osbuild-dnf-json/"
+SOCK_NAME = "api.sock"
+
+class SystemDActivationSocketServer(socketserver.ForkingMixIn, socketserver.UnixStreamServer):
+    def server_bind(self):
+        log.debug("service bind")
+        if LISTEN_FDS == 0:
+            log.debug("create new socket")
+            socketserver.UnixStreamServer.server_bind(self)
+        else:
+            log.debug("rebind socket")
+            log.debug("address_family: %d ", self.address_family)
+            log.debug("socket_type: %d ", self.socket_type)
+            self.socket = socket.fromfd(3, self.address_family, self.socket_type)
+
+pathlib.Path(SOCK_PATH).mkdir(parents=True, exist_ok=True)
+server = SystemDActivationSocketServer(f"{SOCK_PATH}{SOCK_NAME}", DnfJsonRequestHandler)
+server.serve_forever()
diff --git a/internal/rpmmd/repository.go b/internal/rpmmd/repository.go
index a338c01b3..1f87ac431 100644
--- a/internal/rpmmd/repository.go
+++ b/internal/rpmmd/repository.go
@@ -1,12 +1,15 @@
 package rpmmd
 
 import (
+	"bytes"
+	"context"
 	"encoding/json"
 	"fmt"
 	"io/ioutil"
 	"log"
+	"net"
+	"net/http"
 	"os"
-	"os/exec"
 	"path/filepath"
 	"sort"
 	"strconv"
@@ -313,7 +316,7 @@ func LoadRepositories(confPaths []string, distro string) (map[string][]RepoConfi
 	return repoConfigs, nil
 }
 
-func runDNF(dnfJsonPath string, command string, arguments interface{}, result interface{}) error {
+func runDNF(command string, arguments interface{}, result interface{}) error {
 	var call = struct {
 		Command   string      `json:"command"`
 		Arguments interface{} `json:"arguments,omitempty"`
@@ -322,39 +325,31 @@ func runDNF(dnfJsonPath string, command string, arguments interface{}, result in
 		arguments,
 	}
 
-	cmd := exec.Command(dnfJsonPath)
-
-	stdin, err := cmd.StdinPipe()
-	if err != nil {
-		return err
-	}
-
-	cmd.Stderr = os.Stderr
-	stdout, err := cmd.StdoutPipe()
-	if err != nil {
-		return err
+	httpc := http.Client{
+		Transport: &http.Transport{
+			DialContext: func(_ context.Context, _, _ string) (net.Conn, error) {
+				return net.Dial("unix", "/run/osbuild-dnf-json/api.sock")
+			},
+		},
 	}
 
-	err = cmd.Start()
+	bpost, err := json.Marshal(call)
 	if err != nil {
 		return err
 	}
 
-	err = json.NewEncoder(stdin).Encode(call)
+	response, err := httpc.Post("http://unix", "application/json", bytes.NewReader(bpost))
 	if err != nil {
 		return err
 	}
-	stdin.Close()
+	defer response.Body.Close()
 
-	output, err := ioutil.ReadAll(stdout)
+	output, err := ioutil.ReadAll(response.Body)
 	if err != nil {
 		return err
 	}
 
-	err = cmd.Wait()
-
-	const DnfErrorExitCode = 10
-	if runError, ok := err.(*exec.ExitError); ok && runError.ExitCode() == DnfErrorExitCode {
+	if response.StatusCode != http.StatusOK {
 		var dnfError DNFError
 		err = json.Unmarshal(output, &dnfError)
 		if err != nil {
@@ -376,10 +371,9 @@ func runDNF(dnfJsonPath string, command string, arguments interface{}, result in
 type rpmmdImpl struct {
 	CacheDir      string
 	subscriptions *rhsm.Subscriptions
-	dnfJsonPath   string
 }
 
-func NewRPMMD(cacheDir, dnfJsonPath string) RPMMD {
+func NewRPMMD(cacheDir string) RPMMD {
 	subscriptions, err := rhsm.LoadSystemSubscriptions()
 	if err != nil {
 		log.Println("Failed to load subscriptions. osbuild-composer will fail to build images if the "+
@@ -391,7 +385,6 @@ func NewRPMMD(cacheDir, dnfJsonPath string) RPMMD {
 	return &rpmmdImpl{
 		CacheDir:      cacheDir,
 		subscriptions: subscriptions,
-		dnfJsonPath:   dnfJsonPath,
 	}
 }
 
@@ -442,7 +435,7 @@ func (r *rpmmdImpl) FetchMetadata(repos []RepoConfig, modulePlatformID, arch, re
 		Packages  PackageList       `json:"packages"`
 	}
 
-	err := runDNF(r.dnfJsonPath, "dump", arguments, &reply)
+	err := runDNF("dump", arguments, &reply)
 
 	sort.Slice(reply.Packages, func(i, j int) bool {
 		return reply.Packages[i].Name < reply.Packages[j].Name
@@ -477,7 +470,7 @@ func (r *rpmmdImpl) Depsolve(packageSet PackageSet, repos []RepoConfig, modulePl
 		Checksums    map[string]string `json:"checksums"`
 		Dependencies []dnfPackageSpec  `json:"dependencies"`
 	}
-	err := runDNF(r.dnfJsonPath, "depsolve", arguments, &reply)
+	err := runDNF("depsolve", arguments, &reply)
 
 	dependencies := make([]PackageSpec, len(reply.Dependencies))
 	for i, pack := range reply.Dependencies {
diff --git a/osbuild-composer.spec b/osbuild-composer.spec
index f34acce21..9c8dbc026 100644
--- a/osbuild-composer.spec
+++ b/osbuild-composer.spec
@@ -167,34 +167,34 @@ go build -tags=integration -ldflags="${TEST_LDFLAGS}" -o _bin/osbuild-mock-openi
 %endif
 
 %install
-install -m 0755 -vd                                             %{buildroot}%{_libexecdir}/osbuild-composer
-install -m 0755 -vp _bin/osbuild-composer                       %{buildroot}%{_libexecdir}/osbuild-composer/
-install -m 0755 -vp _bin/osbuild-worker                         %{buildroot}%{_libexecdir}/osbuild-composer/
-install -m 0755 -vp dnf-json                                    %{buildroot}%{_libexecdir}/osbuild-composer/
+install -m 0755 -vd                                                %{buildroot}%{_libexecdir}/osbuild-composer
+install -m 0755 -vp _bin/osbuild-composer                          %{buildroot}%{_libexecdir}/osbuild-composer/
+install -m 0755 -vp _bin/osbuild-worker                            %{buildroot}%{_libexecdir}/osbuild-composer/
+install -m 0755 -vp dnf-json                                       %{buildroot}%{_libexecdir}/osbuild-composer/
 
 # Only include repositories for the distribution and release
-install -m 0755 -vd                                             %{buildroot}%{_datadir}/osbuild-composer/repositories
+install -m 0755 -vd                                                %{buildroot}%{_datadir}/osbuild-composer/repositories
 # CentOS also defines rhel so we check for centos first
 %if 0%{?centos}
 
 # CentOS 9 supports building for CentOS 8 and later
 %if 0%{?centos} >= 9
-install -m 0644 -vp repositories/centos-*                       %{buildroot}%{_datadir}/osbuild-composer/repositories/
+install -m 0644 -vp repositories/centos-*                          %{buildroot}%{_datadir}/osbuild-composer/repositories/
 %else
 # CentOS 8 only supports building for CentOS 8
-install -m 0644 -vp repositories/centos-%{centos}*              %{buildroot}%{_datadir}/osbuild-composer/repositories/
-install -m 0644 -vp repositories/centos-stream-%{centos}*       %{buildroot}%{_datadir}/osbuild-composer/repositories/
+install -m 0644 -vp repositories/centos-%{centos}*                 %{buildroot}%{_datadir}/osbuild-composer/repositories/
+install -m 0644 -vp repositories/centos-stream-%{centos}*          %{buildroot}%{_datadir}/osbuild-composer/repositories/
 
 %endif
 %else
 %if 0%{?rhel}
 # RHEL 9 supports building for RHEL 8 and later
 %if 0%{?rhel} >= 9
-install -m 0644 -vp repositories/rhel-*                         %{buildroot}%{_datadir}/osbuild-composer/repositories/
+install -m 0644 -vp repositories/rhel-*                            %{buildroot}%{_datadir}/osbuild-composer/repositories/
 
 %else
 # RHEL 8 only supports building for 8
-install -m 0644 -vp repositories/rhel-%{rhel}*                  %{buildroot}%{_datadir}/osbuild-composer/repositories/
+install -m 0644 -vp repositories/rhel-%{rhel}*                     %{buildroot}%{_datadir}/osbuild-composer/repositories/
 
 %endif
 %endif
@@ -202,84 +202,85 @@ install -m 0644 -vp repositories/rhel-%{rhel}*                  %{buildroot}%{_d
 
 # Fedora can build for all included fedora releases
 %if 0%{?fedora}
-install -m 0644 -vp repositories/fedora-*                       %{buildroot}%{_datadir}/osbuild-composer/repositories/
+install -m 0644 -vp repositories/fedora-*                          %{buildroot}%{_datadir}/osbuild-composer/repositories/
 %endif
 
-install -m 0755 -vd                                             %{buildroot}%{_unitdir}
-install -m 0644 -vp distribution/*.{service,socket}             %{buildroot}%{_unitdir}/
+install -m 0755 -vd                                                %{buildroot}%{_unitdir}
+install -m 0644 -vp distribution/*.{service,socket}                %{buildroot}%{_unitdir}/
 
-install -m 0755 -vd                                             %{buildroot}%{_sysusersdir}
-install -m 0644 -vp distribution/osbuild-composer.conf          %{buildroot}%{_sysusersdir}/
+install -m 0755 -vd                                                %{buildroot}%{_sysusersdir}
+install -m 0644 -vp distribution/osbuild-composer.conf             %{buildroot}%{_sysusersdir}/
 
-install -m 0755 -vd                                             %{buildroot}%{_localstatedir}/cache/osbuild-composer/dnf-cache
+install -m 0755 -vd                                                %{buildroot}%{_localstatedir}/cache/osbuild-composer/dnf-cache
 
-install -m 0755 -vd                                             %{buildroot}%{_mandir}/man7
-install -m 0644 -vp docs/*.7                                    %{buildroot}%{_mandir}/man7/
+install -m 0755 -vd                                                %{buildroot}%{_mandir}/man7
+install -m 0644 -vp docs/*.7                                       %{buildroot}%{_mandir}/man7/
 
 %if %{with tests} || 0%{?rhel}
 
-install -m 0755 -vd                                             %{buildroot}%{_libexecdir}/osbuild-composer-test
-install -m 0755 -vp _bin/osbuild-composer-cli-tests             %{buildroot}%{_libexecdir}/osbuild-composer-test/
-install -m 0755 -vp _bin/osbuild-weldr-tests                    %{buildroot}%{_libexecdir}/osbuild-composer-test/
-install -m 0755 -vp _bin/osbuild-dnf-json-tests                 %{buildroot}%{_libexecdir}/osbuild-composer-test/
-install -m 0755 -vp _bin/osbuild-image-tests                    %{buildroot}%{_libexecdir}/osbuild-composer-test/
-install -m 0755 -vp _bin/osbuild-auth-tests                     %{buildroot}%{_libexecdir}/osbuild-composer-test/
-install -m 0755 -vp _bin/osbuild-koji-tests                     %{buildroot}%{_libexecdir}/osbuild-composer-test/
-install -m 0755 -vp _bin/osbuild-composer-dbjobqueue-tests      %{buildroot}%{_libexecdir}/osbuild-composer-test/
-install -m 0755 -vp _bin/osbuild-composer-manifest-tests        %{buildroot}%{_libexecdir}/osbuild-composer-test/
-install -m 0755 -vp _bin/cloud-cleaner                          %{buildroot}%{_libexecdir}/osbuild-composer-test/
-install -m 0755 -vp _bin/osbuild-mock-openid-provider           %{buildroot}%{_libexecdir}/osbuild-composer-test/
-install -m 0755 -vp tools/define-compose-url.sh                 %{buildroot}%{_libexecdir}/osbuild-composer-test/
-install -m 0755 -vp tools/provision.sh                          %{buildroot}%{_libexecdir}/osbuild-composer-test/
-install -m 0755 -vp tools/gen-certs.sh                          %{buildroot}%{_libexecdir}/osbuild-composer-test/
-install -m 0755 -vp tools/gen-ssh.sh                            %{buildroot}%{_libexecdir}/osbuild-composer-test/
-install -m 0755 -vp tools/image-info                            %{buildroot}%{_libexecdir}/osbuild-composer-test/
-install -m 0755 -vp tools/run-koji-container.sh                 %{buildroot}%{_libexecdir}/osbuild-composer-test/
-install -m 0755 -vp tools/koji-compose.py                       %{buildroot}%{_libexecdir}/osbuild-composer-test/
-install -m 0755 -vp tools/libvirt_test.sh                       %{buildroot}%{_libexecdir}/osbuild-composer-test/
-install -m 0755 -vp tools/set-env-variables.sh                  %{buildroot}%{_libexecdir}/osbuild-composer-test/
-install -m 0755 -vd                                             %{buildroot}%{_libexecdir}/tests/osbuild-composer
-install -m 0755 -vp test/cases/*                                %{buildroot}%{_libexecdir}/tests/osbuild-composer/
-
-install -m 0755 -vd                                             %{buildroot}%{_datadir}/tests/osbuild-composer/ansible
-install -m 0644 -vp test/data/ansible/*                         %{buildroot}%{_datadir}/tests/osbuild-composer/ansible/
-
-install -m 0755 -vd                                             %{buildroot}%{_datadir}/tests/osbuild-composer/azure
-install -m 0644 -vp test/data/azure/*                           %{buildroot}%{_datadir}/tests/osbuild-composer/azure/
-
-install -m 0755 -vd                                             %{buildroot}%{_datadir}/tests/osbuild-composer/manifests
-install -m 0644 -vp test/data/manifests/*                       %{buildroot}%{_datadir}/tests/osbuild-composer/manifests/
-
-install -m 0755 -vd                                             %{buildroot}%{_datadir}/tests/osbuild-composer/cloud-init
-install -m 0644 -vp test/data/cloud-init/*                      %{buildroot}%{_datadir}/tests/osbuild-composer/cloud-init/
-
-install -m 0755 -vd                                             %{buildroot}%{_datadir}/tests/osbuild-composer/composer
-install -m 0644 -vp test/data/composer/*                        %{buildroot}%{_datadir}/tests/osbuild-composer/composer/
-
-install -m 0755 -vd                                             %{buildroot}%{_datadir}/tests/osbuild-composer/worker
-install -m 0644 -vp test/data/worker/*                          %{buildroot}%{_datadir}/tests/osbuild-composer/worker/
-
-install -m 0755 -vd                                             %{buildroot}%{_datadir}/tests/osbuild-composer/repositories
-install -m 0644 -vp test/data/repositories/*                    %{buildroot}%{_datadir}/tests/osbuild-composer/repositories/
-
-install -m 0755 -vd                                             %{buildroot}%{_datadir}/tests/osbuild-composer/kerberos
-install -m 0644 -vp test/data/kerberos/*                        %{buildroot}%{_datadir}/tests/osbuild-composer/kerberos/
-
-install -m 0755 -vd                                             %{buildroot}%{_datadir}/tests/osbuild-composer/keyring
-install -m 0644 -vp test/data/keyring/id_rsa.pub                %{buildroot}%{_datadir}/tests/osbuild-composer/keyring/
-install -m 0600 -vp test/data/keyring/id_rsa                    %{buildroot}%{_datadir}/tests/osbuild-composer/keyring/
-
-install -m 0755 -vd                                             %{buildroot}%{_datadir}/tests/osbuild-composer/koji
-install -m 0644 -vp test/data/koji/*                            %{buildroot}%{_datadir}/tests/osbuild-composer/koji/
-
-install -m 0755 -vd                                             %{buildroot}%{_datadir}/tests/osbuild-composer/x509
-install -m 0644 -vp test/data/x509/*                            %{buildroot}%{_datadir}/tests/osbuild-composer/x509/
-
-install -m 0755 -vd                                             %{buildroot}%{_datadir}/tests/osbuild-composer/openshift
-install -m 0644 -vp test/data/openshift/*                       %{buildroot}%{_datadir}/tests/osbuild-composer/openshift/
-
-install -m 0755 -vd                                             %{buildroot}%{_datadir}/tests/osbuild-composer/schemas
-install -m 0644 -vp internal/jobqueue/dbjobqueue/schemas/*      %{buildroot}%{_datadir}/tests/osbuild-composer/schemas/
+install -m 0755 -vd                                                %{buildroot}%{_libexecdir}/osbuild-composer-test
+install -m 0755 -vp _bin/osbuild-composer-cli-tests                %{buildroot}%{_libexecdir}/osbuild-composer-test/
+install -m 0755 -vp _bin/osbuild-weldr-tests                       %{buildroot}%{_libexecdir}/osbuild-composer-test/
+install -m 0755 -vp _bin/osbuild-dnf-json-tests                    %{buildroot}%{_libexecdir}/osbuild-composer-test/
+install -m 0755 -vp _bin/osbuild-image-tests                       %{buildroot}%{_libexecdir}/osbuild-composer-test/
+install -m 0755 -vp _bin/osbuild-auth-tests                        %{buildroot}%{_libexecdir}/osbuild-composer-test/
+install -m 0755 -vp _bin/osbuild-koji-tests                        %{buildroot}%{_libexecdir}/osbuild-composer-test/
+install -m 0755 -vp _bin/osbuild-composer-dbjobqueue-tests         %{buildroot}%{_libexecdir}/osbuild-composer-test/
+install -m 0755 -vp _bin/osbuild-composer-manifest-tests           %{buildroot}%{_libexecdir}/osbuild-composer-test/
+install -m 0755 -vp _bin/cloud-cleaner                             %{buildroot}%{_libexecdir}/osbuild-composer-test/
+install -m 0755 -vp _bin/osbuild-mock-openid-provider              %{buildroot}%{_libexecdir}/osbuild-composer-test/
+install -m 0755 -vp tools/define-compose-url.sh                    %{buildroot}%{_libexecdir}/osbuild-composer-test/
+install -m 0755 -vp tools/provision.sh                             %{buildroot}%{_libexecdir}/osbuild-composer-test/
+install -m 0755 -vp tools/gen-certs.sh                             %{buildroot}%{_libexecdir}/osbuild-composer-test/
+install -m 0755 -vp tools/gen-ssh.sh                               %{buildroot}%{_libexecdir}/osbuild-composer-test/
+install -m 0755 -vp tools/image-info                               %{buildroot}%{_libexecdir}/osbuild-composer-test/
+install -m 0755 -vp tools/run-koji-container.sh                    %{buildroot}%{_libexecdir}/osbuild-composer-test/
+install -m 0755 -vp tools/koji-compose.py                          %{buildroot}%{_libexecdir}/osbuild-composer-test/
+install -m 0755 -vp tools/libvirt_test.sh                          %{buildroot}%{_libexecdir}/osbuild-composer-test/
+install -m 0755 -vp tools/set-env-variables.sh                     %{buildroot}%{_libexecdir}/osbuild-composer-test/
+install -m 0755 -vp tools/test-case-generators/generate-test-cases %{buildroot}%{_libexecdir}/osbuild-composer-test/
+install -m 0755 -vd                                                %{buildroot}%{_libexecdir}/tests/osbuild-composer
+install -m 0755 -vp test/cases/*                                   %{buildroot}%{_libexecdir}/tests/osbuild-composer/
+
+install -m 0755 -vd                                                %{buildroot}%{_datadir}/tests/osbuild-composer/ansible
+install -m 0644 -vp test/data/ansible/*                            %{buildroot}%{_datadir}/tests/osbuild-composer/ansible/
+
+install -m 0755 -vd                                                %{buildroot}%{_datadir}/tests/osbuild-composer/azure
+install -m 0644 -vp test/data/azure/*                              %{buildroot}%{_datadir}/tests/osbuild-composer/azure/
+
+install -m 0755 -vd                                                %{buildroot}%{_datadir}/tests/osbuild-composer/manifests
+install -m 0644 -vp test/data/manifests/*                          %{buildroot}%{_datadir}/tests/osbuild-composer/manifests/
+
+install -m 0755 -vd                                                %{buildroot}%{_datadir}/tests/osbuild-composer/cloud-init
+install -m 0644 -vp test/data/cloud-init/*                         %{buildroot}%{_datadir}/tests/osbuild-composer/cloud-init/
+
+install -m 0755 -vd                                                %{buildroot}%{_datadir}/tests/osbuild-composer/composer
+install -m 0644 -vp test/data/composer/*                           %{buildroot}%{_datadir}/tests/osbuild-composer/composer/
+
+install -m 0755 -vd                                                %{buildroot}%{_datadir}/tests/osbuild-composer/worker
+install -m 0644 -vp test/data/worker/*                             %{buildroot}%{_datadir}/tests/osbuild-composer/worker/
+
+install -m 0755 -vd                                                %{buildroot}%{_datadir}/tests/osbuild-composer/repositories
+install -m 0644 -vp test/data/repositories/*                       %{buildroot}%{_datadir}/tests/osbuild-composer/repositories/
+
+install -m 0755 -vd                                                %{buildroot}%{_datadir}/tests/osbuild-composer/kerberos
+install -m 0644 -vp test/data/kerberos/*                           %{buildroot}%{_datadir}/tests/osbuild-composer/kerberos/
+
+install -m 0755 -vd                                                %{buildroot}%{_datadir}/tests/osbuild-composer/keyring
+install -m 0644 -vp test/data/keyring/id_rsa.pub                   %{buildroot}%{_datadir}/tests/osbuild-composer/keyring/
+install -m 0600 -vp test/data/keyring/id_rsa                       %{buildroot}%{_datadir}/tests/osbuild-composer/keyring/
+
+install -m 0755 -vd                                                %{buildroot}%{_datadir}/tests/osbuild-composer/koji
+install -m 0644 -vp test/data/koji/*                               %{buildroot}%{_datadir}/tests/osbuild-composer/koji/
+
+install -m 0755 -vd                                                %{buildroot}%{_datadir}/tests/osbuild-composer/x509
+install -m 0644 -vp test/data/x509/*                               %{buildroot}%{_datadir}/tests/osbuild-composer/x509/
+
+install -m 0755 -vd                                                %{buildroot}%{_datadir}/tests/osbuild-composer/openshift
+install -m 0644 -vp test/data/openshift/*                          %{buildroot}%{_datadir}/tests/osbuild-composer/openshift/
+
+install -m 0755 -vd                                                %{buildroot}%{_datadir}/tests/osbuild-composer/schemas
+install -m 0644 -vp internal/jobqueue/dbjobqueue/schemas/*         %{buildroot}%{_datadir}/tests/osbuild-composer/schemas/
 
 %endif
 
@@ -375,6 +376,8 @@ The dnf-json binary used by osbuild-composer and the workers.
 
 %files dnf-json
 %{_libexecdir}/osbuild-composer/dnf-json
+%{_unitdir}/osbuild-dnf-json.service
+%{_unitdir}/osbuild-dnf-json.socket
 
 %if %{with tests} || 0%{?rhel}
 
diff --git a/schutzbot/scheduled_cloud_cleaner.sh b/schutzbot/scheduled_cloud_cleaner.sh
index 71306bb3b..2cf440def 100755
--- a/schutzbot/scheduled_cloud_cleaner.sh
+++ b/schutzbot/scheduled_cloud_cleaner.sh
@@ -5,7 +5,12 @@ function greenprint {
     echo -e "\033[1;32m[$(date -Isecond)] ${1}\033[0m"
 }
 
-# Azure cleanup
+#---------------------------------------------------------------
+#                       Azure cleanup
+#---------------------------------------------------------------
+
+greenprint "Starting azure cleanup"
+
 if ! hash az; then
     # this installation method is taken from the official docs:
     # https://docs.microsoft.com/cs-cz/cli/azure/install-azure-cli-linux?pivots=dnf
@@ -211,3 +216,63 @@ for object in ${OBJECTS}; do
         fi
 done
 
+#---------------------------------------------------------------
+#                       GCP cleanup
+#---------------------------------------------------------------
+
+greenprint "starting gcp cleanup"
+
+# We need Google Gloud SDK to comunicate with gcp
+if ! hash gcloud; then
+  echo "Using 'gcloud' from a container"
+  sudo ${CONTAINER_RUNTIME} pull ${CONTAINER_IMAGE_CLOUD_TOOLS}
+
+  # directory mounted to the container, in which gcloud stores the credentials after logging in
+  GCP_CMD_CREDS_DIR="${TEMPDIR}/gcloud_credentials"
+  mkdir "${GCP_CMD_CREDS_DIR}"
+
+  GCP_CMD="sudo ${CONTAINER_RUNTIME} run --rm \
+    -v ${GCP_CMD_CREDS_DIR}:/root/.config/gcloud:Z \
+    -v ${GOOGLE_APPLICATION_CREDENTIALS}:${GOOGLE_APPLICATION_CREDENTIALS}:Z \
+    -v ${TEMPDIR}:${TEMPDIR}:Z \
+    ${CONTAINER_IMAGE_CLOUD_TOOLS} gcloud --format=json"
+else
+  echo "Using pre-installed 'gcloud' from the system"
+  GCP_CMD="gcloud --format=json --quiet"
+fi
+$GCP_CMD --version
+
+# Authenticate
+$GCP_CMD auth activate-service-account --key-file "$GOOGLE_APPLICATION_CREDENTIALS"
+# Extract and set the default project to be used for commands
+GCP_PROJECT=$(jq -r '.project_id' "$GOOGLE_APPLICATION_CREDENTIALS")
+$GCP_CMD config set project "$GCP_PROJECT"
+
+# List tagged intances and remove the old enough ones
+INSTANCES=$($GCP_CMD compute instances list --filter='labels.gitlab-ci-test:true' \
+	| jq -c '.[] | {"name": .name, "creationTimestamp": .creationTimestamp, "zone": .zone}')
+
+for instance in ${INSTANCES}; do                
+	CREATION_TIME=$(echo "${instance}" | jq '.creationTimestamp' | tr -d '"')
+
+        if [[ $(date -d "${CREATION_TIME}" +%s) -lt ${DELETE_TIME} ]]; then
+                ZONE=$(echo "${instance}" | jq '.zone' | awk -F / '{print $NF}' | tr -d '"')
+                NAME=$(echo "${instance}" | jq '.name' | tr -d '"')
+                $GCP_CMD compute instances delete --zone="$ZONE" "$NAME"
+                echo "deleted instance: ${NAME}"
+        fi
+done
+
+# List tagged images and remove the old enough ones
+IMAGES=$($GCP_CMD compute images list --filter='labels.gitlab-ci-test:true' \
+	| jq -c '.[] | {"name": .name, "creationTimestamp": .creationTimestamp}')
+
+for image in $IMAGES; do
+        CREATION_TIME=$(echo "${image}" | jq '.creationTimestamp' | tr -d '"')
+
+        if [[ $(date -d "${CREATION_TIME}" +%s) -lt ${DELETE_TIME} ]]; then
+                NAME=$(echo "${image}" | jq '.name' | tr -d '"')
+                $GCP_CMD compute images delete "$NAME"
+                echo "deleted image: ${NAME}"        
+	fi
+done


Test patch:
diff --git a/cmd/osbuild-dnf-json-tests/main_test.go b/cmd/osbuild-dnf-json-tests/main_test.go
index 1b74dcb0d..003c8d6f0 100644
--- a/cmd/osbuild-dnf-json-tests/main_test.go
+++ b/cmd/osbuild-dnf-json-tests/main_test.go
@@ -5,8 +5,8 @@
 package main
 
 import (
-	"fmt"
 	"io/ioutil"
+	"net/http"
 	"os"
 	"path"
 	"testing"
@@ -23,6 +23,11 @@ import (
 
 func TestFetchChecksum(t *testing.T) {
 	dir, err := test.SetUpTemporaryRepository()
+	fs := http.FileServer(http.Dir(dir))
+	go func() {
+		err := http.ListenAndServe(":9000", fs)
+		assert.Nilf(t, err, "Could not start the http server: %v", err)
+	}()
 	defer func(dir string) {
 		err := test.TearDownTemporaryRepository(dir)
 		assert.Nil(t, err, "Failed to clean up temporary repository.")
@@ -31,13 +36,13 @@ func TestFetchChecksum(t *testing.T) {
 
 	repoCfg := rpmmd.RepoConfig{
 		Name:      "repo",
-		BaseURL:   fmt.Sprintf("file://%s", dir),
+		BaseURL:   "http://localhost:9000",
 		IgnoreSSL: true,
 	}
 
 	// use a fullpath to dnf-json, this allows this test to have an arbitrary
 	// working directory
-	rpmMetadata := rpmmd.NewRPMMD(path.Join(dir, "rpmmd"), "/usr/libexec/osbuild-composer/dnf-json")
+	rpmMetadata := rpmmd.NewRPMMD(path.Join(dir, "rpmmd"))
 	_, c, err := rpmMetadata.FetchMetadata([]rpmmd.RepoConfig{repoCfg}, "platform:f31", "x86_64", "31")
 	assert.Nilf(t, err, "Failed to fetch checksum: %v", err)
 	assert.NotEqual(t, "", c["repo"], "The checksum is empty")
@@ -64,7 +69,7 @@ func TestCrossArchDepsolve(t *testing.T) {
 
 			// use a fullpath to dnf-json, this allows this test to have an arbitrary
 			// working directory
-			rpm := rpmmd.NewRPMMD(dir, "/usr/libexec/osbuild-composer/dnf-json")
+			rpm := rpmmd.NewRPMMD(dir)
 
 			repos, err := rpmmd.LoadRepositories([]string{repoDir}, distroStruct.Name())
 			require.NoErrorf(t, err, "Failed to LoadRepositories %v", distroStruct.Name())
diff --git a/internal/distro/distro_test_common/distro_test_common.go b/internal/distro/distro_test_common/distro_test_common.go
index 6a1437be4..457b0aac9 100644
--- a/internal/distro/distro_test_common/distro_test_common.go
+++ b/internal/distro/distro_test_common/distro_test_common.go
@@ -132,7 +132,7 @@ func TestDistro_Manifest(t *testing.T, pipelinePath string, prefix string, regis
 func getImageTypePkgSpecSets(imageType distro.ImageType, bp blueprint.Blueprint, repos []rpmmd.RepoConfig, cacheDir, dnfJsonPath string) map[string][]rpmmd.PackageSpec {
 	imgPackageSets := imageType.PackageSets(bp)
 
-	rpm_md := rpmmd.NewRPMMD(cacheDir, dnfJsonPath)
+	rpm_md := rpmmd.NewRPMMD(cacheDir)
 
 	imgPackageSpecSets := make(map[string][]rpmmd.PackageSpec)
 	for name, packages := range imgPackageSets {
diff --git a/test/cases/api.sh b/test/cases/api.sh
index 07bf99aab..69d479596 100755
--- a/test/cases/api.sh
+++ b/test/cases/api.sh
@@ -1038,6 +1038,9 @@ function verifyInGCP() {
   GCP_PROJECT=$(jq -r '.project_id' "$GOOGLE_APPLICATION_CREDENTIALS")
   $GCP_CMD config set project "$GCP_PROJECT"
 
+  # Add "gitlab-ci-test" label to the image
+  $GCP_CMD compute images add-labels "$GCP_IMAGE_NAME" --labels=gitlab-ci-test=true
+
   # Verify that the image was shared
   SHARE_OK=1
   $GCP_CMD compute images get-iam-policy "$GCP_IMAGE_NAME" > "$WORKDIR/image-iam-policy.json"
@@ -1074,6 +1077,7 @@ function verifyInGCP() {
     --zone="$GCP_ZONE" \
     --image-project="$GCP_PROJECT" \
     --image="$GCP_IMAGE_NAME" \
+    --labels=gitlab-ci-test=true \
     --metadata-from-file=ssh-keys="$GCP_SSH_METADATA_FILE"
   HOST=$($GCP_CMD compute instances describe "$GCP_INSTANCE_NAME" --zone="$GCP_ZONE" --format='get(networkInterfaces[0].accessConfigs[0].natIP)')
 
diff --git a/test/cases/generation.sh b/test/cases/generation.sh
new file mode 100755
index 000000000..380536c6f
--- /dev/null
+++ b/test/cases/generation.sh
@@ -0,0 +1,44 @@
+#!/usr/bin/bash
+set -euxo pipefail
+
+# Provision the software under test.
+/usr/libexec/osbuild-composer-test/provision.sh
+
+# Get OS data
+source /usr/libexec/osbuild-composer-test/set-env-variables.sh
+
+# Colorful timestamped output.
+function greenprint {
+    echo -e "\033[1;32m[$(date -Isecond)] ${1}\033[0m"
+}
+
+# install requirements
+sudo dnf -q -y install make \
+    git \
+    go \
+    ranger \
+    openssl \
+    rpm-build \
+    aws \
+    qemu \
+    setools-console \
+    setroubleshoot \
+    container-selinux \
+    lvm2
+
+# stop dnf-json socket
+sudo systemctl stop osbuild-dnf-json.socket
+
+# test the test cases generation
+WORKDIR=$(mktemp -d -p /var/tmp)
+
+OSBUILD_LABEL=$(matchpathcon -n "$(which osbuild)")
+chcon "$OSBUILD_LABEL" tools/image-info
+
+# test the test case generation when dnf-json socket is stopped
+sudo ./tools/test-case-generators/generate-test-cases\
+    --output test/data/manifests\
+    --arch x86_64\
+    --distro rhel-8\
+    --image-type qcow2\
+    --store "$WORKDIR"
diff --git a/test/cases/regression-old-worker-new-composer.sh b/test/cases/regression-old-worker-new-composer.sh
index c01f1acf9..486534257 100644
--- a/test/cases/regression-old-worker-new-composer.sh
+++ b/test/cases/regression-old-worker-new-composer.sh
@@ -68,7 +68,7 @@ sudo podman run  \
      -v "$WELDR_DIR:/run/weldr/":Z \
      -p 8700:8700 \
      "quay.io/osbuild/osbuild-composer-ubi-pr:${CI_COMMIT_SHA}" \
-     --weldr-api --remote-worker-api \
+     --weldr-api --dnf-json --remote-worker-api \
      --no-local-worker-api --no-composer-api
 
 # try starting a worker
diff --git a/tools/test-case-generators/generate-test-cases b/tools/test-case-generators/generate-test-cases
index 8a55d74ab..ec6440ef0 100755
--- a/tools/test-case-generators/generate-test-cases
+++ b/tools/test-case-generators/generate-test-cases
@@ -7,6 +7,9 @@ import os
 import sys
 import tempfile
 
+def is_subprocess_succeeding(*args, **kwargs):
+    sp = subprocess.run(*args, **kwargs, stdout=subprocess.PIPE)
+    return sp.returncode == 0
 
 def get_subprocess_stdout(*args, **kwargs):
     sp = subprocess.run(*args, **kwargs, stdout=subprocess.PIPE)
@@ -63,6 +66,14 @@ class TestCaseGenerator:
     def get_test_case(self, no_image_info, store):
         compose_request = json.dumps(self.test_case["compose-request"])
 
+        command = ["systemctl", "is-active", "--quiet", "osbuild-dnf-json.socket"]
+        if not is_subprocess_succeeding(command):
+            print("osbuild-dnf-json.socket is not running, starting it up")
+            command = ["systemctl", "start", "osbuild-dnf-json.socket", "--quiet"]
+            if not is_subprocess_succeeding(command):
+                print("impossible to start osbuild-dnf-json.socket with systemd")
+                sys.exit(1)
+
         pipeline_command = ["go", "run", "./cmd/osbuild-pipeline", "-"]
         self.test_case["manifest"] = json.loads(get_subprocess_stdout(pipeline_command, input=compose_request, encoding="utf-8"))