Repository: osbuild/osbuild-composer
Pull request #2477 (2022-03-23 13:59:24)

Dependency joberrors
This pull request includes:
Reporting that there has been an issue with a dependency job does not provide enough information about the dependency
failure. It makes more sense to bubble up the original error since this will tell us the information we need. It also has the added benefit of ensuring consistency of the status codes for metrics.

- [ ] adequate testing for the new functionality or fixed issue
- [ ] adequate documentation informing people about the change such as
  - [ ] submit a PR for the [guides](https://github.com/osbuild/guides) repository if this PR changed any behavior described there: https://www.osbuild.org/guides/

<!--
Thanks for proposing a change to osbuild-composer!

Please don't remove the above check list. These are things that each pull
request must have before it is merged. It helps maintainers to not forget
anything.

If the reason for ticking any of the boxes is ambiguous, please add a short
note explaining why.

In addition, if this pull request fixes a downstream issue, please refer to
test/README.md and add these additional items:

- [ ] 1st commit of any `rhbz#` related PR contains bug reproducer; CI reports FAIL or
- [ ] PR contains automated tests for new functionality and
- [ ] QE has approved reproducer/new tests and
- [ ] Subsequent commits provide bug fixes without modifying the reproducer; CI reports PASS and
- [ ] QE approves this PR; RHBZ status is set to `MODIFIED + Verified=Tested`

Information regarding our GitLab pipeline (Schutzbot):

CI will not be ran automatically if WIP label is applied or the PR is in DRAFT state, instead only
a link will be provided to the pipeline which can then be triggered manually if desired. To run the
CI automatically either switch the PR to ready or apply WIP+test label.

Outside contributors need manual approval from one of the osbuild-composer maintainers.

Schutzbot will only be triggered if all Tests jobs in GitHub workflow succeed.
-->


ondrejbudai: I'm not sure if this approach is correct. Generally speaking, a job can have multiple dependencies and each of them can fail. With this approach, we would have to have an array of `JobError`s which sounds complicated. Also, we already have a job (`koji-finalize`) that does some stuff (and thus can fail) even though its dependency failed - so we have to communicate its status back somehow.

Since the api servers are the ones who are responsible for queueing jobs in the correct order, I think they should be ultimately responsible for inspecting the whole job chain and cherry-picking errors from it.
Gundersanne: > I'm not sure if this approach is correct. Generally speaking, a job can have multiple dependencies and each of them can fail. With this approach, we would have to have an array of `JobError`s which sounds complicated. Also, we already have a job (`koji-finalize`) that does some stuff (and thus can fail) even though its dependency failed - so we have to communicate its status back somehow.
> 
> Since the api servers are the ones who are responsible for queueing jobs in the correct order, I think they should be ultimately responsible for inspecting the whole job chain and cherry-picking errors from it.

Didn't consider the koji case in this. In that case is it fine to consider all dependency failure errors as 4xx in metrics? That way it'll only the dependency shows up (in case of 5xxs). 
kingsleyzissou: Okay, will have to re-think this a bit then, I also hadn't considered the koji side of things properly.

> Didn't consider the koji case in this. In that case is it fine to consider all dependency failure errors as 4xx in metrics? That way it'll only the dependency shows up (in case of 5xxs).

I guess technically if a dependency failed, it's bad input for the next job (my two-cents)
kingsleyzissou: @Gundersanne @ondrejbudai  I dropped the commit for propagating the dependency errors. I'll do that in a follow up.
For the meantime I changed the status codes for the metrics when we encounter a dependency error (assuming we're happy to go ahead with that change).

Patch:
diff --git a/.github/workflows/coverity_scan.yml b/.github/workflows/coverity_scan.yml
index e450ae5c1..81f2fb4a4 100644
--- a/.github/workflows/coverity_scan.yml
+++ b/.github/workflows/coverity_scan.yml
@@ -10,7 +10,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Clone repository
-        uses: actions/checkout@v2.4.0
+        uses: actions/checkout@v3
 
       # https://scan.coverity.com/projects/osbuild-osbuild-composer
       - name: Run coverity scan script
diff --git a/.github/workflows/trigger-gitlab.yml b/.github/workflows/trigger-gitlab.yml
index 78b37c6f2..ec4225fcf 100644
--- a/.github/workflows/trigger-gitlab.yml
+++ b/.github/workflows/trigger-gitlab.yml
@@ -22,7 +22,7 @@ jobs:
           sudo apt install -y jq
 
       - name: Clone repository
-        uses: actions/checkout@v2
+        uses: actions/checkout@v3
         with:
           ref: ${{ github.event.workflow_run.head_sha }}
           fetch-depth: 0
@@ -51,7 +51,7 @@ jobs:
           fi
 
       - name: Download artifacts
-        uses: actions/github-script@v5
+        uses: actions/github-script@v6
         with:
           script: |
             let allArtifacts = await github.rest.actions.listWorkflowRunArtifacts({
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 7ac41982e..db1cf45ec 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -303,7 +303,7 @@ OSTree:
           - rhos-01/rhel-8.6-nightly-x86_64-large
           - rhos-01/rhel-9.0-nightly-x86_64
           - rhos-01/centos-stream-8-x86_64
-          - rhos-01/centos-stream-8-x86_64
+          - rhos-01/centos-stream-9-x86_64
 
 New OSTree:
   stage: test
@@ -359,6 +359,7 @@ Rebase OSTree:
           - rhos-01/rhel-8.6-nightly-x86_64-large
           - rhos-01/rhel-9.0-nightly-x86_64-large
           - rhos-01/centos-stream-8-x86_64-large
+          - rhos-01/centos-stream-9-x86_64-large
 
 .integration:
   stage: test
diff --git a/cmd/osbuild-composer/composer.go b/cmd/osbuild-composer/composer.go
index 2083a5d62..b25991317 100644
--- a/cmd/osbuild-composer/composer.go
+++ b/cmd/osbuild-composer/composer.go
@@ -330,7 +330,9 @@ func (c *Composer) Start() error {
 	logrus.Info("Shutting down.")
 
 	if c.apiListener != nil {
+		// First, close all listeners and then wait for all goroutines to finish.
 		err := composerAPI.Shutdown(context.Background())
+		c.api.Shutdown()
 		if err != nil {
 			panic(err)
 		}
diff --git a/cmd/osbuild-koji/main.go b/cmd/osbuild-koji/main.go
index fb3c0728a..42da02c5d 100644
--- a/cmd/osbuild-koji/main.go
+++ b/cmd/osbuild-koji/main.go
@@ -8,7 +8,6 @@ import (
 	"time"
 
 	"github.com/google/uuid"
-	"github.com/hashicorp/go-retryablehttp"
 	"github.com/osbuild/osbuild-composer/internal/rpmmd"
 	"github.com/osbuild/osbuild-composer/internal/upload/koji"
 	"github.com/sirupsen/logrus"
@@ -42,7 +41,7 @@ func main() {
 	}
 	defer file.Close()
 
-	transport := &retryablehttp.RoundTripper{}
+	transport := koji.CreateRetryableTransport()
 	k, err := koji.NewFromPlain(server, "osbuild", "osbuildpass", transport)
 	if err != nil {
 		println(err.Error())
diff --git a/cmd/osbuild-worker/jobimpl-osbuild-koji.go b/cmd/osbuild-worker/jobimpl-osbuild-koji.go
index a6dcc51e4..0a656ac4e 100644
--- a/cmd/osbuild-worker/jobimpl-osbuild-koji.go
+++ b/cmd/osbuild-worker/jobimpl-osbuild-koji.go
@@ -124,12 +124,12 @@ func (impl *OSBuildKojiJobImpl) Run(job worker.Job) error {
 			args.Manifest = manifestJR.Manifest
 			if len(args.Manifest) == 0 {
 				err := fmt.Errorf("Received empty manifest")
-				result.JobError = clienterrors.WorkerClientError(clienterrors.ErrorManifestDependency, err.Error())
+				result.JobError = clienterrors.WorkerClientError(clienterrors.ErrorEmptyManifest, err.Error())
 				return err
 			}
 		} else {
 			err := fmt.Errorf("Job has no manifest")
-			result.JobError = clienterrors.WorkerClientError(clienterrors.ErrorManifestDependency, err.Error())
+			result.JobError = clienterrors.WorkerClientError(clienterrors.ErrorEmptyManifest, err.Error())
 			return err
 		}
 	}
diff --git a/internal/cloudapi/server.go b/internal/cloudapi/server.go
index bc1f4ac77..4c515ea1b 100644
--- a/internal/cloudapi/server.go
+++ b/internal/cloudapi/server.go
@@ -23,3 +23,7 @@ func NewServer(workers *worker.Server, distros *distroregistry.Registry, config
 func (server *Server) V2(path string) http.Handler {
 	return server.v2.Handler(path)
 }
+
+func (server *Server) Shutdown() {
+	server.v2.Shutdown()
+}
diff --git a/internal/cloudapi/v2/v2.go b/internal/cloudapi/v2/handler.go
similarity index 77%
rename from internal/cloudapi/v2/v2.go
rename to internal/cloudapi/v2/handler.go
index fa5913a21..cde88bfa6 100644
--- a/internal/cloudapi/v2/v2.go
+++ b/internal/cloudapi/v2/handler.go
@@ -2,7 +2,6 @@
 package v2
 
 import (
-	"context"
 	"crypto/rand"
 	"encoding/json"
 	"fmt"
@@ -11,72 +10,27 @@ import (
 	"net/http"
 	"strconv"
 	"strings"
-	"time"
 
 	"github.com/google/uuid"
 	"github.com/labstack/echo/v4"
-	"github.com/labstack/echo/v4/middleware"
-	"github.com/sirupsen/logrus"
 
 	"github.com/osbuild/osbuild-composer/internal/auth"
 	"github.com/osbuild/osbuild-composer/internal/blueprint"
-	"github.com/osbuild/osbuild-composer/internal/common"
 	"github.com/osbuild/osbuild-composer/internal/distro"
-	"github.com/osbuild/osbuild-composer/internal/distroregistry"
-	"github.com/osbuild/osbuild-composer/internal/jobqueue"
 	osbuild "github.com/osbuild/osbuild-composer/internal/osbuild2"
 	"github.com/osbuild/osbuild-composer/internal/ostree"
-	"github.com/osbuild/osbuild-composer/internal/prometheus"
 	"github.com/osbuild/osbuild-composer/internal/rpmmd"
 	"github.com/osbuild/osbuild-composer/internal/target"
 	"github.com/osbuild/osbuild-composer/internal/worker"
 	"github.com/osbuild/osbuild-composer/internal/worker/clienterrors"
 )
 
-// Server represents the state of the cloud Server
-type Server struct {
-	workers *worker.Server
-	distros *distroregistry.Registry
-	config  ServerConfig
-}
-
-type ServerConfig struct {
-	AWSBucket            string
-	TenantProviderFields []string
-	JWTEnabled           bool
-}
-
 type apiHandlers struct {
 	server *Server
 }
 
 type binder struct{}
 
-func NewServer(workers *worker.Server, distros *distroregistry.Registry, config ServerConfig) *Server {
-	server := &Server{
-		workers: workers,
-		distros: distros,
-		config:  config,
-	}
-	return server
-}
-
-func (server *Server) Handler(path string) http.Handler {
-	e := echo.New()
-	e.Binder = binder{}
-	e.HTTPErrorHandler = server.HTTPErrorHandler
-	e.Pre(common.OperationIDMiddleware)
-	e.Use(middleware.Recover())
-	e.Logger = common.Logger()
-
-	handler := apiHandlers{
-		server: server,
-	}
-	RegisterHandlers(e.Group(path, prometheus.MetricsMiddleware), &handler)
-
-	return e
-}
-
 func (b binder) Bind(i interface{}, ctx echo.Context) error {
 	contentType := ctx.Request().Header["Content-Type"]
 	if len(contentType) != 1 || contentType[0] != "application/json" {
@@ -484,12 +438,12 @@ func (h *apiHandlers) PostCompose(ctx echo.Context) error {
 
 	var id uuid.UUID
 	if request.Koji != nil {
-		id, err = enqueueKojiCompose(h.server.workers, uint64(request.Koji.TaskId), request.Koji.Server, request.Koji.Name, request.Koji.Version, request.Koji.Release, distribution, bp, manifestSeed, irs, channel)
+		id, err = h.server.enqueueKojiCompose(uint64(request.Koji.TaskId), request.Koji.Server, request.Koji.Name, request.Koji.Version, request.Koji.Release, distribution, bp, manifestSeed, irs, channel)
 		if err != nil {
 			return err
 		}
 	} else {
-		id, err = enqueueCompose(h.server.workers, distribution, bp, manifestSeed, irs, channel)
+		id, err = h.server.enqueueCompose(distribution, bp, manifestSeed, irs, channel)
 		if err != nil {
 			return err
 		}
@@ -507,215 +461,6 @@ func (h *apiHandlers) PostCompose(ctx echo.Context) error {
 	})
 }
 
-func enqueueCompose(workers *worker.Server, distribution distro.Distro, bp blueprint.Blueprint, manifestSeed int64, irs []imageRequest, channel string) (uuid.UUID, error) {
-	var id uuid.UUID
-	if len(irs) != 1 {
-		return id, HTTPError(ErrorInvalidNumberOfImageBuilds)
-	}
-	ir := irs[0]
-
-	depsolveJobID, err := workers.EnqueueDepsolve(&worker.DepsolveJob{
-		PackageSets:      ir.imageType.PackageSets(bp),
-		Repos:            ir.repositories,
-		ModulePlatformID: distribution.ModulePlatformID(),
-		Arch:             ir.arch.Name(),
-		Releasever:       distribution.Releasever(),
-		PackageSetsRepos: ir.packageSetsRepositories,
-	}, channel)
-	if err != nil {
-		return id, HTTPErrorWithInternal(ErrorEnqueueingJob, err)
-	}
-
-	manifestJobID, err := workers.EnqueueManifestJobByID(&worker.ManifestJobByID{}, depsolveJobID, channel)
-	if err != nil {
-		return id, HTTPErrorWithInternal(ErrorEnqueueingJob, err)
-	}
-
-	id, err = workers.EnqueueOSBuildAsDependency(ir.arch.Name(), &worker.OSBuildJob{
-		Targets:         []*target.Target{ir.target},
-		Exports:         ir.imageType.Exports(),
-		StreamOptimized: ir.imageType.Name() == "vmdk", // https://github.com/osbuild/osbuild/issues/528,
-		PipelineNames: &worker.PipelineNames{
-			Build:   ir.imageType.BuildPipelines(),
-			Payload: ir.imageType.PayloadPipelines(),
-		},
-	}, manifestJobID, channel)
-	if err != nil {
-		return id, HTTPErrorWithInternal(ErrorEnqueueingJob, err)
-	}
-
-	ctx, cancel := context.WithTimeout(context.Background(), time.Minute*5)
-	go generateManifest(ctx, cancel, workers, depsolveJobID, manifestJobID, ir.imageType, ir.repositories, ir.imageOptions, manifestSeed, bp.Customizations)
-
-	return id, nil
-}
-
-func enqueueKojiCompose(workers *worker.Server, taskID uint64, server, name, version, release string, distribution distro.Distro, bp blueprint.Blueprint, manifestSeed int64, irs []imageRequest, channel string) (uuid.UUID, error) {
-	var id uuid.UUID
-	kojiDirectory := "osbuild-composer-koji-" + uuid.New().String()
-
-	initID, err := workers.EnqueueKojiInit(&worker.KojiInitJob{
-		Server:  server,
-		Name:    name,
-		Version: version,
-		Release: release,
-	}, channel)
-	if err != nil {
-		return id, HTTPErrorWithInternal(ErrorEnqueueingJob, err)
-	}
-
-	var kojiFilenames []string
-	var buildIDs []uuid.UUID
-	for _, ir := range irs {
-		depsolveJobID, err := workers.EnqueueDepsolve(&worker.DepsolveJob{
-			PackageSets:      ir.imageType.PackageSets(bp),
-			Repos:            ir.repositories,
-			ModulePlatformID: distribution.ModulePlatformID(),
-			Arch:             ir.arch.Name(),
-			Releasever:       distribution.Releasever(),
-			PackageSetsRepos: ir.packageSetsRepositories,
-		}, channel)
-		if err != nil {
-			return id, HTTPErrorWithInternal(ErrorEnqueueingJob, err)
-		}
-
-		manifestJobID, err := workers.EnqueueManifestJobByID(&worker.ManifestJobByID{}, depsolveJobID, channel)
-		if err != nil {
-			return id, HTTPErrorWithInternal(ErrorEnqueueingJob, err)
-		}
-		kojiFilename := fmt.Sprintf(
-			"%s-%s-%s.%s%s",
-			name,
-			version,
-			release,
-			ir.arch.Name(),
-			splitExtension(ir.imageType.Filename()),
-		)
-		buildID, err := workers.EnqueueOSBuildKojiAsDependency(ir.arch.Name(), &worker.OSBuildKojiJob{
-			ImageName: ir.imageType.Filename(),
-			Exports:   ir.imageType.Exports(),
-			PipelineNames: &worker.PipelineNames{
-				Build:   ir.imageType.BuildPipelines(),
-				Payload: ir.imageType.PayloadPipelines(),
-			},
-			KojiServer:    server,
-			KojiDirectory: kojiDirectory,
-			KojiFilename:  kojiFilename,
-		}, manifestJobID, initID, channel)
-		if err != nil {
-			return id, HTTPErrorWithInternal(ErrorEnqueueingJob, err)
-		}
-		kojiFilenames = append(kojiFilenames, kojiFilename)
-		buildIDs = append(buildIDs, buildID)
-
-		ctx, cancel := context.WithTimeout(context.Background(), time.Minute*5)
-		go generateManifest(ctx, cancel, workers, depsolveJobID, manifestJobID, ir.imageType, ir.repositories, ir.imageOptions, manifestSeed, bp.Customizations)
-	}
-	id, err = workers.EnqueueKojiFinalize(&worker.KojiFinalizeJob{
-		Server:        server,
-		Name:          name,
-		Version:       version,
-		Release:       release,
-		KojiFilenames: kojiFilenames,
-		KojiDirectory: kojiDirectory,
-		TaskID:        taskID,
-		StartTime:     uint64(time.Now().Unix()),
-	}, initID, buildIDs, channel)
-	if err != nil {
-		return id, HTTPErrorWithInternal(ErrorEnqueueingJob, err)
-	}
-
-	return id, nil
-}
-
-func generateManifest(ctx context.Context, cancel context.CancelFunc, workers *worker.Server, depsolveJobID uuid.UUID, manifestJobID uuid.UUID, imageType distro.ImageType, repos []rpmmd.RepoConfig, options distro.ImageOptions, seed int64, b *blueprint.Customizations) {
-	defer cancel()
-
-	// wait until job is in a pending state
-	var token uuid.UUID
-	var dynArgs []json.RawMessage
-	var err error
-	logWithId := logrus.WithField("jobId", manifestJobID)
-	for {
-		_, token, _, _, dynArgs, err = workers.RequestJobById(ctx, "", manifestJobID)
-		if err == jobqueue.ErrNotPending {
-			logWithId.Debug("Manifest job not pending, waiting for depsolve job to finish")
-			time.Sleep(time.Millisecond * 50)
-			select {
-			case <-ctx.Done():
-				logWithId.Warning("Manifest job dependencies took longer than 5 minutes to finish, returning to avoid dangling routines")
-				break
-			default:
-				continue
-			}
-		}
-		if err != nil {
-			logWithId.Errorf("Error requesting manifest job: %v", err)
-			return
-		}
-		break
-	}
-
-	var jobResult *worker.ManifestJobByIDResult = &worker.ManifestJobByIDResult{
-		Manifest: nil,
-	}
-
-	defer func() {
-		if jobResult.JobError != nil {
-			logWithId.Errorf("Error in manifest job %v: %v", jobResult.JobError.Reason, err)
-		}
-
-		result, err := json.Marshal(jobResult)
-		if err != nil {
-			logWithId.Errorf("Error marshalling manifest job results: %v", err)
-		}
-
-		err = workers.FinishJob(token, result)
-		if err != nil {
-			logWithId.Errorf("Error finishing manifest job: %v", err)
-		}
-	}()
-
-	if len(dynArgs) == 0 {
-		reason := "No dynamic arguments"
-		jobResult.JobError = clienterrors.WorkerClientError(clienterrors.ErrorNoDynamicArgs, reason)
-		return
-	}
-
-	var depsolveResults worker.DepsolveJobResult
-	err = json.Unmarshal(dynArgs[0], &depsolveResults)
-	if err != nil {
-		reason := "Error parsing dynamic arguments"
-		jobResult.JobError = clienterrors.WorkerClientError(clienterrors.ErrorParsingDynamicArgs, reason)
-		return
-	}
-
-	_, _, err = workers.DepsolveJobStatus(depsolveJobID, &depsolveResults)
-	if err != nil {
-		reason := "Error reading depsolve status"
-		jobResult.JobError = clienterrors.WorkerClientError(clienterrors.ErrorReadingJobStatus, reason)
-		return
-	}
-
-	if jobErr := depsolveResults.JobError; jobErr != nil {
-		if jobErr.ID == clienterrors.ErrorDNFDepsolveError || jobErr.ID == clienterrors.ErrorDNFMarkingErrors {
-			jobResult.JobError = clienterrors.WorkerClientError(clienterrors.ErrorDepsolveDependency, "Error in depsolve job dependency input, bad package set requested")
-			return
-		}
-		jobResult.JobError = clienterrors.WorkerClientError(clienterrors.ErrorDepsolveDependency, "Error in depsolve job dependency")
-		return
-	}
-
-	manifest, err := imageType.Manifest(b, options, repos, depsolveResults.PackageSpecs, seed)
-	if err != nil {
-		reason := "Error generating manifest"
-		jobResult.JobError = clienterrors.WorkerClientError(clienterrors.ErrorManifestGeneration, reason)
-		return
-	}
-
-	jobResult.Manifest = manifest
-}
-
 func imageTypeFromApiImageType(it ImageTypes, arch distro.Arch) string {
 	switch it {
 	case ImageTypesAws:
diff --git a/internal/cloudapi/v2/server.go b/internal/cloudapi/v2/server.go
new file mode 100644
index 000000000..8c0833f44
--- /dev/null
+++ b/internal/cloudapi/v2/server.go
@@ -0,0 +1,294 @@
+package v2
+
+import (
+	"context"
+	"encoding/json"
+	"fmt"
+	"net/http"
+	"sync"
+	"time"
+
+	"github.com/google/uuid"
+	"github.com/labstack/echo/v4"
+	"github.com/labstack/echo/v4/middleware"
+	"github.com/sirupsen/logrus"
+
+	"github.com/osbuild/osbuild-composer/internal/blueprint"
+	"github.com/osbuild/osbuild-composer/internal/common"
+	"github.com/osbuild/osbuild-composer/internal/distro"
+	"github.com/osbuild/osbuild-composer/internal/distroregistry"
+	"github.com/osbuild/osbuild-composer/internal/jobqueue"
+	"github.com/osbuild/osbuild-composer/internal/prometheus"
+	"github.com/osbuild/osbuild-composer/internal/rpmmd"
+	"github.com/osbuild/osbuild-composer/internal/target"
+	"github.com/osbuild/osbuild-composer/internal/worker"
+	"github.com/osbuild/osbuild-composer/internal/worker/clienterrors"
+)
+
+// Server represents the state of the cloud Server
+type Server struct {
+	workers *worker.Server
+	distros *distroregistry.Registry
+	config  ServerConfig
+
+	goroutinesCtx       context.Context
+	goroutinesCtxCancel context.CancelFunc
+	goroutinesGroup     sync.WaitGroup
+}
+
+type ServerConfig struct {
+	AWSBucket            string
+	TenantProviderFields []string
+	JWTEnabled           bool
+}
+
+func NewServer(workers *worker.Server, distros *distroregistry.Registry, config ServerConfig) *Server {
+	ctx, cancel := context.WithCancel(context.Background())
+	server := &Server{
+		workers: workers,
+		distros: distros,
+		config:  config,
+
+		goroutinesCtx:       ctx,
+		goroutinesCtxCancel: cancel,
+	}
+	return server
+}
+
+func (s *Server) Handler(path string) http.Handler {
+	e := echo.New()
+	e.Binder = binder{}
+	e.HTTPErrorHandler = s.HTTPErrorHandler
+	e.Pre(common.OperationIDMiddleware)
+	e.Use(middleware.Recover())
+	e.Logger = common.Logger()
+
+	handler := apiHandlers{
+		server: s,
+	}
+	RegisterHandlers(e.Group(path, prometheus.MetricsMiddleware), &handler)
+
+	return e
+}
+
+func (s *Server) Shutdown() {
+	s.goroutinesCtxCancel()
+	s.goroutinesGroup.Wait()
+}
+
+func (s *Server) enqueueCompose(distribution distro.Distro, bp blueprint.Blueprint, manifestSeed int64, irs []imageRequest, channel string) (uuid.UUID, error) {
+	var id uuid.UUID
+	if len(irs) != 1 {
+		return id, HTTPError(ErrorInvalidNumberOfImageBuilds)
+	}
+	ir := irs[0]
+
+	depsolveJobID, err := s.workers.EnqueueDepsolve(&worker.DepsolveJob{
+		PackageSets:      ir.imageType.PackageSets(bp),
+		Repos:            ir.repositories,
+		ModulePlatformID: distribution.ModulePlatformID(),
+		Arch:             ir.arch.Name(),
+		Releasever:       distribution.Releasever(),
+		PackageSetsRepos: ir.packageSetsRepositories,
+	}, channel)
+	if err != nil {
+		return id, HTTPErrorWithInternal(ErrorEnqueueingJob, err)
+	}
+
+	manifestJobID, err := s.workers.EnqueueManifestJobByID(&worker.ManifestJobByID{}, depsolveJobID, channel)
+	if err != nil {
+		return id, HTTPErrorWithInternal(ErrorEnqueueingJob, err)
+	}
+
+	id, err = s.workers.EnqueueOSBuildAsDependency(ir.arch.Name(), &worker.OSBuildJob{
+		Targets:         []*target.Target{ir.target},
+		Exports:         ir.imageType.Exports(),
+		StreamOptimized: ir.imageType.Name() == "vmdk", // https://github.com/osbuild/osbuild/issues/528,
+		PipelineNames: &worker.PipelineNames{
+			Build:   ir.imageType.BuildPipelines(),
+			Payload: ir.imageType.PayloadPipelines(),
+		},
+	}, manifestJobID, channel)
+	if err != nil {
+		return id, HTTPErrorWithInternal(ErrorEnqueueingJob, err)
+	}
+
+	s.goroutinesGroup.Add(1)
+	go func() {
+		generateManifest(s.goroutinesCtx, s.workers, depsolveJobID, manifestJobID, ir.imageType, ir.repositories, ir.imageOptions, manifestSeed, bp.Customizations)
+		defer s.goroutinesGroup.Done()
+	}()
+
+	return id, nil
+}
+
+func (s *Server) enqueueKojiCompose(taskID uint64, server, name, version, release string, distribution distro.Distro, bp blueprint.Blueprint, manifestSeed int64, irs []imageRequest, channel string) (uuid.UUID, error) {
+	var id uuid.UUID
+	kojiDirectory := "osbuild-composer-koji-" + uuid.New().String()
+
+	initID, err := s.workers.EnqueueKojiInit(&worker.KojiInitJob{
+		Server:  server,
+		Name:    name,
+		Version: version,
+		Release: release,
+	}, channel)
+	if err != nil {
+		return id, HTTPErrorWithInternal(ErrorEnqueueingJob, err)
+	}
+
+	var kojiFilenames []string
+	var buildIDs []uuid.UUID
+	for _, ir := range irs {
+		depsolveJobID, err := s.workers.EnqueueDepsolve(&worker.DepsolveJob{
+			PackageSets:      ir.imageType.PackageSets(bp),
+			Repos:            ir.repositories,
+			ModulePlatformID: distribution.ModulePlatformID(),
+			Arch:             ir.arch.Name(),
+			Releasever:       distribution.Releasever(),
+			PackageSetsRepos: ir.packageSetsRepositories,
+		}, channel)
+		if err != nil {
+			return id, HTTPErrorWithInternal(ErrorEnqueueingJob, err)
+		}
+
+		manifestJobID, err := s.workers.EnqueueManifestJobByID(&worker.ManifestJobByID{}, depsolveJobID, channel)
+		if err != nil {
+			return id, HTTPErrorWithInternal(ErrorEnqueueingJob, err)
+		}
+		kojiFilename := fmt.Sprintf(
+			"%s-%s-%s.%s%s",
+			name,
+			version,
+			release,
+			ir.arch.Name(),
+			splitExtension(ir.imageType.Filename()),
+		)
+		buildID, err := s.workers.EnqueueOSBuildKojiAsDependency(ir.arch.Name(), &worker.OSBuildKojiJob{
+			ImageName: ir.imageType.Filename(),
+			Exports:   ir.imageType.Exports(),
+			PipelineNames: &worker.PipelineNames{
+				Build:   ir.imageType.BuildPipelines(),
+				Payload: ir.imageType.PayloadPipelines(),
+			},
+			KojiServer:    server,
+			KojiDirectory: kojiDirectory,
+			KojiFilename:  kojiFilename,
+		}, manifestJobID, initID, channel)
+		if err != nil {
+			return id, HTTPErrorWithInternal(ErrorEnqueueingJob, err)
+		}
+		kojiFilenames = append(kojiFilenames, kojiFilename)
+		buildIDs = append(buildIDs, buildID)
+
+		// copy the image request while passing it into the goroutine to prevent data races
+		s.goroutinesGroup.Add(1)
+		go func(ir imageRequest) {
+			generateManifest(s.goroutinesCtx, s.workers, depsolveJobID, manifestJobID, ir.imageType, ir.repositories, ir.imageOptions, manifestSeed, bp.Customizations)
+			defer s.goroutinesGroup.Done()
+		}(ir)
+	}
+	id, err = s.workers.EnqueueKojiFinalize(&worker.KojiFinalizeJob{
+		Server:        server,
+		Name:          name,
+		Version:       version,
+		Release:       release,
+		KojiFilenames: kojiFilenames,
+		KojiDirectory: kojiDirectory,
+		TaskID:        taskID,
+		StartTime:     uint64(time.Now().Unix()),
+	}, initID, buildIDs, channel)
+	if err != nil {
+		return id, HTTPErrorWithInternal(ErrorEnqueueingJob, err)
+	}
+
+	return id, nil
+}
+
+func generateManifest(ctx context.Context, workers *worker.Server, depsolveJobID uuid.UUID, manifestJobID uuid.UUID, imageType distro.ImageType, repos []rpmmd.RepoConfig, options distro.ImageOptions, seed int64, b *blueprint.Customizations) {
+	ctx, cancel := context.WithTimeout(ctx, time.Minute*5)
+	defer cancel()
+
+	// wait until job is in a pending state
+	var token uuid.UUID
+	var dynArgs []json.RawMessage
+	var err error
+	logWithId := logrus.WithField("jobId", manifestJobID)
+	for {
+		_, token, _, _, dynArgs, err = workers.RequestJobById(ctx, "", manifestJobID)
+		if err == jobqueue.ErrNotPending {
+			logWithId.Debug("Manifest job not pending, waiting for depsolve job to finish")
+			time.Sleep(time.Millisecond * 50)
+			select {
+			case <-ctx.Done():
+				logWithId.Warning("Manifest job dependencies took longer than 5 minutes to finish, or the server is shutting down, returning to avoid dangling routines")
+				break
+			default:
+				continue
+			}
+		}
+		if err != nil {
+			logWithId.Errorf("Error requesting manifest job: %v", err)
+			return
+		}
+		break
+	}
+
+	jobResult := &worker.ManifestJobByIDResult{
+		Manifest: nil,
+	}
+
+	defer func() {
+		if jobResult.JobError != nil {
+			logWithId.Errorf("Error in manifest job %v: %v", jobResult.JobError.Reason, err)
+		}
+
+		result, err := json.Marshal(jobResult)
+		if err != nil {
+			logWithId.Errorf("Error marshalling manifest job results: %v", err)
+		}
+
+		err = workers.FinishJob(token, result)
+		if err != nil {
+			logWithId.Errorf("Error finishing manifest job: %v", err)
+		}
+	}()
+
+	if len(dynArgs) == 0 {
+		reason := "No dynamic arguments"
+		jobResult.JobError = clienterrors.WorkerClientError(clienterrors.ErrorNoDynamicArgs, reason)
+		return
+	}
+
+	var depsolveResults worker.DepsolveJobResult
+	err = json.Unmarshal(dynArgs[0], &depsolveResults)
+	if err != nil {
+		reason := "Error parsing dynamic arguments"
+		jobResult.JobError = clienterrors.WorkerClientError(clienterrors.ErrorParsingDynamicArgs, reason)
+		return
+	}
+
+	_, _, err = workers.DepsolveJobStatus(depsolveJobID, &depsolveResults)
+	if err != nil {
+		reason := "Error reading depsolve status"
+		jobResult.JobError = clienterrors.WorkerClientError(clienterrors.ErrorReadingJobStatus, reason)
+		return
+	}
+
+	if jobErr := depsolveResults.JobError; jobErr != nil {
+		if jobErr.ID == clienterrors.ErrorDNFDepsolveError || jobErr.ID == clienterrors.ErrorDNFMarkingErrors {
+			jobResult.JobError = clienterrors.WorkerClientError(clienterrors.ErrorDepsolveDependency, "Error in depsolve job dependency input, bad package set requested")
+			return
+		}
+		jobResult.JobError = clienterrors.WorkerClientError(clienterrors.ErrorDepsolveDependency, "Error in depsolve job dependency")
+		return
+	}
+
+	manifest, err := imageType.Manifest(b, options, repos, depsolveResults.PackageSpecs, seed)
+	if err != nil {
+		reason := "Error generating manifest"
+		jobResult.JobError = clienterrors.WorkerClientError(clienterrors.ErrorManifestGeneration, reason)
+		return
+	}
+
+	jobResult.Manifest = manifest
+}
diff --git a/internal/upload/koji/koji.go b/internal/upload/koji/koji.go
index 5a43f8ea2..084544e7a 100644
--- a/internal/upload/koji/koji.go
+++ b/internal/upload/koji/koji.go
@@ -162,7 +162,7 @@ func NewFromPlain(server, user, password string, transport http.RoundTripper) (*
 	// The API doesn't require sessionID, sessionKey and callnum yet,
 	// so there's no need to use the custom Koji RoundTripper,
 	// let's just use the one that the called passed in.
-	rhTransport := &rh.RoundTripper{Client: rh.NewClient()}
+	rhTransport := CreateRetryableTransport()
 	loginClient, err := xmlrpc.NewClient(server, rhTransport)
 	if err != nil {
 		return nil, err
@@ -295,14 +295,14 @@ func (k *Koji) CGImport(build ImageBuild, buildRoots []BuildRoot, images []Image
 }
 
 // uploadChunk uploads a byte slice to a given filepath/filname at a given offset
-func (k *Koji) uploadChunk(chunk []byte, filepath, filename string, offset uint64) (uint, error) {
+func (k *Koji) uploadChunk(chunk []byte, filepath, filename string, offset uint64) error {
 	// We have to open-code a bastardized version of XML-RPC: We send an octet-stream, as
 	// if it was an RPC call, and get a regular XML-RPC reply back. In addition to the
 	// standard URL parameters, we also have to pass any other parameters as part of the
 	// URL, as the body can only contain the payload.
 	u, err := url.Parse(k.server)
 	if err != nil {
-		return 0, err
+		return err
 	}
 	q := u.Query()
 	q.Add("filepath", filepath)
@@ -311,49 +311,23 @@ func (k *Koji) uploadChunk(chunk []byte, filepath, filename string, offset uint6
 	q.Add("fileverify", "adler32")
 	u.RawQuery = q.Encode()
 
-	retries := uint(0)
-
-	countingCheckRetry := func(ctx context.Context, resp *http.Response, err error) (bool, error) {
-		shouldRetry, retErr := rh.DefaultRetryPolicy(ctx, resp, err)
-
-		// DefaultRetryPolicy denies retrying for any certificate related error.
-		// Override it in case the error is a timeout.
-		if !shouldRetry && err != nil {
-			if v, ok := err.(*url.Error); ok {
-				if _, ok := v.Err.(x509.UnknownAuthorityError); ok {
-					// retry if it's a timeout
-					shouldRetry = strings.Contains(strings.ToLower(v.Error()), "timeout")
-					retErr = v
-				}
-			}
-		}
-
-		if shouldRetry {
-			retries++
-		}
-
-		return shouldRetry, retErr
-	}
-
-	client := rh.NewClient()
+	client := createCustomRetryableClient()
 
 	client.HTTPClient = &http.Client{
 		Transport: k.transport,
 	}
-	client.CheckRetry = countingCheckRetry
-	client.Logger = rh.LeveledLogger(&LeveledLogrus{logrus.StandardLogger()})
 
 	respData, err := client.Post(u.String(), "application/octet-stream", bytes.NewBuffer(chunk))
 
 	if err != nil {
-		return retries, err
+		return err
 	}
 
 	defer respData.Body.Close()
 
 	body, err := ioutil.ReadAll(respData.Body)
 	if err != nil {
-		return retries, err
+		return err
 	}
 
 	var reply struct {
@@ -364,31 +338,30 @@ func (k *Koji) uploadChunk(chunk []byte, filepath, filename string, offset uint6
 	resp := xmlrpc.Response(body)
 
 	if resp.Err() != nil {
-		return retries, fmt.Errorf("xmlrpc server returned an error: %v", resp.Err())
+		return fmt.Errorf("xmlrpc server returned an error: %v", resp.Err())
 	}
 
 	err = resp.Unmarshal(&reply)
 	if err != nil {
-		return retries, fmt.Errorf("cannot unmarshal the xmlrpc response: %v", err)
+		return fmt.Errorf("cannot unmarshal the xmlrpc response: %v", err)
 	}
 
 	if reply.Size != len(chunk) {
-		return retries, fmt.Errorf("Sent a chunk of %d bytes, but server got %d bytes", len(chunk), reply.Size)
+		return fmt.Errorf("Sent a chunk of %d bytes, but server got %d bytes", len(chunk), reply.Size)
 	}
 
 	digest := fmt.Sprintf("%08x", adler32.Checksum(chunk))
 	if reply.HexDigest != digest {
-		return retries, fmt.Errorf("Sent a chunk with Adler32 digest %s, but server computed digest %s", digest, reply.HexDigest)
+		return fmt.Errorf("Sent a chunk with Adler32 digest %s, but server computed digest %s", digest, reply.HexDigest)
 	}
 
-	return retries, nil
+	return nil
 }
 
 // Upload uploads file to the temporary filepath on the kojiserver under the name filename
 // The md5sum and size of the file is returned on success.
 func (k *Koji) Upload(file io.Reader, filepath, filename string) (string, uint64, error) {
 	chunk := make([]byte, 1024*1024) // upload a megabyte at a time
-	retries := uint(0)
 	offset := uint64(0)
 	// Koji uses MD5 hashes
 	/* #nosec G401 */
@@ -401,12 +374,11 @@ func (k *Koji) Upload(file io.Reader, filepath, filename string) (string, uint64
 			}
 			return "", 0, err
 		}
-		r, err := k.uploadChunk(chunk[:n], filepath, filename, offset)
-		retries += r
+		err = k.uploadChunk(chunk[:n], filepath, filename, offset)
 		if err != nil {
-			logrus.Infof("Koji upload failed after %d retries", retries)
 			return "", 0, err
 		}
+
 		offset += uint64(n)
 
 		m, err := hash.Write(chunk[:n])
@@ -417,7 +389,6 @@ func (k *Koji) Upload(file io.Reader, filepath, filename string) (string, uint64
 			return "", 0, fmt.Errorf("sent %d bytes, but hashed %d", n, m)
 		}
 	}
-	logrus.Infof("Koji upload successful after %d retries", retries)
 	return fmt.Sprintf("%x", hash.Sum(nil)), offset, nil
 }
 
@@ -472,7 +443,7 @@ func GSSAPICredentialsFromEnv() (*GSSAPICredentials, error) {
 func CreateKojiTransport(relaxTimeout uint) http.RoundTripper {
 	// Koji for some reason needs TLS renegotiation enabled.
 	// Clone the default http rt and enable renegotiation.
-	rt := &rh.RoundTripper{Client: rh.NewClient()}
+	rt := CreateRetryableTransport()
 
 	transport := rt.Client.HTTPClient.Transport.(*http.Transport)
 
@@ -492,3 +463,33 @@ func CreateKojiTransport(relaxTimeout uint) http.RoundTripper {
 
 	return rt
 }
+
+func customCheckRetry(ctx context.Context, resp *http.Response, err error) (bool, error) {
+	shouldRetry, retErr := rh.DefaultRetryPolicy(ctx, resp, err)
+
+	// DefaultRetryPolicy denies retrying for any certificate related error.
+	// Override it in case the error is a timeout.
+	if !shouldRetry && err != nil {
+		if v, ok := err.(*url.Error); ok {
+			if _, ok := v.Err.(x509.UnknownAuthorityError); ok {
+				// retry if it's a timeout
+				return strings.Contains(strings.ToLower(v.Error()), "timeout"), v
+			}
+		}
+	}
+
+	return shouldRetry, retErr
+}
+
+func createCustomRetryableClient() *rh.Client {
+	client := rh.NewClient()
+	client.Logger = rh.LeveledLogger(&LeveledLogrus{logrus.StandardLogger()})
+	client.CheckRetry = customCheckRetry
+	return client
+}
+
+func CreateRetryableTransport() *rh.RoundTripper {
+	rt := rh.RoundTripper{}
+	rt.Client = createCustomRetryableClient()
+	return &rt
+}
diff --git a/internal/upload/koji/rh-logrus-adapter.go b/internal/upload/koji/rh-logrus-adapter.go
index 444512865..5a5c482c1 100644
--- a/internal/upload/koji/rh-logrus-adapter.go
+++ b/internal/upload/koji/rh-logrus-adapter.go
@@ -1,6 +1,8 @@
 package koji
 
 import (
+	"strings"
+
 	"github.com/sirupsen/logrus"
 )
 
@@ -8,7 +10,9 @@ type LeveledLogrus struct {
 	*logrus.Logger
 }
 
-func (l *LeveledLogrus) fields(keysAndValues ...interface{}) map[string]interface{} {
+const monitoringKeyword = "retrying"
+
+func fields(keysAndValues ...interface{}) map[string]interface{} {
 	fields := make(map[string]interface{})
 
 	for i := 0; i < len(keysAndValues)-1; i += 2 {
@@ -19,16 +23,20 @@ func (l *LeveledLogrus) fields(keysAndValues ...interface{}) map[string]interfac
 }
 
 func (l *LeveledLogrus) Error(msg string, keysAndValues ...interface{}) {
-	l.WithFields(l.fields(keysAndValues...)).Error(msg)
+	l.WithFields(fields(keysAndValues...)).Error(msg)
 }
 
 func (l *LeveledLogrus) Info(msg string, keysAndValues ...interface{}) {
-	l.WithFields(l.fields(keysAndValues...)).Info(msg)
+	l.WithFields(fields(keysAndValues...)).Info(msg)
 }
 func (l *LeveledLogrus) Debug(msg string, keysAndValues ...interface{}) {
-	l.WithFields(l.fields(keysAndValues...)).Debug(msg)
+	if strings.Contains(msg, monitoringKeyword) {
+		l.WithFields(fields(keysAndValues...)).Info(msg)
+	} else {
+		l.WithFields(fields(keysAndValues...)).Debug(msg)
+	}
 }
 
 func (l *LeveledLogrus) Warn(msg string, keysAndValues ...interface{}) {
-	l.WithFields(l.fields(keysAndValues...)).Warn(msg)
+	l.WithFields(fields(keysAndValues...)).Warn(msg)
 }
diff --git a/internal/worker/clienterrors/errors.go b/internal/worker/clienterrors/errors.go
index 9f090672f..c288f2bfe 100644
--- a/internal/worker/clienterrors/errors.go
+++ b/internal/worker/clienterrors/errors.go
@@ -19,6 +19,7 @@ const (
 	ErrorKojiFinalize         ClientErrorCode = 16
 	ErrorInvalidConfig        ClientErrorCode = 17
 	ErrorOldResultCompatible  ClientErrorCode = 18
+	ErrorEmptyManifest        ClientErrorCode = 19
 
 	ErrorDNFDepsolveError ClientErrorCode = 20
 	ErrorDNFMarkingErrors ClientErrorCode = 21
@@ -63,6 +64,12 @@ func GetStatusCode(err *Error) StatusCode {
 		return JobStatusUserInputError
 	case ErrorInvalidTarget:
 		return JobStatusUserInputError
+	case ErrorDepsolveDependency:
+		return JobStatusUserInputError
+	case ErrorManifestDependency:
+		return JobStatusUserInputError
+	case ErrorEmptyManifest:
+		return JobStatusUserInputError
 	default:
 		return JobStatusInternalError
 	}


Test patch:
diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml
index 98e0129ea..b96c7e495 100644
--- a/.github/workflows/tests.yml
+++ b/.github/workflows/tests.yml
@@ -17,13 +17,13 @@ jobs:
     steps:
 
       - name: Set up Go 1.16
-        uses: actions/setup-go@v2.1.5
+        uses: actions/setup-go@v3
         with:
           go-version: 1.16
         id: go
 
       - name: Check out code into the Go module directory
-        uses: actions/checkout@v2.4.0
+        uses: actions/checkout@v3
         with:
           ref: ${{ github.event.pull_request.head.sha }}
 
@@ -57,13 +57,13 @@ jobs:
     steps:
 
       - name: Set up Go 1.16
-        uses: actions/setup-go@v2.1.5
+        uses: actions/setup-go@v3
         with:
           go-version: 1.16
         id: go
 
       - name: Check out code into the Go module directory
-        uses: actions/checkout@v2.4.0
+        uses: actions/checkout@v3
         with:
           ref: ${{ github.event.pull_request.head.sha }}
 
@@ -83,13 +83,13 @@ jobs:
     steps:
 
       - name: Set up Go 1.16
-        uses: actions/setup-go@v2.1.5
+        uses: actions/setup-go@v3
         with:
           go-version: 1.16
         id: go
 
       - name: Check out code into the Go module directory
-        uses: actions/checkout@v2.4.0
+        uses: actions/checkout@v3
         with:
           ref: ${{ github.event.pull_request.head.sha }}
 
@@ -109,7 +109,7 @@ jobs:
     name: "🐚 Shellcheck"
     runs-on: ubuntu-20.04
     steps:
-    - uses: actions/checkout@v2.4.0
+    - uses: actions/checkout@v3
       with:
         ref: ${{ github.event.pull_request.head.sha }}
     - name: Run ShellCheck
@@ -127,7 +127,7 @@ jobs:
       - name: Install dependencies
         run: sudo dnf install -y rpmlint rpm-build make git-core
 
-      - uses: actions/checkout@v2.4.0
+      - uses: actions/checkout@v3
         with:
           ref: ${{ github.event.pull_request.head.sha }}
 
diff --git a/cmd/osbuild-dnf-json-tests/main_test.go b/cmd/osbuild-dnf-json-tests/main_test.go
index 93bc72c17..fae5d29bd 100644
--- a/cmd/osbuild-dnf-json-tests/main_test.go
+++ b/cmd/osbuild-dnf-json-tests/main_test.go
@@ -5,9 +5,7 @@
 package main
 
 import (
-	"io/ioutil"
 	"net/http"
-	"os"
 	"path"
 	"testing"
 
@@ -63,9 +61,7 @@ func TestCrossArchDepsolve(t *testing.T) {
 			t.Parallel()
 
 			// Set up temporary directory for rpm/dnf cache
-			dir, err := ioutil.TempDir("/tmp", "rpmmd-test-")
-			require.Nilf(t, err, "Failed to create tmp dir for depsolve test: %v", err)
-			defer os.RemoveAll(dir)
+			dir := t.TempDir()
 
 			// use a fullpath to dnf-json, this allows this test to have an arbitrary
 			// working directory
diff --git a/internal/cloudapi/v2/v2_koji_test.go b/internal/cloudapi/v2/v2_koji_test.go
index 069693d81..faa08d375 100644
--- a/internal/cloudapi/v2/v2_koji_test.go
+++ b/internal/cloudapi/v2/v2_koji_test.go
@@ -4,10 +4,8 @@ import (
 	"context"
 	"encoding/json"
 	"fmt"
-	"io/ioutil"
 	"net/http"
 	"net/http/httptest"
-	"os"
 	"testing"
 	"time"
 
@@ -28,10 +26,7 @@ type jobResult struct {
 }
 
 func TestKojiCompose(t *testing.T) {
-	dir, err := ioutil.TempDir("", "osbuild-composer-test-api-v2-")
-	require.NoError(t, err)
-	defer os.RemoveAll(dir)
-	kojiServer, workerServer, _, cancel := newV2Server(t, dir, []string{""}, false)
+	kojiServer, workerServer, _, cancel := newV2Server(t, t.TempDir(), []string{""}, false)
 	handler := kojiServer.Handler("/api/image-builder-composer/v2")
 	workerHandler := workerServer.Handler()
 	defer cancel()
@@ -461,10 +456,7 @@ func TestKojiCompose(t *testing.T) {
 }
 
 func TestKojiRequest(t *testing.T) {
-	dir, err := ioutil.TempDir("", "osbuild-composer-test-api-v2-")
-	require.NoError(t, err)
-	defer os.RemoveAll(dir)
-	server, _, _, cancel := newV2Server(t, dir, []string{""}, false)
+	server, _, _, cancel := newV2Server(t, t.TempDir(), []string{""}, false)
 	handler := server.Handler("/api/image-builder-composer/v2")
 	defer cancel()
 
@@ -476,7 +468,7 @@ func TestKojiRequest(t *testing.T) {
 	resp := rec.Result()
 
 	var status api.Status
-	err = json.NewDecoder(resp.Body).Decode(&status)
+	err := json.NewDecoder(resp.Body).Decode(&status)
 	require.NoError(t, err)
 	require.Equal(t, http.StatusNotFound, resp.StatusCode)
 
@@ -493,10 +485,7 @@ func TestKojiRequest(t *testing.T) {
 }
 
 func TestKojiJobTypeValidation(t *testing.T) {
-	dir, err := ioutil.TempDir("", "osbuild-composer-test-api-v2-")
-	require.NoError(t, err)
-	defer os.RemoveAll(dir)
-	server, workers, _, cancel := newV2Server(t, dir, []string{""}, false)
+	server, workers, _, cancel := newV2Server(t, t.TempDir(), []string{""}, false)
 	handler := server.Handler("/api/image-builder-composer/v2")
 	defer cancel()
 
diff --git a/internal/cloudapi/v2/v2_multi_tenancy_test.go b/internal/cloudapi/v2/v2_multi_tenancy_test.go
index d6ff6dda6..1a9df75c2 100644
--- a/internal/cloudapi/v2/v2_multi_tenancy_test.go
+++ b/internal/cloudapi/v2/v2_multi_tenancy_test.go
@@ -4,9 +4,7 @@ import (
 	"context"
 	"encoding/json"
 	"fmt"
-	"io/ioutil"
 	"net/http"
-	"os"
 	"testing"
 	"time"
 
@@ -191,13 +189,9 @@ func runNextJob(t *testing.T, jobs []uuid.UUID, workerHandler http.Handler, orgI
 // that all jobs are assigned to the correct channel, therefore we need also
 // this test.
 func TestMultitenancy(t *testing.T) {
-	dir, err := ioutil.TempDir("", "osbuild-composer-test-api-v2-")
-	require.NoError(t, err)
-	defer os.RemoveAll(dir)
-
 	// Passing an empty list as depsolving channels, we want to do depsolves
 	// ourselvess
-	apiServer, workerServer, q, cancel := newV2Server(t, dir, []string{}, true)
+	apiServer, workerServer, q, cancel := newV2Server(t, t.TempDir(), []string{}, true)
 	handler := apiServer.Handler("/api/image-builder-composer/v2")
 	defer cancel()
 
diff --git a/internal/cloudapi/v2/v2_test.go b/internal/cloudapi/v2/v2_test.go
index 97fd24b87..f9d90c567 100644
--- a/internal/cloudapi/v2/v2_test.go
+++ b/internal/cloudapi/v2/v2_test.go
@@ -4,9 +4,8 @@ import (
 	"context"
 	"encoding/json"
 	"fmt"
-	"io/ioutil"
 	"net/http"
-	"os"
+	"sync"
 	"testing"
 
 	"github.com/stretchr/testify/require"
@@ -40,12 +39,21 @@ func newV2Server(t *testing.T, dir string, depsolveChannels []string, enableJWT
 	}
 	v2Server := v2.NewServer(workerServer, distros, config)
 	require.NotNil(t, v2Server)
+	t.Cleanup(v2Server.Shutdown)
 
 	// start a routine which just completes depsolve jobs
 	depsolveContext, cancel := context.WithCancel(context.Background())
+	var wg sync.WaitGroup
+	wg.Add(1)
 	go func() {
+		defer wg.Done()
 		for {
-			_, token, _, _, _, err := workerServer.RequestJob(context.Background(), test_distro.TestDistroName, []string{"depsolve"}, depsolveChannels)
+			_, token, _, _, _, err := workerServer.RequestJob(depsolveContext, test_distro.TestDistroName, []string{"depsolve"}, depsolveChannels)
+			select {
+			case <-depsolveContext.Done():
+				return
+			default:
+			}
 			if err != nil {
 				continue
 			}
@@ -56,23 +64,19 @@ func newV2Server(t *testing.T, dir string, depsolveChannels []string, enableJWT
 				return
 			}
 
-			select {
-			case <-depsolveContext.Done():
-				return
-			default:
-				continue
-			}
 		}
 	}()
 
-	return v2Server, workerServer, q, cancel
+	cancelWithWait := func() {
+		cancel()
+		wg.Wait()
+	}
+
+	return v2Server, workerServer, q, cancelWithWait
 }
 
 func TestUnknownRoute(t *testing.T) {
-	dir, err := ioutil.TempDir("", "osbuild-composer-test-api-v2-")
-	require.NoError(t, err)
-	defer os.RemoveAll(dir)
-	srv, _, _, cancel := newV2Server(t, dir, []string{""}, false)
+	srv, _, _, cancel := newV2Server(t, t.TempDir(), []string{""}, false)
 	defer cancel()
 
 	test.TestRoute(t, srv.Handler("/api/image-builder-composer/v2"), false, "GET", "/api/image-builder-composer/v2/badroute", ``, http.StatusNotFound, `
@@ -86,10 +90,7 @@ func TestUnknownRoute(t *testing.T) {
 }
 
 func TestGetError(t *testing.T) {
-	dir, err := ioutil.TempDir("", "osbuild-composer-test-api-v2-")
-	require.NoError(t, err)
-	defer os.RemoveAll(dir)
-	srv, _, _, cancel := newV2Server(t, dir, []string{""}, false)
+	srv, _, _, cancel := newV2Server(t, t.TempDir(), []string{""}, false)
 	defer cancel()
 
 	test.TestRoute(t, srv.Handler("/api/image-builder-composer/v2"), false, "GET", "/api/image-builder-composer/v2/errors/4", ``, http.StatusOK, `
@@ -112,10 +113,7 @@ func TestGetError(t *testing.T) {
 }
 
 func TestGetErrorList(t *testing.T) {
-	dir, err := ioutil.TempDir("", "osbuild-composer-test-api-v2-")
-	require.NoError(t, err)
-	defer os.RemoveAll(dir)
-	srv, _, _, cancel := newV2Server(t, dir, []string{""}, false)
+	srv, _, _, cancel := newV2Server(t, t.TempDir(), []string{""}, false)
 	defer cancel()
 
 	test.TestRoute(t, srv.Handler("/api/image-builder-composer/v2"), false, "GET", "/api/image-builder-composer/v2/errors?page=3&size=1", ``, http.StatusOK, `
@@ -134,10 +132,7 @@ func TestGetErrorList(t *testing.T) {
 }
 
 func TestCompose(t *testing.T) {
-	dir, err := ioutil.TempDir("", "osbuild-composer-test-api-v2-")
-	require.NoError(t, err)
-	defer os.RemoveAll(dir)
-	srv, _, _, cancel := newV2Server(t, dir, []string{""}, false)
+	srv, _, _, cancel := newV2Server(t, t.TempDir(), []string{""}, false)
 	defer cancel()
 
 	// create two ostree repos, one to serve the default test_distro ref (for fallback tests) and one to serve a custom ref
@@ -555,10 +550,7 @@ func TestCompose(t *testing.T) {
 }
 
 func TestComposeStatusSuccess(t *testing.T) {
-	dir, err := ioutil.TempDir("", "osbuild-composer-test-api-v2-")
-	require.NoError(t, err)
-	defer os.RemoveAll(dir)
-	srv, wrksrv, _, cancel := newV2Server(t, dir, []string{""}, false)
+	srv, wrksrv, _, cancel := newV2Server(t, t.TempDir(), []string{""}, false)
 	defer cancel()
 
 	test.TestRoute(t, srv.Handler("/api/image-builder-composer/v2"), false, "POST", "/api/image-builder-composer/v2/compose", fmt.Sprintf(`
@@ -628,10 +620,7 @@ func TestComposeStatusSuccess(t *testing.T) {
 }
 
 func TestComposeStatusFailure(t *testing.T) {
-	dir, err := ioutil.TempDir("", "osbuild-composer-test-api-v2-")
-	require.NoError(t, err)
-	defer os.RemoveAll(dir)
-	srv, wrksrv, _, cancel := newV2Server(t, dir, []string{""}, false)
+	srv, wrksrv, _, cancel := newV2Server(t, t.TempDir(), []string{""}, false)
 	defer cancel()
 
 	test.TestRoute(t, srv.Handler("/api/image-builder-composer/v2"), false, "POST", "/api/image-builder-composer/v2/compose", fmt.Sprintf(`
@@ -687,10 +676,7 @@ func TestComposeStatusFailure(t *testing.T) {
 }
 
 func TestComposeLegacyError(t *testing.T) {
-	dir, err := ioutil.TempDir("", "osbuild-composer-test-api-v2-")
-	require.NoError(t, err)
-	defer os.RemoveAll(dir)
-	srv, wrksrv, _, cancel := newV2Server(t, dir, []string{""}, false)
+	srv, wrksrv, _, cancel := newV2Server(t, t.TempDir(), []string{""}, false)
 	defer cancel()
 
 	test.TestRoute(t, srv.Handler("/api/image-builder-composer/v2"), false, "POST", "/api/image-builder-composer/v2/compose", fmt.Sprintf(`
@@ -749,10 +735,7 @@ func TestComposeLegacyError(t *testing.T) {
 }
 
 func TestComposeJobError(t *testing.T) {
-	dir, err := ioutil.TempDir("", "osbuild-composer-test-api-v2-")
-	require.NoError(t, err)
-	defer os.RemoveAll(dir)
-	srv, wrksrv, _, cancel := newV2Server(t, dir, []string{""}, false)
+	srv, wrksrv, _, cancel := newV2Server(t, t.TempDir(), []string{""}, false)
 	defer cancel()
 
 	test.TestRoute(t, srv.Handler("/api/image-builder-composer/v2"), false, "POST", "/api/image-builder-composer/v2/compose", fmt.Sprintf(`
@@ -814,10 +797,7 @@ func TestComposeJobError(t *testing.T) {
 }
 
 func TestComposeCustomizations(t *testing.T) {
-	dir, err := ioutil.TempDir("", "osbuild-composer-test-api-v2-")
-	require.NoError(t, err)
-	defer os.RemoveAll(dir)
-	srv, _, _, cancel := newV2Server(t, dir, []string{""}, false)
+	srv, _, _, cancel := newV2Server(t, t.TempDir(), []string{""}, false)
 	defer cancel()
 
 	test.TestRoute(t, srv.Handler("/api/image-builder-composer/v2"), false, "POST", "/api/image-builder-composer/v2/compose", fmt.Sprintf(`
@@ -863,10 +843,7 @@ func TestComposeCustomizations(t *testing.T) {
 }
 
 func TestImageTypes(t *testing.T) {
-	dir, err := ioutil.TempDir("", "osbuild-composer-test-api-v2-")
-	require.NoError(t, err)
-	defer os.RemoveAll(dir)
-	srv, _, _, cancel := newV2Server(t, dir, []string{""}, false)
+	srv, _, _, cancel := newV2Server(t, t.TempDir(), []string{""}, false)
 	defer cancel()
 
 	test.TestRoute(t, srv.Handler("/api/image-builder-composer/v2"), false, "POST", "/api/image-builder-composer/v2/compose", fmt.Sprintf(`
diff --git a/internal/jobqueue/fsjobqueue/fsjobqueue_test.go b/internal/jobqueue/fsjobqueue/fsjobqueue_test.go
index a920f9c7f..971dec455 100644
--- a/internal/jobqueue/fsjobqueue/fsjobqueue_test.go
+++ b/internal/jobqueue/fsjobqueue/fsjobqueue_test.go
@@ -1,8 +1,6 @@
 package fsjobqueue_test
 
 import (
-	"io/ioutil"
-	"os"
 	"testing"
 
 	"github.com/stretchr/testify/require"
@@ -14,16 +12,12 @@ import (
 
 func TestJobQueueInterface(t *testing.T) {
 	jobqueuetest.TestJobQueue(t, func() (jobqueue.JobQueue, func(), error) {
-		dir, err := ioutil.TempDir("", "jobqueue-test-")
-		if err != nil {
-			return nil, nil, err
-		}
+		dir := t.TempDir()
 		q, err := fsjobqueue.New(dir)
 		if err != nil {
 			return nil, nil, err
 		}
 		stop := func() {
-			_ = os.RemoveAll(dir)
 		}
 		return q, stop, nil
 	})
diff --git a/internal/jsondb/db_private_test.go b/internal/jsondb/db_private_test.go
index d0939196a..377cb6bfe 100644
--- a/internal/jsondb/db_private_test.go
+++ b/internal/jsondb/db_private_test.go
@@ -11,13 +11,7 @@ import (
 )
 
 func TestWriteFileAtomically(t *testing.T) {
-	dir, err := ioutil.TempDir("", "jsondb-test-")
-	require.NoError(t, err)
-
-	defer func() {
-		err := os.RemoveAll(dir)
-		require.NoError(t, err)
-	}()
+	dir := t.TempDir()
 
 	t.Run("success", func(t *testing.T) {
 		octopus := []byte("🐙\n")
@@ -25,7 +19,7 @@ func TestWriteFileAtomically(t *testing.T) {
 		// use an uncommon mode to check it's set correctly
 		perm := os.FileMode(0750)
 
-		err = writeFileAtomically(dir, "octopus", perm, func(f *os.File) error {
+		err := writeFileAtomically(dir, "octopus", perm, func(f *os.File) error {
 			_, err := f.Write(octopus)
 			return err
 		})
@@ -48,12 +42,12 @@ func TestWriteFileAtomically(t *testing.T) {
 	})
 
 	t.Run("error", func(t *testing.T) {
-		err = writeFileAtomically(dir, "no-octopus", 0750, func(f *os.File) error {
+		err := writeFileAtomically(dir, "no-octopus", 0750, func(f *os.File) error {
 			return errors.New("something went wrong")
 		})
 		require.Error(t, err)
 
-		_, err := os.Stat(path.Join(dir, "no-octopus"))
+		_, err = os.Stat(path.Join(dir, "no-octopus"))
 		require.Error(t, err)
 
 		// ensure there are no stray temporary files
diff --git a/internal/jsondb/db_test.go b/internal/jsondb/db_test.go
index 9349e664d..fadfa6e87 100644
--- a/internal/jsondb/db_test.go
+++ b/internal/jsondb/db_test.go
@@ -17,11 +17,6 @@ type document struct {
 	CanSwim bool   `json:"can-swim"`
 }
 
-func cleanupTempDir(t *testing.T, dir string) {
-	err := os.RemoveAll(dir)
-	require.NoError(t, err)
-}
-
 // If the passed directory is not readable (writable), we should notice on the
 // first read (write).
 func TestDegenerate(t *testing.T) {
@@ -42,14 +37,12 @@ func TestDegenerate(t *testing.T) {
 	})
 
 	t.Run("invalid-json", func(t *testing.T) {
-		dir, err := ioutil.TempDir("", "jsondb-test-")
-		require.NoError(t, err)
-		defer cleanupTempDir(t, dir)
+		dir := t.TempDir()
 
 		db := jsondb.New(dir, 0755)
 
 		// write-only file
-		err = ioutil.WriteFile(path.Join(dir, "one.json"), []byte("{"), 0600)
+		err := ioutil.WriteFile(path.Join(dir, "one.json"), []byte("{"), 0600)
 		require.NoError(t, err)
 
 		var d document
@@ -59,11 +52,9 @@ func TestDegenerate(t *testing.T) {
 }
 
 func TestCorrupt(t *testing.T) {
-	dir, err := ioutil.TempDir("", "jsondb-test-")
-	require.NoError(t, err)
-	defer cleanupTempDir(t, dir)
+	dir := t.TempDir()
 
-	err = ioutil.WriteFile(path.Join(dir, "one.json"), []byte("{"), 0600)
+	err := ioutil.WriteFile(path.Join(dir, "one.json"), []byte("{"), 0600)
 	require.NoError(t, err)
 
 	db := jsondb.New(dir, 0755)
@@ -73,11 +64,9 @@ func TestCorrupt(t *testing.T) {
 }
 
 func TestRead(t *testing.T) {
-	dir, err := ioutil.TempDir("", "jsondb-test-")
-	require.NoError(t, err)
-	defer cleanupTempDir(t, dir)
+	dir := t.TempDir()
 
-	err = ioutil.WriteFile(path.Join(dir, "one.json"), []byte("true"), 0600)
+	err := ioutil.WriteFile(path.Join(dir, "one.json"), []byte("true"), 0600)
 	require.NoError(t, err)
 
 	db := jsondb.New(dir, 0755)
@@ -106,9 +95,7 @@ func TestRead(t *testing.T) {
 }
 
 func TestMultiple(t *testing.T) {
-	dir, err := ioutil.TempDir("", "jsondb-test-")
-	require.NoError(t, err)
-	defer cleanupTempDir(t, dir)
+	dir := t.TempDir()
 
 	perm := os.FileMode(0600)
 	documents := map[string]document{
@@ -120,7 +107,7 @@ func TestMultiple(t *testing.T) {
 	db := jsondb.New(dir, perm)
 
 	for name, doc := range documents {
-		err = db.Write(name, doc)
+		err := db.Write(name, doc)
 		require.NoError(t, err)
 	}
 	names, err := db.List()
diff --git a/internal/kojiapi/server_test.go b/internal/kojiapi/server_test.go
index f133a7e0b..2f4abe5d2 100644
--- a/internal/kojiapi/server_test.go
+++ b/internal/kojiapi/server_test.go
@@ -4,11 +4,8 @@ import (
 	"context"
 	"encoding/json"
 	"fmt"
-	"io/ioutil"
-	"log"
 	"net/http"
 	"net/http/httptest"
-	"os"
 	"sync"
 	"testing"
 	"time"
@@ -43,13 +40,7 @@ func newTestKojiServer(t *testing.T, dir string) (*kojiapi.Server, *worker.Serve
 }
 
 func TestStatus(t *testing.T) {
-	dir, err := ioutil.TempDir("", "osbuild-composer-test-kojiapi-")
-	if err != nil {
-		log.Fatal(err)
-	}
-	defer os.RemoveAll(dir)
-
-	kojiServer, _ := newTestKojiServer(t, dir)
+	kojiServer, _ := newTestKojiServer(t, t.TempDir())
 	handler := kojiServer.Handler("/api/composer-koji/v1")
 	test.TestRoute(t, handler, false, "GET", "/api/composer-koji/v1/status", ``, http.StatusOK, `{"status":"OK"}`, "message")
 }
@@ -59,13 +50,7 @@ type jobResult struct {
 }
 
 func TestCompose(t *testing.T) {
-	dir, err := ioutil.TempDir("", "osbuild-composer-test-kojiapi-")
-	if err != nil {
-		log.Fatal(err)
-	}
-	defer os.RemoveAll(dir)
-
-	kojiServer, workerServer := newTestKojiServer(t, dir)
+	kojiServer, workerServer := newTestKojiServer(t, t.TempDir())
 	handler := kojiServer.Handler("/api/composer-koji/v1")
 	workerHandler := workerServer.Handler()
 
@@ -455,13 +440,7 @@ func TestCompose(t *testing.T) {
 }
 
 func TestRequest(t *testing.T) {
-	dir, err := ioutil.TempDir("", "osbuild-composer-test-kojiapi-")
-	if err != nil {
-		log.Fatal(err)
-	}
-	defer os.RemoveAll(dir)
-
-	server, _ := newTestKojiServer(t, dir)
+	server, _ := newTestKojiServer(t, t.TempDir())
 	handler := server.Handler("/api/composer-koji/v1")
 
 	// Make request to an invalid route
@@ -472,7 +451,7 @@ func TestRequest(t *testing.T) {
 	resp := rec.Result()
 
 	var status api.Status
-	err = json.NewDecoder(resp.Body).Decode(&status)
+	err := json.NewDecoder(resp.Body).Decode(&status)
 	require.NoError(t, err)
 	require.Equal(t, http.StatusNotFound, resp.StatusCode)
 
@@ -489,11 +468,7 @@ func TestRequest(t *testing.T) {
 }
 
 func TestJobTypeValidation(t *testing.T) {
-	dir, err := ioutil.TempDir("", "osbuild-composer-test-kojiapi-")
-	if err != nil {
-		log.Fatal(err)
-	}
-	defer os.RemoveAll(dir)
+	dir := t.TempDir()
 
 	server, workers := newTestKojiServer(t, dir)
 	handler := server.Handler("/api/composer-koji/v1")
diff --git a/internal/store/store_test.go b/internal/store/store_test.go
index e036c140e..28448fd3c 100644
--- a/internal/store/store_test.go
+++ b/internal/store/store_test.go
@@ -1,8 +1,6 @@
 package store
 
 import (
-	"io/ioutil"
-	"os"
 	"testing"
 	"time"
 
@@ -112,20 +110,13 @@ func (suite *storeTest) SetupSuite() {
 
 //setup before each test
 func (suite *storeTest) SetupTest() {
-	tmpDir, err := ioutil.TempDir("/tmp", "osbuild-composer-test-")
-	suite.NoError(err)
 	distro := test_distro.New()
 	arch, err := distro.GetArch(test_distro.TestArchName)
 	suite.NoError(err)
-	suite.dir = tmpDir
+	suite.dir = suite.T().TempDir()
 	suite.myStore = New(&suite.dir, arch, nil)
 }
 
-//teardown after each test
-func (suite *storeTest) TearDownTest() {
-	os.RemoveAll(suite.dir)
-}
-
 func (suite *storeTest) TestRandomSHA1String() {
 	hash, err := randomSHA1String()
 	suite.NoError(err)
diff --git a/internal/weldr/api_test.go b/internal/weldr/api_test.go
index b588615b9..1d0980e5e 100644
--- a/internal/weldr/api_test.go
+++ b/internal/weldr/api_test.go
@@ -125,11 +125,7 @@ func TestBasic(t *testing.T) {
 		{"/api/v1/compose/types?distro=fedora-1", http.StatusBadRequest, `{"status":false,"errors":[{"id":"DistroError","msg":"Invalid distro: fedora-1"}]}`},
 	}
 
-	tempdir, err := ioutil.TempDir("", "weldr-tests-")
-	require.NoError(t, err)
-	defer os.RemoveAll(tempdir)
-
-	api, _ := createWeldrAPI(tempdir, rpmmd_mock.BaseFixture)
+	api, _ := createWeldrAPI(t.TempDir(), rpmmd_mock.BaseFixture)
 	for _, c := range cases {
 		test.TestRoute(t, api, true, "GET", c.Path, ``, c.ExpectedStatus, c.ExpectedJSON)
 	}
@@ -153,9 +149,7 @@ func TestBlueprintsNew(t *testing.T) {
 		{"POST", "/api/v0/blueprints/new", `{"name":"test","description":"Test","distro":"fedora-1","packages":[],"version":""}`, http.StatusBadRequest, `{"status":false,"errors":[{"id":"BlueprintsError","msg":"'fedora-1' is not a valid distribution"}]}`},
 	}
 
-	tempdir, err := ioutil.TempDir("", "weldr-tests-")
-	require.NoError(t, err)
-	defer os.RemoveAll(tempdir)
+	tempdir := t.TempDir()
 
 	for _, c := range cases {
 		api, _ := createWeldrAPI(tempdir, rpmmd_mock.BaseFixture)
@@ -164,10 +158,6 @@ func TestBlueprintsNew(t *testing.T) {
 }
 
 func TestBlueprintsNewToml(t *testing.T) {
-	tempdir, err := ioutil.TempDir("", "weldr-tests-")
-	require.NoError(t, err)
-	defer os.RemoveAll(tempdir)
-
 	blueprint := `
 name = "test"
 description = "Test"
@@ -181,7 +171,7 @@ version = "2.4.*"`
 	req.Header.Set("Content-Type", "text/x-toml")
 	recorder := httptest.NewRecorder()
 
-	api, _ := createWeldrAPI(tempdir, rpmmd_mock.BaseFixture)
+	api, _ := createWeldrAPI(t.TempDir(), rpmmd_mock.BaseFixture)
 	api.ServeHTTP(recorder, req)
 
 	r := recorder.Result()
@@ -189,15 +179,11 @@ version = "2.4.*"`
 }
 
 func TestBlueprintsEmptyToml(t *testing.T) {
-	tempdir, err := ioutil.TempDir("", "weldr-tests-")
-	require.NoError(t, err)
-	defer os.RemoveAll(tempdir)
-
 	req := httptest.NewRequest("POST", "/api/v0/blueprints/new", bytes.NewReader(nil))
 	req.Header.Set("Content-Type", "text/x-toml")
 	recorder := httptest.NewRecorder()
 
-	api, _ := createWeldrAPI(tempdir, rpmmd_mock.BaseFixture)
+	api, _ := createWeldrAPI(t.TempDir(), rpmmd_mock.BaseFixture)
 	api.ServeHTTP(recorder, req)
 
 	r := recorder.Result()
@@ -205,10 +191,6 @@ func TestBlueprintsEmptyToml(t *testing.T) {
 }
 
 func TestBlueprintsInvalidToml(t *testing.T) {
-	tempdir, err := ioutil.TempDir("", "weldr-tests-")
-	require.NoError(t, err)
-	defer os.RemoveAll(tempdir)
-
 	blueprint := `
 name = "test"
 description = "Test"
@@ -222,7 +204,7 @@ version = "2.4.*"`
 	req.Header.Set("Content-Type", "text/x-toml")
 	recorder := httptest.NewRecorder()
 
-	api, _ := createWeldrAPI(tempdir, rpmmd_mock.BaseFixture)
+	api, _ := createWeldrAPI(t.TempDir(), rpmmd_mock.BaseFixture)
 	api.ServeHTTP(recorder, req)
 
 	r := recorder.Result()
@@ -242,9 +224,7 @@ func TestBlueprintsWorkspaceJSON(t *testing.T) {
 		{"POST", "/api/v0/blueprints/workspace", ``, http.StatusBadRequest, `{"status":false,"errors":[{"id":"BlueprintsError","msg":"Missing blueprint"}]}`},
 	}
 
-	tempdir, err := ioutil.TempDir("", "weldr-tests-")
-	require.NoError(t, err)
-	defer os.RemoveAll(tempdir)
+	tempdir := t.TempDir()
 
 	for _, c := range cases {
 		api, _ := createWeldrAPI(tempdir, rpmmd_mock.BaseFixture)
@@ -253,10 +233,6 @@ func TestBlueprintsWorkspaceJSON(t *testing.T) {
 }
 
 func TestBlueprintsWorkspaceTOML(t *testing.T) {
-	tempdir, err := ioutil.TempDir("", "weldr-tests-")
-	require.NoError(t, err)
-	defer os.RemoveAll(tempdir)
-
 	blueprint := `
 name = "test"
 description = "Test"
@@ -270,7 +246,7 @@ version = "2.4.*"`
 	req.Header.Set("Content-Type", "text/x-toml")
 	recorder := httptest.NewRecorder()
 
-	api, _ := createWeldrAPI(tempdir, rpmmd_mock.BaseFixture)
+	api, _ := createWeldrAPI(t.TempDir(), rpmmd_mock.BaseFixture)
 	api.ServeHTTP(recorder, req)
 
 	r := recorder.Result()
@@ -278,15 +254,11 @@ version = "2.4.*"`
 }
 
 func TestBlueprintsWorkspaceEmptyTOML(t *testing.T) {
-	tempdir, err := ioutil.TempDir("", "weldr-tests-")
-	require.NoError(t, err)
-	defer os.RemoveAll(tempdir)
-
 	req := httptest.NewRequest("POST", "/api/v0/blueprints/workspace", bytes.NewReader(nil))
 	req.Header.Set("Content-Type", "text/x-toml")
 	recorder := httptest.NewRecorder()
 
-	api, _ := createWeldrAPI(tempdir, rpmmd_mock.BaseFixture)
+	api, _ := createWeldrAPI(t.TempDir(), rpmmd_mock.BaseFixture)
 	api.ServeHTTP(recorder, req)
 
 	r := recorder.Result()
@@ -294,10 +266,6 @@ func TestBlueprintsWorkspaceEmptyTOML(t *testing.T) {
 }
 
 func TestBlueprintsWorkspaceInvalidTOML(t *testing.T) {
-	tempdir, err := ioutil.TempDir("", "weldr-tests-")
-	require.NoError(t, err)
-	defer os.RemoveAll(tempdir)
-
 	blueprint := `
 name = "test"
 description = "Test"
@@ -311,7 +279,7 @@ version = "2.4.*"`
 	req.Header.Set("Content-Type", "text/x-toml")
 	recorder := httptest.NewRecorder()
 
-	api, _ := createWeldrAPI(tempdir, rpmmd_mock.BaseFixture)
+	api, _ := createWeldrAPI(t.TempDir(), rpmmd_mock.BaseFixture)
 	api.ServeHTTP(recorder, req)
 
 	r := recorder.Result()
@@ -333,9 +301,7 @@ func TestBlueprintsInfo(t *testing.T) {
 		{"GET", "/api/v0/blueprints/info/test3-non", ``, http.StatusOK, `{"blueprints":[],"changes":[],"errors":[{"id":"UnknownBlueprint","msg":"test3-non: "}]}`},
 	}
 
-	tempdir, err := ioutil.TempDir("", "weldr-tests-")
-	require.NoError(t, err)
-	defer os.RemoveAll(tempdir)
+	tempdir := t.TempDir()
 
 	for _, c := range cases {
 		api, _ := createWeldrAPI(tempdir, rpmmd_mock.BaseFixture)
@@ -349,11 +315,7 @@ func TestBlueprintsInfo(t *testing.T) {
 }
 
 func TestBlueprintsInfoToml(t *testing.T) {
-	tempdir, err := ioutil.TempDir("", "weldr-tests-")
-	require.NoError(t, err)
-	defer os.RemoveAll(tempdir)
-
-	api, _ := createWeldrAPI(tempdir, rpmmd_mock.BaseFixture)
+	api, _ := createWeldrAPI(t.TempDir(), rpmmd_mock.BaseFixture)
 	test.SendHTTP(api, true, "POST", "/api/v0/blueprints/new", `{"name":"test1","description":"Test","packages":[{"name":"httpd","version":"2.4.*"}],"version":"0.0.0"}`)
 
 	req := httptest.NewRequest("GET", "/api/v0/blueprints/info/test1?format=toml", nil)
@@ -364,7 +326,7 @@ func TestBlueprintsInfoToml(t *testing.T) {
 	require.Equal(t, http.StatusOK, resp.StatusCode)
 
 	var got blueprint.Blueprint
-	_, err = toml.DecodeReader(resp.Body, &got)
+	_, err := toml.DecodeReader(resp.Body, &got)
 	require.NoErrorf(t, err, "error decoding toml file")
 
 	expected := blueprint.Blueprint{
@@ -384,11 +346,7 @@ func TestBlueprintsInfoToml(t *testing.T) {
 }
 
 func TestNonExistentBlueprintsInfoToml(t *testing.T) {
-	tempdir, err := ioutil.TempDir("", "weldr-tests-")
-	require.NoError(t, err)
-	defer os.RemoveAll(tempdir)
-
-	api, _ := createWeldrAPI(tempdir, rpmmd_mock.BaseFixture)
+	api, _ := createWeldrAPI(t.TempDir(), rpmmd_mock.BaseFixture)
 	req := httptest.NewRequest("GET", "/api/v0/blueprints/info/test3-non?format=toml", nil)
 	recorder := httptest.NewRecorder()
 	api.ServeHTTP(recorder, req)
@@ -453,9 +411,7 @@ func TestBlueprintsFreeze(t *testing.T) {
 			{rpmmd_mock.BaseFixture, "/api/v0/blueprints/freeze/test,test2", http.StatusOK, `{"blueprints":[{"blueprint":{"name":"test","description":"Test","distro":"","version":"0.0.1","packages":[{"name":"dep-package1","version":"1.33-2.fc30.x86_64"},{"name":"dep-package3","version":"7:3.0.3-1.fc30.x86_64"}],"modules":[{"name":"dep-package2","version":"2.9-1.fc30.x86_64"}],"groups":[]}},{"blueprint":{"name":"test2","description":"Test","distro":"","version":"0.0.0","packages":[{"name":"dep-package1","version":"1.33-2.fc30.x86_64"},{"name":"dep-package3","version":"7:3.0.3-1.fc30.x86_64"}],"modules":[{"name":"dep-package2","version":"2.9-1.fc30.x86_64"}],"groups":[]}}],"errors":[]}`},
 		}
 
-		tempdir, err := ioutil.TempDir("", "weldr-tests-")
-		require.NoError(t, err)
-		defer os.RemoveAll(tempdir)
+		tempdir := t.TempDir()
 
 		for _, c := range cases {
 			api, _ := createWeldrAPI(tempdir, c.Fixture)
@@ -476,9 +432,7 @@ func TestBlueprintsFreeze(t *testing.T) {
 			{rpmmd_mock.BaseFixture, "/api/v0/blueprints/freeze/missing?format=toml", http.StatusOK, ""},
 		}
 
-		tempdir, err := ioutil.TempDir("", "weldr-tests-")
-		require.NoError(t, err)
-		defer os.RemoveAll(tempdir)
+		tempdir := t.TempDir()
 
 		for _, c := range cases {
 			api, _ := createWeldrAPI(tempdir, c.Fixture)
@@ -497,9 +451,7 @@ func TestBlueprintsFreeze(t *testing.T) {
 			{rpmmd_mock.BaseFixture, "/api/v0/blueprints/freeze/test,test2?format=toml", http.StatusBadRequest, "{\"status\":false,\"errors\":[{\"id\":\"HTTPError\",\"msg\":\"toml format only supported when requesting one blueprint\"}]}"},
 		}
 
-		tempdir, err := ioutil.TempDir("", "weldr-tests-")
-		require.NoError(t, err)
-		defer os.RemoveAll(tempdir)
+		tempdir := t.TempDir()
 
 		for _, c := range cases {
 			api, _ := createWeldrAPI(tempdir, c.Fixture)
@@ -521,9 +473,7 @@ func TestBlueprintsDiff(t *testing.T) {
 		{"GET", "/api/v0/blueprints/diff/test/NEWEST/WORKSPACE", ``, http.StatusOK, `{"diff":[{"new":{"Package":{"name":"systemd","version":"123"}},"old":null},{"new":null,"old":{"Package":{"name":"httpd","version":"2.4.*"}}}]}`},
 	}
 
-	tempdir, err := ioutil.TempDir("", "weldr-tests-")
-	require.NoError(t, err)
-	defer os.RemoveAll(tempdir)
+	tempdir := t.TempDir()
 
 	for _, c := range cases {
 		api, _ := createWeldrAPI(tempdir, rpmmd_mock.BaseFixture)
@@ -546,9 +496,7 @@ func TestBlueprintsDelete(t *testing.T) {
 		{"DELETE", "/api/v0/blueprints/delete/test3-non", ``, http.StatusBadRequest, `{"status":false,"errors":[{"id":"BlueprintsError","msg":"Unknown blueprint: test3-non"}]}`},
 	}
 
-	tempdir, err := ioutil.TempDir("", "weldr-tests-")
-	require.NoError(t, err)
-	defer os.RemoveAll(tempdir)
+	tempdir := t.TempDir()
 
 	for _, c := range cases {
 		api, _ := createWeldrAPI(tempdir, rpmmd_mock.BaseFixture)
@@ -559,11 +507,7 @@ func TestBlueprintsDelete(t *testing.T) {
 }
 
 func TestBlueprintsChanges(t *testing.T) {
-	tempdir, err := ioutil.TempDir("", "weldr-tests-")
-	require.NoError(t, err)
-	defer os.RemoveAll(tempdir)
-
-	api, _ := createWeldrAPI(tempdir, rpmmd_mock.BaseFixture)
+	api, _ := createWeldrAPI(t.TempDir(), rpmmd_mock.BaseFixture)
 	rand.Seed(time.Now().UnixNano())
 	// math/rand is good enough in this case
 	/* #nosec G404 */
@@ -590,9 +534,7 @@ func TestBlueprintsDepsolve(t *testing.T) {
 		{rpmmd_mock.BadDepsolve, http.StatusOK, `{"blueprints":[{"blueprint":{"name":"test","description":"Test","distro":"","version":"0.0.1","packages":[{"name":"dep-package1","version":"*"}],"groups":[],"modules":[{"name":"dep-package3","version":"*"}]},"dependencies":[]}],"errors":[{"id":"BlueprintsError","msg":"test: DNF error occured: DepsolveError: There was a problem depsolving ['go2rpm']: \n Problem: conflicting requests\n  - nothing provides askalono-cli needed by go2rpm-1-4.fc31.noarch"}]}`},
 	}
 
-	tempdir, err := ioutil.TempDir("", "weldr-tests-")
-	require.NoError(t, err)
-	defer os.RemoveAll(tempdir)
+	tempdir := t.TempDir()
 
 	for _, c := range cases {
 		api, _ := createWeldrAPI(tempdir, c.Fixture)
@@ -605,11 +547,7 @@ func TestBlueprintsDepsolve(t *testing.T) {
 // TestOldBlueprintsUndo run tests with blueprint changes after a service restart
 // Old blueprints are not saved, after a restart the changes are listed, but cannot be recalled
 func TestOldBlueprintsUndo(t *testing.T) {
-	tempdir, err := ioutil.TempDir("", "weldr-tests-")
-	require.NoError(t, err)
-	defer os.RemoveAll(tempdir)
-
-	api, _ := createWeldrAPI(tempdir, rpmmd_mock.OldChangesFixture)
+	api, _ := createWeldrAPI(t.TempDir(), rpmmd_mock.OldChangesFixture)
 	rand.Seed(time.Now().UnixNano())
 	// math/rand is good enough in this case
 	/* #nosec G404 */
@@ -643,11 +581,7 @@ func TestOldBlueprintsUndo(t *testing.T) {
 
 // TestNewBlueprintsUndo run tests with blueprint changes without a service restart
 func TestNewBlueprintsUndo(t *testing.T) {
-	tempdir, err := ioutil.TempDir("", "weldr-tests-")
-	require.NoError(t, err)
-	defer os.RemoveAll(tempdir)
-
-	api, _ := createWeldrAPI(tempdir, rpmmd_mock.BaseFixture)
+	api, _ := createWeldrAPI(t.TempDir(), rpmmd_mock.BaseFixture)
 	rand.Seed(time.Now().UnixNano())
 	// math/rand is good enough in this case
 	/* #nosec G404 */
@@ -949,9 +883,7 @@ func TestCompose(t *testing.T) {
 		},
 	}
 
-	tempdir, err := ioutil.TempDir("", "weldr-tests-")
-	require.NoError(t, err)
-	defer os.RemoveAll(tempdir)
+	tempdir := t.TempDir()
 
 	for _, c := range cases {
 		api, s := createWeldrAPI(tempdir, rpmmd_mock.NoComposesFixture)
@@ -997,9 +929,7 @@ func TestComposeDelete(t *testing.T) {
 		{"/api/v0/compose/delete/30000000-0000-0000-0000-000000000003,42000000-0000-0000-0000-000000000000", `{"uuids":[{"uuid":"30000000-0000-0000-0000-000000000003","status":true}],"errors":[{"id":"UnknownUUID","msg":"compose 42000000-0000-0000-0000-000000000000 doesn't exist"}]}`, []string{"30000000-0000-0000-0000-000000000000", "30000000-0000-0000-0000-000000000001", "30000000-0000-0000-0000-000000000002", "30000000-0000-0000-0000-000000000004"}},
 	}
 
-	tempdir, err := ioutil.TempDir("", "weldr-tests-")
-	require.NoError(t, err)
-	defer os.RemoveAll(tempdir)
+	tempdir := t.TempDir()
 
 	for _, c := range cases {
 		api, s := createWeldrAPI(tempdir, rpmmd_mock.BaseFixture)
@@ -1036,9 +966,7 @@ func TestComposeStatus(t *testing.T) {
 		t.Skip("This test is for internal testing only")
 	}
 
-	tempdir, err := ioutil.TempDir("", "weldr-tests-")
-	require.NoError(t, err)
-	defer os.RemoveAll(tempdir)
+	tempdir := t.TempDir()
 
 	for _, c := range cases {
 		api, _ := createWeldrAPI(tempdir, rpmmd_mock.BaseFixture)
@@ -1065,9 +993,7 @@ func TestComposeInfo(t *testing.T) {
 		t.Skip("This test is for internal testing only")
 	}
 
-	tempdir, err := ioutil.TempDir("", "weldr-tests-")
-	require.NoError(t, err)
-	defer os.RemoveAll(tempdir)
+	tempdir := t.TempDir()
 
 	for _, c := range cases {
 		api, _ := createWeldrAPI(tempdir, rpmmd_mock.BaseFixture)
@@ -1095,9 +1021,7 @@ func TestComposeLogs(t *testing.T) {
 		{"/api/v1/compose/results/30000000-0000-0000-0000-000000000002", "attachment; filename=30000000-0000-0000-0000-000000000002.tar", "application/x-tar", "30000000-0000-0000-0000-000000000002.json", "{\"sources\":{},\"pipeline\":{}}"},
 	}
 
-	tempdir, err := ioutil.TempDir("", "weldr-tests-")
-	require.NoError(t, err)
-	defer os.RemoveAll(tempdir)
+	tempdir := t.TempDir()
 
 	for _, c := range successCases {
 		api, _ := createWeldrAPI(tempdir, rpmmd_mock.BaseFixture)
@@ -1166,9 +1090,7 @@ func TestComposeLog(t *testing.T) {
 		t.Skip("This test is for internal testing only")
 	}
 
-	tempdir, err := ioutil.TempDir("", "weldr-tests-")
-	require.NoError(t, err)
-	defer os.RemoveAll(tempdir)
+	tempdir := t.TempDir()
 
 	for _, c := range cases {
 		api, _ := createWeldrAPI(tempdir, rpmmd_mock.BaseFixture)
@@ -1194,9 +1116,7 @@ func TestComposeQueue(t *testing.T) {
 		t.Skip("This test is for internal testing only")
 	}
 
-	tempdir, err := ioutil.TempDir("", "weldr-tests-")
-	require.NoError(t, err)
-	defer os.RemoveAll(tempdir)
+	tempdir := t.TempDir()
 
 	for _, c := range cases {
 		api, _ := createWeldrAPI(tempdir, c.Fixture)
@@ -1222,9 +1142,7 @@ func TestComposeFinished(t *testing.T) {
 		t.Skip("This test is for internal testing only")
 	}
 
-	tempdir, err := ioutil.TempDir("", "weldr-tests-")
-	require.NoError(t, err)
-	defer os.RemoveAll(tempdir)
+	tempdir := t.TempDir()
 
 	for _, c := range cases {
 		api, _ := createWeldrAPI(tempdir, c.Fixture)
@@ -1250,9 +1168,7 @@ func TestComposeFailed(t *testing.T) {
 		t.Skip("This test is for internal testing only")
 	}
 
-	tempdir, err := ioutil.TempDir("", "weldr-tests-")
-	require.NoError(t, err)
-	defer os.RemoveAll(tempdir)
+	tempdir := t.TempDir()
 
 	for _, c := range cases {
 		api, _ := createWeldrAPI(tempdir, c.Fixture)
@@ -1281,9 +1197,7 @@ func TestSourcesNew(t *testing.T) {
 		{"POST", "/api/v1/projects/source/new", `{"id": "fish","name":"fish repo","url": "https://download.opensuse.org/repositories/shells:/fish:/release:/3/Fedora_29/","type": "yum-baseurl","check_ssl": false,"check_gpg": false,"distros":["fedora-1"]}`, http.StatusBadRequest, `{"status":false, "errors":[{"id":"ProjectsError", "msg":"Invalid distributions: fedora-1"}]}`},
 	}
 
-	tempdir, err := ioutil.TempDir("", "weldr-tests-")
-	require.NoError(t, err)
-	defer os.RemoveAll(tempdir)
+	tempdir := t.TempDir()
 
 	for _, c := range cases {
 		api, _ := createWeldrAPI(tempdir, rpmmd_mock.BaseFixture)
@@ -1293,9 +1207,7 @@ func TestSourcesNew(t *testing.T) {
 }
 
 func TestSourcesNewTomlV0(t *testing.T) {
-	tempdir, err := ioutil.TempDir("", "weldr-tests-")
-	require.NoError(t, err)
-	defer os.RemoveAll(tempdir)
+	tempdir := t.TempDir()
 
 	sources := []string{`
 name = "fish"
@@ -1327,9 +1239,7 @@ check_gpg = false
 
 // Empty TOML, and invalid TOML should return an error
 func TestSourcesNewWrongTomlV0(t *testing.T) {
-	tempdir, err := ioutil.TempDir("", "weldr-tests-")
-	require.NoError(t, err)
-	defer os.RemoveAll(tempdir)
+	tempdir := t.TempDir()
 
 	sources := []string{``, `
 url = "https://download.opensuse.org/repositories/shells:/fish:/release:/3/Fedora_29/"
@@ -1352,9 +1262,7 @@ check_gpg = false
 
 // TestSourcesNewTomlV1 tests the v1 sources API with id and name
 func TestSourcesNewTomlV1(t *testing.T) {
-	tempdir, err := ioutil.TempDir("", "weldr-tests-")
-	require.NoError(t, err)
-	defer os.RemoveAll(tempdir)
+	tempdir := t.TempDir()
 
 	sources := []string{`
 id = "fish"
@@ -1393,10 +1301,6 @@ check_gpg = false
 }
 
 func TestSourcesInfoTomlV1(t *testing.T) {
-	tempdir, err := ioutil.TempDir("", "weldr-tests-")
-	require.NoError(t, err)
-	defer os.RemoveAll(tempdir)
-
 	source := `
 id = "fish"
 name = "fish"
@@ -1411,7 +1315,7 @@ rhsm = true
 	req.Header.Set("Content-Type", "text/x-toml")
 	recorder := httptest.NewRecorder()
 
-	api, _ := createWeldrAPI(tempdir, rpmmd_mock.BaseFixture)
+	api, _ := createWeldrAPI(t.TempDir(), rpmmd_mock.BaseFixture)
 	api.ServeHTTP(recorder, req)
 
 	r := recorder.Result()
@@ -1422,9 +1326,7 @@ rhsm = true
 
 // TestSourcesNewWrongTomlV1 Tests that Empty TOML, and invalid TOML should return an error
 func TestSourcesNewWrongTomlV1(t *testing.T) {
-	tempdir, err := ioutil.TempDir("", "weldr-tests-")
-	require.NoError(t, err)
-	defer os.RemoveAll(tempdir)
+	tempdir := t.TempDir()
 
 	sources := []string{``, `
 url = "https://download.opensuse.org/repositories/shells:/fish:/release:/3/Fedora_29/"
@@ -1458,13 +1360,9 @@ check_gpg = false
 }
 
 func TestSourcesInfo(t *testing.T) {
-	tempdir, err := ioutil.TempDir("", "weldr-tests-")
-	require.NoError(t, err)
-	defer os.RemoveAll(tempdir)
-
 	sourceStr := `{"name":"fish","type":"yum-baseurl","url":"https://download.opensuse.org/repositories/shells:/fish:/release:/3/Fedora_29/","check_gpg":false,"check_ssl":false,"system":false}`
 
-	api, _ := createWeldrAPI(tempdir, rpmmd_mock.BaseFixture)
+	api, _ := createWeldrAPI(t.TempDir(), rpmmd_mock.BaseFixture)
 	test.SendHTTP(api, true, "POST", "/api/v0/projects/source/new", sourceStr)
 	test.TestRoute(t, api, true, "GET", "/api/v0/projects/source/info/fish", ``, 200, `{"sources":{"fish":`+sourceStr+`},"errors":[]}`)
 	test.TestRoute(t, api, true, "GET", "/api/v0/projects/source/info/fish?format=json", ``, 200, `{"sources":{"fish":`+sourceStr+`},"errors":[]}`)
@@ -1472,13 +1370,9 @@ func TestSourcesInfo(t *testing.T) {
 }
 
 func TestSourcesInfoToml(t *testing.T) {
-	tempdir, err := ioutil.TempDir("", "weldr-tests-")
-	require.NoError(t, err)
-	defer os.RemoveAll(tempdir)
-
 	sourceStr := `{"name":"fish","type":"yum-baseurl","url":"https://download.opensuse.org/repositories/shells:/fish:/release:/3/Fedora_29/","check_gpg":false,"check_ssl":false,"system":false}`
 
-	api, _ := createWeldrAPI(tempdir, rpmmd_mock.BaseFixture)
+	api, _ := createWeldrAPI(t.TempDir(), rpmmd_mock.BaseFixture)
 	test.SendHTTP(api, true, "POST", "/api/v0/projects/source/new", sourceStr)
 
 	req := httptest.NewRequest("GET", "/api/v0/projects/source/info/fish?format=toml", nil)
@@ -1487,7 +1381,7 @@ func TestSourcesInfoToml(t *testing.T) {
 	resp := recorder.Result()
 
 	var sources map[string]store.SourceConfig
-	_, err = toml.DecodeReader(resp.Body, &sources)
+	_, err := toml.DecodeReader(resp.Body, &sources)
 	require.NoErrorf(t, err, "error decoding toml file")
 
 	expected := map[string]store.SourceConfig{
@@ -1514,9 +1408,7 @@ func TestSourcesDelete(t *testing.T) {
 		{"DELETE", "/api/v0/projects/source/delete/unknown", ``, http.StatusBadRequest, `{"status":false,"errors":[{"id":"UnknownSource","msg":"unknown is not a valid source."}]}`},
 	}
 
-	tempdir, err := ioutil.TempDir("", "weldr-tests-")
-	require.NoError(t, err)
-	defer os.RemoveAll(tempdir)
+	tempdir := t.TempDir()
 
 	for _, c := range cases {
 		api, _ := createWeldrAPI(tempdir, rpmmd_mock.BaseFixture)
@@ -1540,9 +1432,7 @@ func TestProjectsDepsolve(t *testing.T) {
 		{rpmmd_mock.BaseFixture, "/api/v0/projects/depsolve/fish?distro=fedora-1", http.StatusBadRequest, `{"status":false,"errors":[{"id":"DistroError","msg":"Invalid distro: fedora-1"}]}`},
 	}
 
-	tempdir, err := ioutil.TempDir("", "weldr-tests-")
-	require.NoError(t, err)
-	defer os.RemoveAll(tempdir)
+	tempdir := t.TempDir()
 
 	for _, c := range cases {
 		api, _ := createWeldrAPI(tempdir, c.Fixture)
@@ -1567,9 +1457,7 @@ func TestProjectsInfo(t *testing.T) {
 		{rpmmd_mock.BaseFixture, "/api/v0/projects/info/package16?distro=fedora-1", http.StatusBadRequest, `{"status":false,"errors":[{"id":"DistroError","msg":"Invalid distro: fedora-1"}]}`},
 	}
 
-	tempdir, err := ioutil.TempDir("", "weldr-tests-")
-	require.NoError(t, err)
-	defer os.RemoveAll(tempdir)
+	tempdir := t.TempDir()
 
 	for _, c := range cases {
 		api, _ := createWeldrAPI(tempdir, c.Fixture)
@@ -1596,9 +1484,7 @@ func TestModulesInfo(t *testing.T) {
 		{rpmmd_mock.BaseFixture, "/api/v1/modules/info/package16?distro=fedora-1", http.StatusBadRequest, `{"status":false,"errors":[{"id":"DistroError","msg":"Invalid distro: fedora-1"}]}`},
 	}
 
-	tempdir, err := ioutil.TempDir("", "weldr-tests-")
-	require.NoError(t, err)
-	defer os.RemoveAll(tempdir)
+	tempdir := t.TempDir()
 
 	for _, c := range cases {
 		api, _ := createWeldrAPI(tempdir, c.Fixture)
@@ -1621,9 +1507,7 @@ func TestProjectsList(t *testing.T) {
 		{rpmmd_mock.BaseFixture, "/api/v0/projects/list?distro=fedora-1&offset=1&limit=1", http.StatusBadRequest, `{"status":false,"errors":[{"id":"DistroError","msg":"Invalid distro: fedora-1"}]}`},
 	}
 
-	tempdir, err := ioutil.TempDir("", "weldr-tests-")
-	require.NoError(t, err)
-	defer os.RemoveAll(tempdir)
+	tempdir := t.TempDir()
 
 	for _, c := range cases {
 		api, _ := createWeldrAPI(tempdir, c.Fixture)
@@ -1649,9 +1533,7 @@ func TestModulesList(t *testing.T) {
 		{rpmmd_mock.BaseFixture, "/api/v0/modules/list/package2*,package16?distro=fedora-1&offset=1&limit=1", http.StatusBadRequest, `{"status":false,"errors":[{"id":"DistroError","msg":"Invalid distro: fedora-1"}]}`},
 	}
 
-	tempdir, err := ioutil.TempDir("", "weldr-tests-")
-	require.NoError(t, err)
-	defer os.RemoveAll(tempdir)
+	tempdir := t.TempDir()
 
 	for _, c := range cases {
 		api, _ := createWeldrAPI(tempdir, c.Fixture)
@@ -1729,9 +1611,7 @@ func TestComposeTypes_ImageTypeDenylist(t *testing.T) {
 		},
 	}
 
-	tempdir, err := ioutil.TempDir("", "weldr-tests-")
-	require.NoError(t, err)
-	defer os.RemoveAll(tempdir)
+	tempdir := t.TempDir()
 
 	for _, c := range cases {
 		api, _ := createWeldrAPI2(tempdir, rpmmd_mock.BaseFixture, c.ImageTypeDenylist)
@@ -1891,9 +1771,7 @@ func TestComposePOST_ImageTypeDenylist(t *testing.T) {
 		},
 	}
 
-	tempdir, err := ioutil.TempDir("", "weldr-tests-")
-	require.NoError(t, err)
-	defer os.RemoveAll(tempdir)
+	tempdir := t.TempDir()
 
 	for _, c := range cases {
 		api, s := createWeldrAPI2(tempdir, rpmmd_mock.NoComposesFixture, c.imageTypeDenylist)
diff --git a/internal/weldr/compose_test.go b/internal/weldr/compose_test.go
index d63de4ace..a04f27039 100644
--- a/internal/weldr/compose_test.go
+++ b/internal/weldr/compose_test.go
@@ -3,7 +3,6 @@ package weldr
 import (
 	"context"
 	"encoding/json"
-	"io/ioutil"
 	"os"
 	"testing"
 
@@ -22,11 +21,7 @@ func TestComposeStatusFromLegacyError(t *testing.T) {
 		t.Skip("This test is for internal testing only")
 	}
 
-	tempdir, err := ioutil.TempDir("", "weldr-tests-")
-	require.NoError(t, err)
-	defer os.RemoveAll(tempdir)
-
-	api, _ := createWeldrAPI(tempdir, rpmmd_mock.BaseFixture)
+	api, _ := createWeldrAPI(t.TempDir(), rpmmd_mock.BaseFixture)
 
 	distroStruct := test_distro.New()
 	arch, err := distroStruct.GetArch(test_distro.TestArchName)
@@ -68,11 +63,7 @@ func TestComposeStatusFromJobError(t *testing.T) {
 		t.Skip("This test is for internal testing only")
 	}
 
-	tempdir, err := ioutil.TempDir("", "weldr-tests-")
-	require.NoError(t, err)
-	defer os.RemoveAll(tempdir)
-
-	api, _ := createWeldrAPI(tempdir, rpmmd_mock.BaseFixture)
+	api, _ := createWeldrAPI(t.TempDir(), rpmmd_mock.BaseFixture)
 
 	distroStruct := test_distro.New()
 	arch, err := distroStruct.GetArch(test_distro.TestArchName)
diff --git a/internal/worker/server_test.go b/internal/worker/server_test.go
index 36bf653ba..0e70f969a 100644
--- a/internal/worker/server_test.go
+++ b/internal/worker/server_test.go
@@ -4,10 +4,8 @@ import (
 	"context"
 	"encoding/json"
 	"fmt"
-	"io/ioutil"
 	"net/http"
 	"net/http/httptest"
-	"os"
 	"strings"
 	"testing"
 	"time"
@@ -40,11 +38,7 @@ func newTestServer(t *testing.T, tempdir string, jobRequestTimeout time.Duration
 
 // Ensure that the status request returns OK.
 func TestStatus(t *testing.T) {
-	tempdir, err := ioutil.TempDir("", "worker-tests-")
-	require.NoError(t, err)
-	defer os.RemoveAll(tempdir)
-
-	server := newTestServer(t, tempdir, time.Duration(0), "/api/worker/v1")
+	server := newTestServer(t, t.TempDir(), time.Duration(0), "/api/worker/v1")
 	handler := server.Handler()
 	test.TestRoute(t, handler, false, "GET", "/api/worker/v1/status", ``, http.StatusOK, `{"status":"OK", "href": "/api/worker/v1/status", "kind":"Status"}`, "message", "id")
 }
@@ -70,9 +64,7 @@ func TestErrors(t *testing.T) {
 		{"PATCH", "/api/worker/v1/jobs/aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa", `{"status":"FINISHED"}`, http.StatusNotFound},
 	}
 
-	tempdir, err := ioutil.TempDir("", "worker-tests-")
-	require.NoError(t, err)
-	defer os.RemoveAll(tempdir)
+	tempdir := t.TempDir()
 
 	for _, c := range cases {
 		server := newTestServer(t, tempdir, time.Duration(0), "/api/worker/v1")
@@ -102,9 +94,7 @@ func TestErrorsAlteredBasePath(t *testing.T) {
 		{"PATCH", "/api/image-builder-worker/v1/jobs/aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa", `{"status":"FINISHED"}`, http.StatusNotFound},
 	}
 
-	tempdir, err := ioutil.TempDir("", "worker-tests-")
-	require.NoError(t, err)
-	defer os.RemoveAll(tempdir)
+	tempdir := t.TempDir()
 
 	for _, c := range cases {
 		server := newTestServer(t, tempdir, time.Duration(0), "/api/image-builder-worker/v1")
@@ -114,10 +104,6 @@ func TestErrorsAlteredBasePath(t *testing.T) {
 }
 
 func TestCreate(t *testing.T) {
-	tempdir, err := ioutil.TempDir("", "worker-tests-")
-	require.NoError(t, err)
-	defer os.RemoveAll(tempdir)
-
 	distroStruct := test_distro.New()
 	arch, err := distroStruct.GetArch(test_distro.TestArchName)
 	if err != nil {
@@ -131,7 +117,7 @@ func TestCreate(t *testing.T) {
 	if err != nil {
 		t.Fatalf("error creating osbuild manifest: %v", err)
 	}
-	server := newTestServer(t, tempdir, time.Duration(0), "/api/worker/v1")
+	server := newTestServer(t, t.TempDir(), time.Duration(0), "/api/worker/v1")
 	handler := server.Handler()
 
 	_, err = server.EnqueueOSBuild(arch.Name(), &worker.OSBuildJob{Manifest: manifest}, "")
@@ -143,10 +129,6 @@ func TestCreate(t *testing.T) {
 }
 
 func TestCancel(t *testing.T) {
-	tempdir, err := ioutil.TempDir("", "worker-tests-")
-	require.NoError(t, err)
-	defer os.RemoveAll(tempdir)
-
 	distroStruct := test_distro.New()
 	arch, err := distroStruct.GetArch(test_distro.TestArchName)
 	if err != nil {
@@ -160,7 +142,7 @@ func TestCancel(t *testing.T) {
 	if err != nil {
 		t.Fatalf("error creating osbuild manifest: %v", err)
 	}
-	server := newTestServer(t, tempdir, time.Duration(0), "/api/worker/v1")
+	server := newTestServer(t, t.TempDir(), time.Duration(0), "/api/worker/v1")
 	handler := server.Handler()
 
 	jobId, err := server.EnqueueOSBuild(arch.Name(), &worker.OSBuildJob{Manifest: manifest}, "")
@@ -184,10 +166,6 @@ func TestCancel(t *testing.T) {
 }
 
 func TestUpdate(t *testing.T) {
-	tempdir, err := ioutil.TempDir("", "worker-tests-")
-	require.NoError(t, err)
-	defer os.RemoveAll(tempdir)
-
 	distroStruct := test_distro.New()
 	arch, err := distroStruct.GetArch(test_distro.TestArchName)
 	if err != nil {
@@ -201,7 +179,7 @@ func TestUpdate(t *testing.T) {
 	if err != nil {
 		t.Fatalf("error creating osbuild manifest: %v", err)
 	}
-	server := newTestServer(t, tempdir, time.Duration(0), "/api/worker/v1")
+	server := newTestServer(t, t.TempDir(), time.Duration(0), "/api/worker/v1")
 	handler := server.Handler()
 
 	jobId, err := server.EnqueueOSBuild(arch.Name(), &worker.OSBuildJob{Manifest: manifest}, "")
@@ -230,10 +208,7 @@ func TestArgs(t *testing.T) {
 	manifest, err := imageType.Manifest(nil, distro.ImageOptions{Size: imageType.Size(0)}, nil, nil, 0)
 	require.NoError(t, err)
 
-	tempdir, err := ioutil.TempDir("", "worker-tests-")
-	require.NoError(t, err)
-	defer os.RemoveAll(tempdir)
-	server := newTestServer(t, tempdir, time.Duration(0), "/api/worker/v1")
+	server := newTestServer(t, t.TempDir(), time.Duration(0), "/api/worker/v1")
 
 	job := worker.OSBuildJob{
 		Manifest:  manifest,
@@ -257,10 +232,6 @@ func TestArgs(t *testing.T) {
 }
 
 func TestUpload(t *testing.T) {
-	tempdir, err := ioutil.TempDir("", "worker-tests-")
-	require.NoError(t, err)
-	defer os.RemoveAll(tempdir)
-
 	distroStruct := test_distro.New()
 	arch, err := distroStruct.GetArch(test_distro.TestArchName)
 	if err != nil {
@@ -274,7 +245,7 @@ func TestUpload(t *testing.T) {
 	if err != nil {
 		t.Fatalf("error creating osbuild manifest: %v", err)
 	}
-	server := newTestServer(t, tempdir, time.Duration(0), "/api/worker/v1")
+	server := newTestServer(t, t.TempDir(), time.Duration(0), "/api/worker/v1")
 	handler := server.Handler()
 
 	jobID, err := server.EnqueueOSBuild(arch.Name(), &worker.OSBuildJob{Manifest: manifest}, "")
@@ -291,10 +262,6 @@ func TestUpload(t *testing.T) {
 }
 
 func TestUploadAlteredBasePath(t *testing.T) {
-	tempdir, err := ioutil.TempDir("", "worker-tests-")
-	require.NoError(t, err)
-	defer os.RemoveAll(tempdir)
-
 	distroStruct := test_distro.New()
 	arch, err := distroStruct.GetArch(test_distro.TestArchName)
 	if err != nil {
@@ -308,7 +275,7 @@ func TestUploadAlteredBasePath(t *testing.T) {
 	if err != nil {
 		t.Fatalf("error creating osbuild manifest: %v", err)
 	}
-	server := newTestServer(t, tempdir, time.Duration(0), "/api/image-builder-worker/v1")
+	server := newTestServer(t, t.TempDir(), time.Duration(0), "/api/image-builder-worker/v1")
 	handler := server.Handler()
 
 	jobID, err := server.EnqueueOSBuild(arch.Name(), &worker.OSBuildJob{Manifest: manifest}, "")
@@ -325,9 +292,7 @@ func TestUploadAlteredBasePath(t *testing.T) {
 }
 
 func TestOAuth(t *testing.T) {
-	tempdir, err := ioutil.TempDir("", "worker-tests-")
-	require.NoError(t, err)
-	defer os.RemoveAll(tempdir)
+	tempdir := t.TempDir()
 
 	q, err := fsjobqueue.New(tempdir)
 	require.NoError(t, err)
@@ -414,16 +379,12 @@ func TestOAuth(t *testing.T) {
 }
 
 func TestTimeout(t *testing.T) {
-	tempdir, err := ioutil.TempDir("", "worker-tests-")
-	require.NoError(t, err)
-	defer os.RemoveAll(tempdir)
-
 	distroStruct := test_distro.New()
 	arch, err := distroStruct.GetArch(test_distro.TestArchName)
 	if err != nil {
 		t.Fatalf("error getting arch from distro: %v", err)
 	}
-	server := newTestServer(t, tempdir, time.Millisecond*10, "/api/image-builder-worker/v1")
+	server := newTestServer(t, t.TempDir(), time.Millisecond*10, "/api/image-builder-worker/v1")
 
 	_, _, _, _, _, err = server.RequestJob(context.Background(), arch.Name(), []string{"osbuild"}, []string{""})
 	require.Equal(t, jobqueue.ErrDequeueTimeout, err)
@@ -433,16 +394,12 @@ func TestTimeout(t *testing.T) {
 }
 
 func TestRequestJobById(t *testing.T) {
-	tempdir, err := ioutil.TempDir("", "worker-tests-")
-	require.NoError(t, err)
-	defer os.RemoveAll(tempdir)
-
 	distroStruct := test_distro.New()
 	arch, err := distroStruct.GetArch(test_distro.TestArchName)
 	if err != nil {
 		t.Fatalf("error getting arch from distro: %v", err)
 	}
-	server := newTestServer(t, tempdir, time.Duration(0), "/api/worker/v1")
+	server := newTestServer(t, t.TempDir(), time.Duration(0), "/api/worker/v1")
 	handler := server.Handler()
 
 	depsolveJobId, err := server.EnqueueDepsolve(&worker.DepsolveJob{}, "")
@@ -481,12 +438,9 @@ func TestRequestJobById(t *testing.T) {
 // for the new job.
 func TestMixedOSBuildJob(t *testing.T) {
 	require := require.New(t)
-	tempdir, err := ioutil.TempDir("", "worker-tests-")
-	require.NoError(err)
-	defer os.RemoveAll(tempdir)
 
 	emptyManifestV2 := distro.Manifest(`{"version":"2","pipelines":{}}`)
-	server := newTestServer(t, tempdir, time.Millisecond*10, "/")
+	server := newTestServer(t, t.TempDir(), time.Millisecond*10, "/")
 	fbPipelines := &worker.PipelineNames{Build: distro.BuildPipelinesFallback(), Payload: distro.PayloadPipelinesFallback()}
 
 	oldJob := worker.OSBuildJob{
@@ -623,12 +577,9 @@ func TestMixedOSBuildJob(t *testing.T) {
 // for the new job.
 func TestMixedOSBuildKojiJob(t *testing.T) {
 	require := require.New(t)
-	tempdir, err := ioutil.TempDir("", "worker-tests-")
-	require.NoError(err)
-	defer os.RemoveAll(tempdir)
 
 	emptyManifestV2 := distro.Manifest(`{"version":"2","pipelines":{}}`)
-	server := newTestServer(t, tempdir, time.Duration(0), "/api/worker/v1")
+	server := newTestServer(t, t.TempDir(), time.Duration(0), "/api/worker/v1")
 	fbPipelines := &worker.PipelineNames{Build: distro.BuildPipelinesFallback(), Payload: distro.PayloadPipelinesFallback()}
 
 	enqueueKojiJob := func(job *worker.OSBuildKojiJob) uuid.UUID {
@@ -656,7 +607,7 @@ func TestMixedOSBuildKojiJob(t *testing.T) {
 	newJobID := enqueueKojiJob(&newJob)
 
 	var oldJobRead worker.OSBuildKojiJob
-	err = server.OSBuildKojiJob(oldJobID, &oldJobRead)
+	err := server.OSBuildKojiJob(oldJobID, &oldJobRead)
 	require.NoError(err)
 	require.NotNil(oldJobRead.PipelineNames)
 	// OldJob gets default pipeline names when read
@@ -779,16 +730,12 @@ func TestMixedOSBuildKojiJob(t *testing.T) {
 }
 
 func TestDepsolveLegacyErrorConversion(t *testing.T) {
-	tempdir, err := ioutil.TempDir("", "worker-tests-")
-	require.NoError(t, err)
-	defer os.RemoveAll(tempdir)
-
 	distroStruct := test_distro.New()
 	arch, err := distroStruct.GetArch(test_distro.TestArchName)
 	if err != nil {
 		t.Fatalf("error getting arch from distro: %v", err)
 	}
-	server := newTestServer(t, tempdir, time.Duration(0), "/api/worker/v1")
+	server := newTestServer(t, t.TempDir(), time.Duration(0), "/api/worker/v1")
 
 	depsolveJobId, err := server.EnqueueDepsolve(&worker.DepsolveJob{}, "")
 	require.NoError(t, err)
@@ -822,12 +769,9 @@ func TestDepsolveLegacyErrorConversion(t *testing.T) {
 // that both kinds of errors can be read back
 func TestMixedOSBuildJobErrors(t *testing.T) {
 	require := require.New(t)
-	tempdir, err := ioutil.TempDir("", "worker-tests-")
-	require.NoError(err)
-	defer os.RemoveAll(tempdir)
 
 	emptyManifestV2 := distro.Manifest(`{"version":"2","pipelines":{}}`)
-	server := newTestServer(t, tempdir, time.Millisecond*10, "/")
+	server := newTestServer(t, t.TempDir(), time.Millisecond*10, "/")
 
 	oldJob := worker.OSBuildJob{
 		Manifest:  emptyManifestV2,
@@ -929,12 +873,9 @@ func TestMixedOSBuildJobErrors(t *testing.T) {
 // that both kinds of errors can be read back
 func TestMixedOSBuildKojiJobErrors(t *testing.T) {
 	require := require.New(t)
-	tempdir, err := ioutil.TempDir("", "worker-tests-")
-	require.NoError(err)
-	defer os.RemoveAll(tempdir)
 
 	emptyManifestV2 := distro.Manifest(`{"version":"2","pipelines":{}}`)
-	server := newTestServer(t, tempdir, time.Duration(0), "/api/worker/v1")
+	server := newTestServer(t, t.TempDir(), time.Duration(0), "/api/worker/v1")
 
 	enqueueKojiJob := func(job *worker.OSBuildKojiJob) uuid.UUID {
 		initJob := new(worker.KojiInitJob)
@@ -961,7 +902,7 @@ func TestMixedOSBuildKojiJobErrors(t *testing.T) {
 	newJobID := enqueueKojiJob(&newJob)
 
 	oldJobRead := new(worker.OSBuildKojiJob)
-	err = server.OSBuildKojiJob(oldJobID, oldJobRead)
+	err := server.OSBuildKojiJob(oldJobID, oldJobRead)
 	require.NoError(err)
 	// Not entirely equal
 	require.NotEqual(oldJob, oldJobRead)
diff --git a/test/cases/ostree-rebase.sh b/test/cases/ostree-rebase.sh
index 5bf60bb09..c9c73b0e5 100755
--- a/test/cases/ostree-rebase.sh
+++ b/test/cases/ostree-rebase.sh
@@ -125,8 +125,8 @@ case "${ID}-${VERSION_ID}" in
         ;;
     "centos-9")
         OSTREE_REF="centos/9/${ARCH}/edge"
-        OS_VARIANT="centos9"
-        BOOT_LOCATION="${COMPOSE_URL:-}/compose/BaseOS/x86_64/os/"
+        OS_VARIANT="centos-stream9"
+        BOOT_LOCATION="https://odcs.stream.centos.org/production/latest-CentOS-Stream/compose/BaseOS/x86_64/os/"
         PARENT_REF="centos/9/${ARCH}/edge"
         ;;
     *)