Repository: osbuild/osbuild-composer
Pull request #2736 (2022-06-10 09:36:56)

jobqueue: Move jobqueue out of internal packages
TODO:
- [x] Move maintenance related queries out of dbjoqueue

teg: Should we use `/pkg`?

Ref https://github.com/golang-standards/project-layout#pkg
bcl: > Should we use `/pkg`?

I think `/jobqueue` is just fine, AFAIK that page of suggestions is kind of a catch-all of the way people have done things throughout the history of go and not all of it applies anymore. It seems more natural to me to import from the top level of the module instead of having to add `/pkg/` to it.

teg: I'm fine with whatever is the current convention
Gundersanne: > Should we use `/pkg`?
> 
> Ref https://github.com/golang-standards/project-layout#pkg

I thought the same, but after reading issue 10 on that project I wasn't entirely sure anymore. Looking at some of the projects we depend on, some have it, others don't.
ondrejbudai: > It's ok not to use it if your app project is really small and where an extra level of nesting doesn't add much value (unless you really want to :-)). Think about it when it's getting big enough and your root directory gets pretty busy (especially if you have a lot of non-Go app components).
>
> *Taken from: https://github.com/golang-standards/project-layout#pkg*

This seems like a good argument for using `/pkg` as we have plenty of non-go code in the repository. Would be great if we can restrict go code to just `cmd`, `internal` and `pkg`. WDYT?
Gundersanne: > > It's ok not to use it if your app project is really small and where an extra level of nesting doesn't add much value (unless you really want to :-)). Think about it when it's getting big enough and your root directory gets pretty busy (especially if you have a lot of non-Go app components).
> > _Taken from: https://github.com/golang-standards/project-layout#pkg_
> 
> This seems like a good argument for using `/pkg` as we have plenty of non-go code in the repository. Would be great if we can restrict go code to just `cmd`, `internal` and `pkg`. WDYT?

I'm happy with either. We already have a lot of stuff in our top level directory so it makes sense.
teg: I would, as an extra bonus, be interested to understand why we made different choices and if we should unify? We should be having very similar requirements.
teg: Thanks for the detailed answer, that's very interesting!

cc @diaasami 
lzap: I will file PRs for pool/log abstraction as soon as this gets merged. Looks like ready for review 🥽 
ondrejbudai: > Hm, I think we need the string formatting for on-prem, json messages in the journal wouldn't be ideal imo, plain text is still our default too.

Definitely but if we can use `fmt.Sprintf`, it works for me.

> Also it's not packaged in fedora (but that might not be a huge deal anymore soon as we might switch to vendoring).


I think it is: https://src.fedoraproject.org/rpms/golang-github-logr

lzap: > json messages in the journal wouldn't be ideal imo

Oh no, please don't say that! :) Syslog is indeed text based, journald is actually meant for structured logging. That's what I was advocating for a very long time in Satellite which now has a fully structured logging (request id, session id, login name, account info...). It is very useful.

But I get the point, your app does have text-based logging approach and that's fine.

> I think it is: https://src.fedoraproject.org/rpms/golang-github-logr

Indeed, however, composer will also need https://github.com/bombsimon/logrusr and that one does not exist, so I think that is the blocker.

Allright, that leaves us with:

```
$ ys golang | grep logging
Poslední kontrola metadat: před 0:44:54, St 22. června 2022, 12:38:39.
golang-github-grpc-ecosystem-middleware-devel.noarch : Golang gRPC Middlewares: interceptor chaining, auth, logging, retries and more
golang-github-hashicorp-logutils-devel.noarch : Utilities for slightly better logging in go (golang)
golang-github-op-logging-devel.noarch : Golang logging library
golang-github-schollz-logger-devel.noarch : Simplistic, opinionated logging for Golang
compat-golang-github-sirupsen-logrus-devel.noarch : Structured, pluggable logging for Go
compat-golang-gopkg-inconshreveable-log15-2-devel.noarch : Structured, composable logging for go
golang-github-apex-log-devel.noarch : Structured logging package for Go
golang-github-apex-log-handlers-kinesis-devel.noarch : Structured logging package for Go
golang-github-coreos-log-devel.noarch : Go logging library with systemd journal support
golang-github-devigned-tab-devel.noarch : OpenTracing and OpenCensus abstraction for tracing and logging
golang-github-facebookincubator-flog-devel.noarch : A Go library for logging
golang-github-gobuffalo-logger-devel.noarch : Common logging interface for the Buffalo ecosystem
golang-github-googlecloudplatform-guest-logging-devel.noarch : Common logger used by GCP guest environment
golang-github-haproxytech-logger-devel.noarch : Go package that provides interface for logging
golang-github-hashicorp-hclog.x86_64 : Common logging package for hashicorp tools
golang-github-hashicorp-hclog-devel.noarch : Common logging package for hashicorp tools
golang-github-inconshreveable-log15-devel.noarch : Structured, composable logging for go
golang-github-ipfs-log-devel.noarch : A logging library used by go-ipfs
golang-github-istio-glog-devel.noarch : A replacement for the glog package that directs logging to zap
golang-github-jpillora-requestlog-devel.noarch : Simple request logging
golang-github-logr-devel.noarch : Simple logging interface for Go
golang-github-projectdiscovery-gologger-devel.noarch : A simple layer for leveled logging in go
golang-github-quay-zlog-devel.noarch : Contextual logging for go
golang-github-revel-log15-devel.noarch : Structured, composable logging for Go
golang-github-sirupsen-logrus-devel.noarch : Structured, pluggable logging for Go
golang-github-spacemonkeygo-spacelog-devel.noarch : Hierarchical, leveled, and structured logging library for go
golang-github-spf13-jwalterweatherman-devel.noarch : Seamless printing to the terminal and logging to a file
golang-uber-zap-devel.noarch : Blazing fast, structured, leveled logging in Go
```

In that case, I will create just a simple interface custom-built for your use case around logrus API. It looks like the code only uses Error/Info/Debug so let's start with that. We can discuss the details in the upcoming PR.
Gundersanne:  
> > Also it's not packaged in fedora (but that might not be a huge deal anymore soon as we might switch to vendoring).
> 
> I think it is: https://src.fedoraproject.org/rpms/golang-github-logr

Ugh I missed that :facepalm: 

Anyway we also have an adapter for echo https://github.com/osbuild/osbuild-composer/blob/main/internal/common/echo_logrus.go ; something similar would work. 
lzap: Yeah that's exactly what I have on my mind, it even supports Infof and Infoj (j stands for JSON) for structured logging which I think can be really useful for jobs (job uuid searchable via CloudWatch can be quite useful. I will write a simple adapter similar to this one.

Patch:
diff --git a/Makefile b/Makefile
index 5a22e1f37..f371da4cd 100644
--- a/Makefile
+++ b/Makefile
@@ -126,6 +126,7 @@ build:
 	go test -c -tags=integration -o bin/osbuild-auth-tests ./cmd/osbuild-auth-tests/
 	go test -c -tags=integration -o bin/osbuild-koji-tests ./cmd/osbuild-koji-tests/
 	go test -c -tags=integration -o bin/osbuild-composer-dbjobqueue-tests ./cmd/osbuild-composer-dbjobqueue-tests/
+	go test -c -tags=integration -o bin/osbuild-composer-maintenance-tests ./cmd/osbuild-service-maintenance/
 	go test -c -tags=integration -o bin/osbuild-composer-manifest-tests ./cmd/osbuild-composer-manifest-tests/
 
 .PHONY: install
diff --git a/cmd/osbuild-composer/composer.go b/cmd/osbuild-composer/composer.go
index 40c380daf..b6f0df851 100644
--- a/cmd/osbuild-composer/composer.go
+++ b/cmd/osbuild-composer/composer.go
@@ -16,6 +16,8 @@ import (
 	"syscall"
 	"time"
 
+	"github.com/osbuild/osbuild-composer/pkg/jobqueue"
+	"github.com/osbuild/osbuild-composer/pkg/jobqueue/dbjobqueue"
 	"github.com/prometheus/client_golang/prometheus/promhttp"
 	logrus "github.com/sirupsen/logrus"
 
@@ -24,8 +26,6 @@ import (
 	v2 "github.com/osbuild/osbuild-composer/internal/cloudapi/v2"
 	"github.com/osbuild/osbuild-composer/internal/distroregistry"
 	"github.com/osbuild/osbuild-composer/internal/dnfjson"
-	"github.com/osbuild/osbuild-composer/internal/jobqueue"
-	"github.com/osbuild/osbuild-composer/internal/jobqueue/dbjobqueue"
 	"github.com/osbuild/osbuild-composer/internal/jobqueue/fsjobqueue"
 	"github.com/osbuild/osbuild-composer/internal/kojiapi"
 	"github.com/osbuild/osbuild-composer/internal/weldr"
diff --git a/cmd/osbuild-service-maintenance/db.go b/cmd/osbuild-service-maintenance/db.go
index 5f8f8ebbc..eb6adf543 100644
--- a/cmd/osbuild-service-maintenance/db.go
+++ b/cmd/osbuild-service-maintenance/db.go
@@ -1,29 +1,157 @@
 package main
 
 import (
+	"context"
 	"fmt"
 	"time"
 
+	"github.com/google/uuid"
+	"github.com/jackc/pgx/v4"
 	"github.com/sirupsen/logrus"
 
-	"github.com/osbuild/osbuild-composer/internal/jobqueue/dbjobqueue"
 	"github.com/osbuild/osbuild-composer/internal/worker"
 )
 
+const (
+	// Maintenance queries
+	sqlQueryJobsUptoByType = `
+                SELECT array_agg(id), type
+                FROM jobs
+                WHERE type = ANY($1) AND finished_at < $2 AND result IS NOT NULL
+                GROUP BY type`
+	sqlDeleteJobResult = `
+                UPDATE jobs
+                SET result = NULL
+                WHERE id = ANY($1)`
+	sqlVacuumAnalyze = `
+                VACUUM ANALYZE`
+	sqlVacuumStats = `
+                SELECT relname, pg_size_pretty(pg_total_relation_size(relid)),
+                    n_tup_ins, n_tup_upd, n_tup_del, n_live_tup, n_dead_tup,
+                    vacuum_count, autovacuum_count, analyze_count, autoanalyze_count,
+                    last_vacuum, last_autovacuum, last_analyze, last_autoanalyze
+                 FROM pg_stat_user_tables`
+)
+
+type db struct {
+	Conn *pgx.Conn
+}
+
+func newDB(dbURL string) (db, error) {
+	conn, err := pgx.Connect(context.Background(), dbURL)
+	if err != nil {
+		return db{}, err
+	}
+
+	return db{
+		conn,
+	}, nil
+}
+
+func (d *db) Close() {
+	d.Conn.Close(context.Background())
+}
+
+// return map id -> jobtype ?
+func (d *db) JobsUptoByType(jobTypes []string, upto time.Time) (result map[string][]uuid.UUID, err error) {
+	result = make(map[string][]uuid.UUID)
+
+	rows, err := d.Conn.Query(context.Background(), sqlQueryJobsUptoByType, jobTypes, upto)
+	if err != nil {
+		return
+	}
+	defer rows.Close()
+
+	for rows.Next() {
+		var ids []uuid.UUID
+		var jt string
+		err = rows.Scan(&ids, &jt)
+		if err != nil {
+			return
+		}
+
+		result[jt] = ids
+	}
+	err = rows.Err()
+	return
+}
+
+func (d *db) DeleteJobResult(jobIds []uuid.UUID) (int64, error) {
+	tag, err := d.Conn.Exec(context.Background(), sqlDeleteJobResult, jobIds)
+	if err != nil {
+		return tag.RowsAffected(), fmt.Errorf("Error deleting results from jobs: %v", err)
+	}
+	return tag.RowsAffected(), nil
+}
+
+func (d *db) VacuumAnalyze() error {
+	_, err := d.Conn.Exec(context.Background(), sqlVacuumAnalyze)
+	if err != nil {
+		return fmt.Errorf("Error running VACUUM ANALYZE: %v", err)
+	}
+	return nil
+}
+
+func (d *db) LogVacuumStats() error {
+	rows, err := d.Conn.Query(context.Background(), sqlVacuumStats)
+	if err != nil {
+		return fmt.Errorf("Error querying vacuum stats: %v", err)
+	}
+	defer rows.Close()
+
+	for rows.Next() {
+		var relName, relSize string
+		var ins, upd, del, live, dead, vc, avc, ac, aac int64
+		var lvc, lavc, lan, laan *time.Time
+
+		err = rows.Scan(&relName, &relSize, &ins, &upd, &del, &live, &dead,
+			&vc, &avc, &ac, &aac,
+			&lvc, &lavc, &lan, &laan)
+		if err != nil {
+			return err
+		}
+
+		logrus.Infof("Stats for table %s", relName)
+		logrus.Infof("  Total table size: %s", relSize)
+		logrus.Info("  Tuples:")
+		logrus.Infof("    Inserted: %d", ins)
+		logrus.Infof("    Updated: %d", upd)
+		logrus.Infof("    Deleted: %d", del)
+		logrus.Infof("    Live: %d", live)
+		logrus.Infof("    Dead: %d", dead)
+		logrus.Info("  Vacuum stats:")
+		logrus.Infof("    Vacuum count: %d", vc)
+		logrus.Infof("    AutoVacuum count: %d", avc)
+		logrus.Infof("    Last vacuum: %v", lvc)
+		logrus.Infof("    Last autovacuum: %v", lavc)
+		logrus.Info("  Analyze stats:")
+		logrus.Infof("    Analyze count: %d", ac)
+		logrus.Infof("    AutoAnalyze count: %d", aac)
+		logrus.Infof("    Last analyze: %v", lan)
+		logrus.Infof("    Last autoanalyze: %v", laan)
+		logrus.Info("---")
+	}
+	if rows.Err() != nil {
+		return rows.Err()
+	}
+	return nil
+
+}
+
 func DBCleanup(dbURL string, dryRun bool, cutoff time.Time) error {
-	jobs, err := dbjobqueue.New(dbURL)
+	db, err := newDB(dbURL)
 	if err != nil {
 		return err
 	}
 
 	// The results of these jobs take up the most space and can contain sensitive data. Delete
 	// them after a while.
-	jobsByType, err := jobs.JobsUptoByType([]string{worker.JobTypeManifestIDOnly, worker.JobTypeDepsolve}, cutoff)
+	jobsByType, err := db.JobsUptoByType([]string{worker.JobTypeManifestIDOnly, worker.JobTypeDepsolve}, cutoff)
 	if err != nil {
 		return fmt.Errorf("Error querying jobs: %v", err)
 	}
 
-	err = jobs.LogVacuumStats()
+	err = db.LogVacuumStats()
 	if err != nil {
 		logrus.Errorf("Error running vacuum stats: %v", err)
 	}
@@ -42,20 +170,20 @@ func DBCleanup(dbURL string, dryRun bool, cutoff time.Time) error {
 				max = len(v)
 			}
 
-			rows, err := jobs.DeleteJobResult(v[i:max])
+			rows, err := db.DeleteJobResult(v[i:max])
 			if err != nil {
 				logrus.Errorf("Error deleting results for jobs: %v, %d rows affected", rows, err)
 				continue
 			}
 			logrus.Infof("Deleted results from %d jobs out of %d job ids", rows, len(v))
-			err = jobs.VacuumAnalyze()
+			err = db.VacuumAnalyze()
 			if err != nil {
 				logrus.Errorf("Error running vacuum analyze: %v", err)
 			}
 		}
 	}
 
-	err = jobs.LogVacuumStats()
+	err = db.LogVacuumStats()
 	if err != nil {
 		logrus.Errorf("Error running vacuum stats: %v", err)
 	}
diff --git a/distribution/Dockerfile-ubi b/distribution/Dockerfile-ubi
index d010f5601..d67823a1c 100644
--- a/distribution/Dockerfile-ubi
+++ b/distribution/Dockerfile-ubi
@@ -19,7 +19,7 @@ COPY --from=builder /opt/app-root/src/go/bin/osbuild-composer /usr/libexec/osbui
 COPY ./containers/osbuild-composer/entrypoint.py /opt/entrypoint.py
 COPY ./dnf-json /usr/libexec/osbuild-composer/
 
-COPY ./internal/jobqueue/dbjobqueue/schemas /opt/migrate/schemas
+COPY ./pkg/jobqueue/dbjobqueue/schemas /opt/migrate/schemas
 COPY --from=builder2 /opt/app-root/src/go/bin/tern /opt/migrate/
 
 EXPOSE 8080 8700
diff --git a/internal/cloudapi/v2/server.go b/internal/cloudapi/v2/server.go
index 7ba674822..eeb838a7d 100644
--- a/internal/cloudapi/v2/server.go
+++ b/internal/cloudapi/v2/server.go
@@ -14,13 +14,13 @@ import (
 	"github.com/google/uuid"
 	"github.com/labstack/echo/v4"
 	"github.com/labstack/echo/v4/middleware"
+	"github.com/osbuild/osbuild-composer/pkg/jobqueue"
 	"github.com/sirupsen/logrus"
 
 	"github.com/osbuild/osbuild-composer/internal/blueprint"
 	"github.com/osbuild/osbuild-composer/internal/common"
 	"github.com/osbuild/osbuild-composer/internal/distro"
 	"github.com/osbuild/osbuild-composer/internal/distroregistry"
-	"github.com/osbuild/osbuild-composer/internal/jobqueue"
 	"github.com/osbuild/osbuild-composer/internal/prometheus"
 	"github.com/osbuild/osbuild-composer/internal/rpmmd"
 	"github.com/osbuild/osbuild-composer/internal/target"
diff --git a/internal/dnfjson/dnfjson.go b/internal/dnfjson/dnfjson.go
index 4f561a0fc..cf5c305b9 100644
--- a/internal/dnfjson/dnfjson.go
+++ b/internal/dnfjson/dnfjson.go
@@ -442,6 +442,17 @@ func parseError(data []byte, repos []repoConfig) Error {
 
 	return e
 }
+func ParseError(data []byte) Error {
+	var e Error
+	if err := json.Unmarshal(data, &e); err != nil {
+		// dumping the error into the Reason can get noisy, but it's good for troubleshooting
+		return Error{
+			Kind:   "InternalError",
+			Reason: fmt.Sprintf("Failed to unmarshal dnf-json error output %q: %s", string(data), err.Error()),
+		}
+	}
+	return e
+}
 
 func run(dnfJsonCmd []string, req *Request) ([]byte, error) {
 	if len(dnfJsonCmd) == 0 {
diff --git a/internal/jobqueue/fsjobqueue/fsjobqueue.go b/internal/jobqueue/fsjobqueue/fsjobqueue.go
index 3a38d7483..62451fab5 100644
--- a/internal/jobqueue/fsjobqueue/fsjobqueue.go
+++ b/internal/jobqueue/fsjobqueue/fsjobqueue.go
@@ -20,8 +20,8 @@ import (
 	"time"
 
 	"github.com/google/uuid"
+	"github.com/osbuild/osbuild-composer/pkg/jobqueue"
 
-	"github.com/osbuild/osbuild-composer/internal/jobqueue"
 	"github.com/osbuild/osbuild-composer/internal/jsondb"
 )
 
diff --git a/internal/weldr/api.go b/internal/weldr/api.go
index 91ee1e921..f8041f554 100644
--- a/internal/weldr/api.go
+++ b/internal/weldr/api.go
@@ -28,13 +28,13 @@ import (
 	"github.com/gobwas/glob"
 	"github.com/google/uuid"
 	"github.com/julienschmidt/httprouter"
+	"github.com/osbuild/osbuild-composer/pkg/jobqueue"
 
 	"github.com/osbuild/osbuild-composer/internal/blueprint"
 	"github.com/osbuild/osbuild-composer/internal/common"
 	"github.com/osbuild/osbuild-composer/internal/distro"
 	"github.com/osbuild/osbuild-composer/internal/distroregistry"
 	"github.com/osbuild/osbuild-composer/internal/dnfjson"
-	"github.com/osbuild/osbuild-composer/internal/jobqueue"
 	osbuild "github.com/osbuild/osbuild-composer/internal/osbuild2"
 	"github.com/osbuild/osbuild-composer/internal/ostree"
 	"github.com/osbuild/osbuild-composer/internal/reporegistry"
diff --git a/internal/worker/server.go b/internal/worker/server.go
index b5e060453..6eef0b92e 100644
--- a/internal/worker/server.go
+++ b/internal/worker/server.go
@@ -17,11 +17,11 @@ import (
 	"github.com/google/uuid"
 	"github.com/labstack/echo/v4"
 	"github.com/labstack/echo/v4/middleware"
+	"github.com/osbuild/osbuild-composer/pkg/jobqueue"
 	"github.com/sirupsen/logrus"
 
 	"github.com/osbuild/osbuild-composer/internal/auth"
 	"github.com/osbuild/osbuild-composer/internal/common"
-	"github.com/osbuild/osbuild-composer/internal/jobqueue"
 	"github.com/osbuild/osbuild-composer/internal/prometheus"
 	"github.com/osbuild/osbuild-composer/internal/worker/api"
 	"github.com/osbuild/osbuild-composer/internal/worker/clienterrors"
diff --git a/osbuild-composer.spec b/osbuild-composer.spec
index 1d1aeb331..fbd8273cb 100644
--- a/osbuild-composer.spec
+++ b/osbuild-composer.spec
@@ -129,6 +129,7 @@ go test -c -tags=integration -ldflags="${TEST_LDFLAGS}" -o _bin/osbuild-auth-tes
 go test -c -tags=integration -ldflags="${TEST_LDFLAGS}" -o _bin/osbuild-koji-tests %{goipath}/cmd/osbuild-koji-tests
 go test -c -tags=integration -ldflags="${TEST_LDFLAGS}" -o _bin/osbuild-composer-dbjobqueue-tests %{goipath}/cmd/osbuild-composer-dbjobqueue-tests
 go test -c -tags=integration -ldflags="${TEST_LDFLAGS}" -o _bin/osbuild-composer-manifest-tests %{goipath}/cmd/osbuild-composer-manifest-tests
+go test -c -tags=integration -ldflags="${TEST_LDFLAGS}" -o _bin/osbuild-service-maintenance-tests %{goipath}/cmd/osbuild-service-maintenance
 go build -tags=integration -ldflags="${TEST_LDFLAGS}" -o _bin/osbuild-mock-openid-provider %{goipath}/cmd/osbuild-mock-openid-provider
 
 %endif
@@ -194,6 +195,7 @@ install -m 0755 -vp _bin/osbuild-auth-tests                        %{buildroot}%
 install -m 0755 -vp _bin/osbuild-koji-tests                        %{buildroot}%{_libexecdir}/osbuild-composer-test/
 install -m 0755 -vp _bin/osbuild-composer-dbjobqueue-tests         %{buildroot}%{_libexecdir}/osbuild-composer-test/
 install -m 0755 -vp _bin/osbuild-composer-manifest-tests           %{buildroot}%{_libexecdir}/osbuild-composer-test/
+install -m 0755 -vp _bin/osbuild-service-maintenance-tests         %{buildroot}%{_libexecdir}/osbuild-composer-test/
 install -m 0755 -vp _bin/osbuild-mock-openid-provider              %{buildroot}%{_libexecdir}/osbuild-composer-test/
 install -m 0755 -vp tools/define-compose-url.sh                    %{buildroot}%{_libexecdir}/osbuild-composer-test/
 install -m 0755 -vp tools/provision.sh                             %{buildroot}%{_libexecdir}/osbuild-composer-test/
@@ -250,7 +252,7 @@ install -m 0755 -vd                                                %{buildroot}%
 install -m 0644 -vp test/data/openshift/*                          %{buildroot}%{_datadir}/tests/osbuild-composer/openshift/
 
 install -m 0755 -vd                                                %{buildroot}%{_datadir}/tests/osbuild-composer/schemas
-install -m 0644 -vp internal/jobqueue/dbjobqueue/schemas/*         %{buildroot}%{_datadir}/tests/osbuild-composer/schemas/
+install -m 0644 -vp pkg/jobqueue/dbjobqueue/schemas/*              %{buildroot}%{_datadir}/tests/osbuild-composer/schemas/
 
 install -m 0755 -vd                                               %{buildroot}%{_datadir}/tests/osbuild-composer/upgrade8to9
 install -m 0644 -vp test/data/upgrade8to9/*                       %{buildroot}%{_datadir}/tests/osbuild-composer/upgrade8to9/
diff --git a/internal/jobqueue/dbjobqueue/dbjobqueue.go b/pkg/jobqueue/dbjobqueue/dbjobqueue.go
similarity index 80%
rename from internal/jobqueue/dbjobqueue/dbjobqueue.go
rename to pkg/jobqueue/dbjobqueue/dbjobqueue.go
index e3059dc25..4c1e60105 100644
--- a/internal/jobqueue/dbjobqueue/dbjobqueue.go
+++ b/pkg/jobqueue/dbjobqueue/dbjobqueue.go
@@ -18,10 +18,9 @@ import (
 	"github.com/jackc/pgtype"
 	"github.com/jackc/pgx/v4"
 	"github.com/jackc/pgx/v4/pgxpool"
+	"github.com/osbuild/osbuild-composer/pkg/jobqueue"
 
 	logrus "github.com/sirupsen/logrus"
-
-	"github.com/osbuild/osbuild-composer/internal/jobqueue"
 )
 
 const (
@@ -99,25 +98,6 @@ const (
 	sqlDeleteHeartbeat = `
                 DELETE FROM heartbeats
                 WHERE id = $1`
-
-	// Maintenance queries
-	sqlQueryJobsUptoByType = `
-                SELECT array_agg(id), type
-                FROM jobs
-                WHERE type = ANY($1) AND finished_at < $2 AND result IS NOT NULL
-                GROUP BY type`
-	sqlDeleteJobResult = `
-                UPDATE jobs
-                SET result = NULL
-                WHERE id = ANY($1)`
-	sqlVacuumAnalyze = `
-                VACUUM ANALYZE`
-	sqlVacuumStats = `
-                SELECT relname, pg_size_pretty(pg_total_relation_size(relid)),
-                    n_tup_ins, n_tup_upd, n_tup_del, n_live_tup, n_dead_tup,
-                    vacuum_count, autovacuum_count, analyze_count, autoanalyze_count,
-                    last_vacuum, last_autovacuum, last_analyze, last_autoanalyze
-                 FROM pg_stat_user_tables`
 )
 
 type DBJobQueue struct {
@@ -621,115 +601,3 @@ func (q *DBJobQueue) jobDependencies(ctx context.Context, conn *pgxpool.Conn, id
 
 	return dependencies, nil
 }
-
-// return map id -> jobtype ?
-func (q *DBJobQueue) JobsUptoByType(jobTypes []string, upto time.Time) (result map[string][]uuid.UUID, err error) {
-	result = make(map[string][]uuid.UUID)
-
-	conn, err := q.pool.Acquire(context.Background())
-	if err != nil {
-		err = fmt.Errorf("error connecting to database: %v", err)
-		return
-	}
-	defer conn.Release()
-
-	rows, err := conn.Query(context.Background(), sqlQueryJobsUptoByType, jobTypes, upto)
-	if err != nil {
-		return
-	}
-	defer rows.Close()
-
-	for rows.Next() {
-		var ids []uuid.UUID
-		var jt string
-		err = rows.Scan(&ids, &jt)
-		if err != nil {
-			return
-		}
-
-		result[jt] = ids
-	}
-	err = rows.Err()
-	return
-}
-
-func (q *DBJobQueue) DeleteJobResult(jobIds []uuid.UUID) (int64, error) {
-	conn, err := q.pool.Acquire(context.Background())
-	if err != nil {
-		return -1, fmt.Errorf("error connecting to database: %v", err)
-	}
-	defer conn.Release()
-
-	tag, err := conn.Exec(context.Background(), sqlDeleteJobResult, jobIds)
-	if err != nil {
-		return tag.RowsAffected(), fmt.Errorf("Error deleting results from jobs: %v", err)
-	}
-	return tag.RowsAffected(), nil
-}
-
-func (q *DBJobQueue) VacuumAnalyze() error {
-	conn, err := q.pool.Acquire(context.Background())
-	if err != nil {
-		return fmt.Errorf("error connecting to database: %v", err)
-	}
-	defer conn.Release()
-
-	_, err = conn.Exec(context.Background(), sqlVacuumAnalyze)
-	if err != nil {
-		return fmt.Errorf("Error running VACUUM ANALYZE: %v", err)
-	}
-
-	return nil
-}
-
-func (q *DBJobQueue) LogVacuumStats() error {
-	conn, err := q.pool.Acquire(context.Background())
-	if err != nil {
-		return fmt.Errorf("error connecting to database: %v", err)
-	}
-	defer conn.Release()
-
-	rows, err := conn.Query(context.Background(), sqlVacuumStats)
-	if err != nil {
-		return fmt.Errorf("Error querying vacuum stats: %v", err)
-	}
-	defer rows.Close()
-
-	for rows.Next() {
-		var relName, relSize string
-		var ins, upd, del, live, dead, vc, avc, ac, aac int64
-		var lvc, lavc, lan, laan *time.Time
-
-		err = rows.Scan(&relName, &relSize, &ins, &upd, &del, &live, &dead,
-			&vc, &avc, &ac, &aac,
-			&lvc, &lavc, &lan, &laan)
-		if err != nil {
-			return err
-		}
-
-		logrus.Infof("Stats for table %s", relName)
-		logrus.Infof("  Total table size: %s", relSize)
-		logrus.Info("  Tuples:")
-		logrus.Infof("    Inserted: %d", ins)
-		logrus.Infof("    Updated: %d", upd)
-		logrus.Infof("    Deleted: %d", del)
-		logrus.Infof("    Live: %d", live)
-		logrus.Infof("    Dead: %d", dead)
-		logrus.Info("  Vacuum stats:")
-		logrus.Infof("    Vacuum count: %d", vc)
-		logrus.Infof("    AutoVacuum count: %d", avc)
-		logrus.Infof("    Last vacuum: %v", lvc)
-		logrus.Infof("    Last autovacuum: %v", lavc)
-		logrus.Info("  Analyze stats:")
-		logrus.Infof("    Analyze count: %d", ac)
-		logrus.Infof("    AutoAnalyze count: %d", aac)
-		logrus.Infof("    Last analyze: %v", lan)
-		logrus.Infof("    Last autoanalyze: %v", laan)
-		logrus.Info("---")
-	}
-	if rows.Err() != nil {
-		return rows.Err()
-	}
-	return nil
-
-}


Test patch:
diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml
index 58073dd95..ddd1ff396 100644
--- a/.github/workflows/tests.yml
+++ b/.github/workflows/tests.yml
@@ -38,6 +38,45 @@ jobs:
       - name: Send coverage to codecov.io
         run: bash <(curl -s https://codecov.io/bash)
 
+  db-tests:
+    name: "🗄 DB tests"
+    runs-on: ubuntu-20.04
+    services:
+      postgres:
+        # 12 is used in deployments
+        image: postgres:12
+        env:
+          POSTGRES_USER: postgres
+          POSTGRES_PASSWORD: foobar
+          POSTGRES_DB: osbuildcomposer
+        options: >-
+          --health-cmd pg_isready
+          --health-interval 10s
+          --health-timeout 5s
+          --health-retries 5
+        ports:
+          - 5432:5432
+    steps:
+    - uses: actions/checkout@v2
+    - uses: actions/setup-go@v2
+      with:
+        go-version: 1.16
+    - env:
+        PGUSER: postgres
+        PGPASSWORD: foobar
+        PGDATABASE: osbuildcomposer
+        PGHOST: localhost
+        PGPORT: 5432
+      run: |
+        WORKDIR=$(readlink -f pkg/jobqueue/dbjobqueue/schemas)
+        pushd $(mktemp -d)
+        go mod init temp
+        go get -u github.com/jackc/tern
+        go run github.com/jackc/tern migrate -m "$WORKDIR"
+        popd
+    - run: go test -tags=integration ./cmd/osbuild-composer-dbjobqueue-tests
+    - run: go test -tags=integration ./cmd/osbuild-service-maintenance
+
   python-lint:
     name: "🐍 Lint (dnf-json)"
     runs-on: ubuntu-latest
diff --git a/cmd/osbuild-composer-dbjobqueue-tests/main_test.go b/cmd/osbuild-composer-dbjobqueue-tests/main_test.go
index 5537f45e4..29ec66f85 100644
--- a/cmd/osbuild-composer-dbjobqueue-tests/main_test.go
+++ b/cmd/osbuild-composer-dbjobqueue-tests/main_test.go
@@ -4,17 +4,13 @@ package main
 
 import (
 	"context"
-	"encoding/json"
 	"fmt"
 	"testing"
-	"time"
 
-	"github.com/google/uuid"
 	"github.com/jackc/pgx/v4"
-	"github.com/stretchr/testify/require"
+	"github.com/osbuild/osbuild-composer/pkg/jobqueue"
+	"github.com/osbuild/osbuild-composer/pkg/jobqueue/dbjobqueue"
 
-	"github.com/osbuild/osbuild-composer/internal/jobqueue"
-	"github.com/osbuild/osbuild-composer/internal/jobqueue/dbjobqueue"
 	"github.com/osbuild/osbuild-composer/internal/jobqueue/jobqueuetest"
 )
 
@@ -50,97 +46,4 @@ func TestJobQueueInterface(t *testing.T) {
 	}
 
 	jobqueuetest.TestJobQueue(t, makeJobQueue)
-
-	wrap := func(f func(t *testing.T, q *dbjobqueue.DBJobQueue)) func(*testing.T) {
-		q, stop, err := makeJobQueue()
-		require.NoError(t, err)
-		return func(t *testing.T) {
-			defer stop() // use defer because f() might call testing.T.FailNow()
-			dbq, ok := q.(*dbjobqueue.DBJobQueue)
-			require.True(t, ok)
-			f(t, dbq)
-		}
-	}
-
-	t.Run("maintenance-query-jobs-before", wrap(testJobsUptoByType))
-	t.Run("maintenance-delete-job-results", wrap(testDeleteJobResult))
-}
-
-func setFinishedAt(t *testing.T, q *dbjobqueue.DBJobQueue, id uuid.UUID, finished time.Time) {
-	conn, err := pgx.Connect(context.Background(), url)
-	require.NoError(t, err)
-	defer conn.Close(context.Background())
-
-	started := finished.Add(-time.Second)
-	queued := started.Add(-time.Second)
-
-	_, err = conn.Exec(context.Background(), "UPDATE jobs SET queued_at = $1, started_at = $2, finished_at = $3, result = '{\"result\": \"success\" }' WHERE id = $4", queued, started, finished, id)
-	require.NoError(t, err)
-}
-
-func testJobsUptoByType(t *testing.T, q *dbjobqueue.DBJobQueue) {
-	date80 := time.Date(1980, time.January, 1, 0, 0, 0, 0, time.UTC)
-	date85 := time.Date(1985, time.January, 1, 0, 0, 0, 0, time.UTC)
-	date90 := time.Date(1990, time.January, 1, 0, 0, 0, 0, time.UTC)
-
-	id80, err := q.Enqueue("octopus", nil, nil, "")
-	require.NoError(t, err)
-	require.NotEqual(t, uuid.Nil, id80)
-	_, _, _, _, _, err = q.Dequeue(context.Background(), []string{"octopus"}, []string{""})
-	require.NoError(t, err)
-	err = q.FinishJob(id80, nil)
-	require.NoError(t, err)
-	setFinishedAt(t, q, id80, date80)
-
-	id85, err := q.Enqueue("octopus", nil, nil, "")
-	require.NoError(t, err)
-	require.NotEqual(t, uuid.Nil, id85)
-	_, _, _, _, _, err = q.Dequeue(context.Background(), []string{"octopus"}, []string{""})
-	require.NoError(t, err)
-	err = q.FinishJob(id85, nil)
-	require.NoError(t, err)
-	setFinishedAt(t, q, id85, date85)
-
-	ids, err := q.JobsUptoByType([]string{"octopus"}, date85)
-	require.NoError(t, err)
-	require.ElementsMatch(t, []uuid.UUID{id80}, ids["octopus"])
-
-	ids, err = q.JobsUptoByType([]string{"octopus"}, date90)
-	require.NoError(t, err)
-	require.ElementsMatch(t, []uuid.UUID{id80, id85}, ids["octopus"])
-}
-
-func testDeleteJobResult(t *testing.T, q *dbjobqueue.DBJobQueue) {
-	id, err := q.Enqueue("octopus", nil, nil, "")
-	require.NoError(t, err)
-	require.NotEqual(t, uuid.Nil, id)
-	_, _, _, _, _, err = q.Dequeue(context.Background(), []string{"octopus"}, []string{""})
-	require.NoError(t, err)
-
-	type Result struct {
-		Result string `json:"result"`
-	}
-	result := Result{
-		"deleteme",
-	}
-
-	res, err := json.Marshal(result)
-	require.NoError(t, err)
-	err = q.FinishJob(id, res)
-	require.NoError(t, err)
-
-	_, _, r, _, _, _, _, _, err := q.JobStatus(id)
-	require.NoError(t, err)
-
-	var r1 Result
-	require.NoError(t, json.Unmarshal(r, &r1))
-	require.Equal(t, result, r1)
-
-	rows, err := q.DeleteJobResult([]uuid.UUID{id})
-	require.NoError(t, err)
-	require.Equal(t, int64(1), rows)
-
-	_, _, r, _, _, _, _, _, err = q.JobStatus(id)
-	require.NoError(t, err)
-	require.Nil(t, r)
 }
diff --git a/cmd/osbuild-service-maintenance/db_test.go b/cmd/osbuild-service-maintenance/db_test.go
new file mode 100644
index 000000000..beeda9266
--- /dev/null
+++ b/cmd/osbuild-service-maintenance/db_test.go
@@ -0,0 +1,118 @@
+// +build integration
+
+package main
+
+import (
+	"context"
+	"encoding/json"
+	"testing"
+	"time"
+
+	"github.com/google/uuid"
+	"github.com/osbuild/osbuild-composer/pkg/jobqueue/dbjobqueue"
+	"github.com/stretchr/testify/require"
+)
+
+const url = "postgres://postgres:foobar@localhost:5432/osbuildcomposer"
+
+func TestDBJobQueueMaintenance(t *testing.T) {
+	dbMaintenance, err := newDB(url)
+	require.NoError(t, err)
+	defer dbMaintenance.Close()
+	q, err := dbjobqueue.New(url)
+	require.NoError(t, err)
+	defer q.Close()
+
+	_, err = dbMaintenance.Conn.Exec(context.Background(), "DELETE FROM jobs")
+	require.NoError(t, err)
+
+	t.Run("testJobsUptoByType", func(t *testing.T) {
+		testJobsUptoByType(t, dbMaintenance, q)
+	})
+	t.Run("testDeleteJobResult", func(t *testing.T) {
+		testDeleteJobResult(t, dbMaintenance, q)
+	})
+	t.Run("testVacuum", func(t *testing.T) {
+		testVacuum(t, dbMaintenance, q)
+	})
+
+}
+
+func setFinishedAt(t *testing.T, d db, id uuid.UUID, finished time.Time) {
+	started := finished.Add(-time.Second)
+	queued := started.Add(-time.Second)
+	_, err := d.Conn.Exec(context.Background(), "UPDATE jobs SET queued_at = $1, started_at = $2, finished_at = $3, result = '{\"result\": \"success\" }' WHERE id = $4", queued, started, finished, id)
+	require.NoError(t, err)
+}
+
+func testJobsUptoByType(t *testing.T, d db, q *dbjobqueue.DBJobQueue) {
+	date80 := time.Date(1980, time.January, 1, 0, 0, 0, 0, time.UTC)
+	date85 := time.Date(1985, time.January, 1, 0, 0, 0, 0, time.UTC)
+	date90 := time.Date(1990, time.January, 1, 0, 0, 0, 0, time.UTC)
+
+	id80, err := q.Enqueue("octopus", nil, nil, "")
+	require.NoError(t, err)
+	require.NotEqual(t, uuid.Nil, id80)
+	_, _, _, _, _, err = q.Dequeue(context.Background(), []string{"octopus"}, []string{""})
+	require.NoError(t, err)
+	err = q.FinishJob(id80, nil)
+	require.NoError(t, err)
+	setFinishedAt(t, d, id80, date80)
+
+	id85, err := q.Enqueue("octopus", nil, nil, "")
+	require.NoError(t, err)
+	require.NotEqual(t, uuid.Nil, id85)
+	_, _, _, _, _, err = q.Dequeue(context.Background(), []string{"octopus"}, []string{""})
+	require.NoError(t, err)
+	err = q.FinishJob(id85, nil)
+	require.NoError(t, err)
+	setFinishedAt(t, d, id85, date85)
+
+	ids, err := d.JobsUptoByType([]string{"octopus"}, date85)
+	require.NoError(t, err)
+	require.ElementsMatch(t, []uuid.UUID{id80}, ids["octopus"])
+
+	ids, err = d.JobsUptoByType([]string{"octopus"}, date90)
+	require.NoError(t, err)
+	require.ElementsMatch(t, []uuid.UUID{id80, id85}, ids["octopus"])
+}
+
+func testDeleteJobResult(t *testing.T, d db, q *dbjobqueue.DBJobQueue) {
+	id, err := q.Enqueue("octopus", nil, nil, "")
+	require.NoError(t, err)
+	require.NotEqual(t, uuid.Nil, id)
+	_, _, _, _, _, err = q.Dequeue(context.Background(), []string{"octopus"}, []string{""})
+	require.NoError(t, err)
+
+	type Result struct {
+		Result string `json:"result"`
+	}
+	result := Result{
+		"deleteme",
+	}
+
+	res, err := json.Marshal(result)
+	require.NoError(t, err)
+	err = q.FinishJob(id, res)
+	require.NoError(t, err)
+
+	_, _, r, _, _, _, _, _, err := q.JobStatus(id)
+	require.NoError(t, err)
+
+	var r1 Result
+	require.NoError(t, json.Unmarshal(r, &r1))
+	require.Equal(t, result, r1)
+
+	rows, err := d.DeleteJobResult([]uuid.UUID{id})
+	require.NoError(t, err)
+	require.Equal(t, int64(1), rows)
+
+	_, _, r, _, _, _, _, _, err = q.JobStatus(id)
+	require.NoError(t, err)
+	require.Nil(t, r)
+}
+
+func testVacuum(t *testing.T, d db, q *dbjobqueue.DBJobQueue) {
+	require.NoError(t, d.VacuumAnalyze())
+	require.NoError(t, d.LogVacuumStats())
+}
diff --git a/internal/cloudapi/v2/v2_multi_tenancy_test.go b/internal/cloudapi/v2/v2_multi_tenancy_test.go
index b80b15c63..d1ff03f1b 100644
--- a/internal/cloudapi/v2/v2_multi_tenancy_test.go
+++ b/internal/cloudapi/v2/v2_multi_tenancy_test.go
@@ -11,11 +11,11 @@ import (
 	"github.com/golang-jwt/jwt/v4"
 	"github.com/google/uuid"
 	"github.com/openshift-online/ocm-sdk-go/authentication"
+	"github.com/osbuild/osbuild-composer/pkg/jobqueue"
 	"github.com/stretchr/testify/require"
 
 	"github.com/osbuild/osbuild-composer/internal/cloudapi/v2"
 	"github.com/osbuild/osbuild-composer/internal/distro/test_distro"
-	"github.com/osbuild/osbuild-composer/internal/jobqueue"
 	"github.com/osbuild/osbuild-composer/internal/test"
 	"github.com/osbuild/osbuild-composer/internal/worker"
 )
diff --git a/internal/cloudapi/v2/v2_test.go b/internal/cloudapi/v2/v2_test.go
index e03952aea..00a7143ec 100644
--- a/internal/cloudapi/v2/v2_test.go
+++ b/internal/cloudapi/v2/v2_test.go
@@ -8,11 +8,11 @@ import (
 	"sync"
 	"testing"
 
+	"github.com/osbuild/osbuild-composer/pkg/jobqueue"
 	"github.com/stretchr/testify/require"
 
 	v2 "github.com/osbuild/osbuild-composer/internal/cloudapi/v2"
 	"github.com/osbuild/osbuild-composer/internal/distro/test_distro"
-	"github.com/osbuild/osbuild-composer/internal/jobqueue"
 	"github.com/osbuild/osbuild-composer/internal/jobqueue/fsjobqueue"
 	distro_mock "github.com/osbuild/osbuild-composer/internal/mocks/distro"
 	"github.com/osbuild/osbuild-composer/internal/osbuild2"
diff --git a/internal/jobqueue/fsjobqueue/fsjobqueue_test.go b/internal/jobqueue/fsjobqueue/fsjobqueue_test.go
index 971dec455..aba51cc1e 100644
--- a/internal/jobqueue/fsjobqueue/fsjobqueue_test.go
+++ b/internal/jobqueue/fsjobqueue/fsjobqueue_test.go
@@ -3,9 +3,9 @@ package fsjobqueue_test
 import (
 	"testing"
 
+	"github.com/osbuild/osbuild-composer/pkg/jobqueue"
 	"github.com/stretchr/testify/require"
 
-	"github.com/osbuild/osbuild-composer/internal/jobqueue"
 	"github.com/osbuild/osbuild-composer/internal/jobqueue/fsjobqueue"
 	"github.com/osbuild/osbuild-composer/internal/jobqueue/jobqueuetest"
 )
diff --git a/internal/jobqueue/jobqueuetest/jobqueuetest.go b/internal/jobqueue/jobqueuetest/jobqueuetest.go
index fd391ad1a..b589aff6d 100644
--- a/internal/jobqueue/jobqueuetest/jobqueuetest.go
+++ b/internal/jobqueue/jobqueuetest/jobqueuetest.go
@@ -11,9 +11,8 @@ import (
 	"time"
 
 	"github.com/google/uuid"
+	"github.com/osbuild/osbuild-composer/pkg/jobqueue"
 	"github.com/stretchr/testify/require"
-
-	"github.com/osbuild/osbuild-composer/internal/jobqueue"
 )
 
 type MakeJobQueue func() (q jobqueue.JobQueue, stop func(), err error)
diff --git a/internal/worker/server_test.go b/internal/worker/server_test.go
index f6b3b9575..4546a3bf8 100644
--- a/internal/worker/server_test.go
+++ b/internal/worker/server_test.go
@@ -11,12 +11,12 @@ import (
 	"time"
 
 	"github.com/google/uuid"
+	"github.com/osbuild/osbuild-composer/pkg/jobqueue"
 	"github.com/stretchr/testify/assert"
 	"github.com/stretchr/testify/require"
 
 	"github.com/osbuild/osbuild-composer/internal/distro"
 	"github.com/osbuild/osbuild-composer/internal/distro/test_distro"
-	"github.com/osbuild/osbuild-composer/internal/jobqueue"
 	"github.com/osbuild/osbuild-composer/internal/jobqueue/fsjobqueue"
 	"github.com/osbuild/osbuild-composer/internal/osbuild2"
 	"github.com/osbuild/osbuild-composer/internal/rpmmd"
diff --git a/internal/jobqueue/dbjobqueue/schemas/001_jobs.sql b/pkg/jobqueue/dbjobqueue/schemas/001_jobs.sql
similarity index 100%
rename from internal/jobqueue/dbjobqueue/schemas/001_jobs.sql
rename to pkg/jobqueue/dbjobqueue/schemas/001_jobs.sql
diff --git a/internal/jobqueue/dbjobqueue/schemas/002_channels.sql b/pkg/jobqueue/dbjobqueue/schemas/002_channels.sql
similarity index 100%
rename from internal/jobqueue/dbjobqueue/schemas/002_channels.sql
rename to pkg/jobqueue/dbjobqueue/schemas/002_channels.sql
diff --git a/internal/jobqueue/dbjobqueue/schemas/003_dependencies_cascade.sql b/pkg/jobqueue/dbjobqueue/schemas/003_dependencies_cascade.sql
similarity index 100%
rename from internal/jobqueue/dbjobqueue/schemas/003_dependencies_cascade.sql
rename to pkg/jobqueue/dbjobqueue/schemas/003_dependencies_cascade.sql
diff --git a/internal/jobqueue/dbjobqueue/schemas/004_heartbeats_cascade.sql b/pkg/jobqueue/dbjobqueue/schemas/004_heartbeats_cascade.sql
similarity index 100%
rename from internal/jobqueue/dbjobqueue/schemas/004_heartbeats_cascade.sql
rename to pkg/jobqueue/dbjobqueue/schemas/004_heartbeats_cascade.sql
diff --git a/internal/jobqueue/jobqueue.go b/pkg/jobqueue/jobqueue.go
similarity index 100%
rename from internal/jobqueue/jobqueue.go
rename to pkg/jobqueue/jobqueue.go
diff --git a/test/cases/base_tests.sh b/test/cases/base_tests.sh
index d262a2e94..ce7762d8a 100755
--- a/test/cases/base_tests.sh
+++ b/test/cases/base_tests.sh
@@ -18,7 +18,6 @@ TEST_CASES=(
   "osbuild-dnf-json-tests"
   "osbuild-composer-cli-tests"
   "osbuild-auth-tests"
-  "osbuild-composer-dbjobqueue-tests"
 )
 
 # Print out a nice test divider so we know when tests stop and start.
@@ -48,27 +47,6 @@ run_test_case () {
 # Provision the software under test.
 /usr/libexec/osbuild-composer-test/provision.sh
 
-# Set up a basic postgres db
-sudo dnf install -y go postgresql postgresql-server postgresql-contrib
-PWFILE=$(sudo -u postgres mktemp)
-cat <<EOF | sudo -u postgres tee "$PWFILE"
-foobar
-EOF
-PGSETUP_INITDB_OPTIONS="-A password -U postgres --pwfile=$PWFILE" \
-    sudo -E postgresql-setup --initdb --unit postgresql
-sudo systemctl start postgresql
-PGUSER=postgres PGPASSWORD=foobar PGHOST=localhost PGPORT=5432 \
-    sudo -E -u postgres psql -c "CREATE DATABASE osbuildcomposer;"
-
-# Initialize a module in a temp dir so we can get tern without introducing
-# vendoring inconsistency
-pushd "$(mktemp -d)"
-go mod init temp
-go get github.com/jackc/tern
-PGUSER=postgres PGPASSWORD=foobar PGDATABASE=osbuildcomposer PGHOST=localhost PGPORT=5432 \
-      go run github.com/jackc/tern migrate -m /usr/share/tests/osbuild-composer/schemas
-popd
-
 # Change to the working directory.
 cd $WORKING_DIRECTORY