Repository: osbuild/osbuild-composer
Pull request #2733 (2022-06-09 11:37:35)

Size-based cleanup for dnf-json cache directories
This PR implements size-based garbage collection for the dnf-json cache directory, based on the outline in https://github.com/osbuild/osbuild-composer/issues/2446.

The workflow is as follows:
- Update the `mtime` for a repository's cache on-disk using `touchRepo()` whenever it's used in a transaction (`Depsolve()` or `FetchMetadata()`).
- Update the internal cache information when needed by calling `updateInfo()` (see note below about the frequency of this call).
- Shrink the cache to below the configured `maxSize` by calling `shrink()`.

The most important work happens in `updateInfo()`.  It collects all the information it needs from the on-disk cache directories and organises it in a way that makes it convenient for the `shrink()` function to run efficiently.  It stores three important pieces of information:
1. `repoElements`: a map that links a repository ID with all the information about a repository's cache:
    - the top-level elements (files and directories) for the cache
    - size of the repository cache (total of all elements)
    - most recent `mtime` from all the elements which, if the `touchRepo()` call is consistently used, should reflect the most recent time the repository was used
2. `repoRecency`: a list of repository IDs sorted by `mtime` (oldest first)
3. size: the total size of the cache (total of all repository caches)

This way, when `shrink()` is called, the paths associated with the least-recently-used repositories can be easily deleted by iterating on `repoRecency`, obtaining the repository info from the map, deleting every path in the `repoElements` array, and subtracting the repository's `size` from the total.  The `shrink()` function stops when the new size is below the `maxSize` (or when all repositories have been deleted).

## Locks

Each cache instance has a RWMutex lock.  A global map of cache locks is maintained, keyed by the absolute path to the cache directory, so multiple cache instances can coexist and share locks if they use the same cache root.

The RWMutex lock allows for multiple concurrent read locks to be held or one exclusive write lock.  A read lock is acquired for every transaction and a write lock is acquired when deleting caches.
The _read lock_ is a bit of a misnomer since a transaction can (and will) write files to an individual repository's metadata cache.  Since `dnf` takes care of locking individual repository caches when in use, we don't need an exclusive lock for these transactions.
The _write lock_ prevents any transactions from running, regardless of repository, when a cleanup (`shrink()`) is being run.

### No interprocess locking

I considered (and [implemented](https://github.com/achilleas-k/osbuild-composer/tree/dnfjson/size-based-cleanup-file-locks)) file-based locking of the cache with the same logic as the RWMutex.  This worked and provided locks across processes, but required more work to release locks when osbuild-composer was interrupted or killed.  I think it's fine to just _require_ (as a policy) that multiple potential instances of osbuild-composer, osbuild-worker, or perhaps a future dnf-json service don't share a cache.

## Note on `updateInfo()`

The function is called after every transaction, which might seem like a lot of work to do every time `Depsolve()` is called, especially when running multiple transactions successively for a single manifest.  I considered at some point keeping an internal timestamp (`lastUpdated`) of the last time the info was updated (the `mtime` and size).  The idea was that if the `lastUpdated` was before `info.mtime`, then the size data might be outdated and it should be reread.
I reconsidered because this felt like excessive book-keeping.  In reality, updating the size info consists of a handful of stat calls per repo and, on top of that, the system caches the stat info anyway, so we can let the system keep it efficient for us.

gicmo: I have only glanced at the code and I think it looks fine!

I just wanted to highlight the behavioural change:  _Currently_ we fill the cache if it "overflows" we clear all of it and let it re-fill. The size-based clearing obviously changes this (intentionally). Let's assume we have N different cache items (say from different RHEL composes) that are all roughly the same size K. N*K is smaller than the limit but (N+1)*K is over it. We would clean up only one cache item and the next new item would kick us over the limit again. That is, we are operating at the cache limit most of the time (more possible cache hits) but also have to do the cleanup more often (depsolve jobs more often will have a small extra latency instead of one bigger one less frequently). I wonder if we could collect metrics on this to better guide us about this, since we could also have an extra parameter that would specify how much below the limit to shrink, i.e. how much we "bounce".


achilleas-k: Yeah you make a good point.  A bounce would be a good solution.
I suspect that this might be less of an issue in reality (though we'd have to investigate).  It depends on how fast the repo turnover is.  It's possible that with a big enough `maxSize` (big enough to hold all frequently used repos and some old unused ones), that we'd be cleaning up a repo very infrequently (only when a new set comes in).  But also, what's the situation in prod?  We have the CDNs, which means the same repos get reused for the lifetime of a distro.  If we can keep around all repo caches for say 5 distributions (RHEL 9.0, RHEL 8.6, RHEL 8.5, CS9, CS8), then as soon as RHEL 8.7 gets added, we'd drop 8.5 and nothing changes again for a long time.

This is oversimplified of course because the sizes aren't uniform and I don't know anything about the RHEL vs CS usage patterns.  It might be more likely that CS8 and CS9 get cleaned because they're rarely used maybe?  User defined repositories will probably always be the first to go and then the cache stays stable for 6 months until the next distro arrives.

@Gundersanne can you comment on this?

Patch:
diff --git a/cmd/gen-manifests/main.go b/cmd/gen-manifests/main.go
index d9887859b..0a1ba1170 100644
--- a/cmd/gen-manifests/main.go
+++ b/cmd/gen-manifests/main.go
@@ -24,6 +24,8 @@ import (
 	"github.com/osbuild/osbuild-composer/internal/rpmmd"
 )
 
+const cacheRoot = "/tmp/rpmmd"
+
 type repository struct {
 	Name           string   `json:"name"`
 	BaseURL        string   `json:"baseurl,omitempty"`
@@ -98,7 +100,7 @@ func makeManifestJob(name string, imgType distro.ImageType, cr composeRequest, d
 		return strings.Replace(s, "-", "_", -1)
 	}
 	filename := fmt.Sprintf("%s-%s-%s-boot.json", u(distroName), u(archName), u(name))
-	cacheDir := filepath.Join("/tmp", "rpmmd", archName+distribution.Name())
+	cacheDir := filepath.Join(cacheRoot, archName+distribution.Name())
 
 	options := distro.ImageOptions{Size: 0}
 	if cr.OSTree != nil {
@@ -378,11 +380,14 @@ func main() {
 		wq.submitJob(j)
 	}
 	errs := wq.wait()
+	exit := 0
 	if len(errs) > 0 {
 		fmt.Printf("Encountered %d errors:\n", len(errs))
 		for idx, err := range errs {
 			fmt.Printf("%3d: %s\n", idx, err.Error())
 		}
-		os.Exit(1)
+		exit = 1
 	}
+	fmt.Printf("RPM metadata cache kept in %s\n", cacheRoot)
+	os.Exit(exit)
 }
diff --git a/internal/dnfjson/cache.go b/internal/dnfjson/cache.go
index 926fdca1d..225e45c9d 100644
--- a/internal/dnfjson/cache.go
+++ b/internal/dnfjson/cache.go
@@ -1,11 +1,200 @@
 package dnfjson
 
 import (
+	"fmt"
 	"io/fs"
 	"os"
 	"path/filepath"
+	"sort"
+	"sync"
+	"time"
+
+	"github.com/gobwas/glob"
 )
 
+// global cache locker
+var cacheLocks sync.Map
+
+// A collection of directory paths, their total size, and their most recent
+// modification time.
+type pathInfo struct {
+	paths []string
+	size  uint64
+	mtime time.Time
+}
+
+type rpmCache struct {
+	// root path for the cache
+	root string
+
+	// individual repository cache data
+	repoElements map[string]pathInfo
+
+	// list of known repository IDs, sorted by mtime
+	repoRecency []string
+
+	// total cache size
+	size uint64
+
+	// max cache size
+	maxSize uint64
+
+	// locker for this cache directory
+	locker *sync.RWMutex
+}
+
+func newRPMCache(path string, maxSize uint64) *rpmCache {
+	absPath, err := filepath.Abs(path) // convert to abs if it's not already
+	if err != nil {
+		panic(err) // can only happen if the CWD does not exist and the path isn't already absolute
+	}
+	path = absPath
+	locker := new(sync.RWMutex)
+	if l, loaded := cacheLocks.LoadOrStore(path, locker); loaded {
+		// value existed and was loaded
+		locker = l.(*sync.RWMutex)
+	}
+	r := &rpmCache{
+		root:         path,
+		repoElements: make(map[string]pathInfo),
+		size:         0,
+		maxSize:      maxSize,
+		locker:       locker,
+	}
+	// collect existing cache paths and timestamps
+	r.updateInfo()
+	return r
+}
+
+// updateInfo updates the repoPaths and repoRecency fields of the rpmCache.
+func (r *rpmCache) updateInfo() {
+	cacheEntries, _ := os.ReadDir(r.root)
+
+	// each repository has multiple cache entries (3 on average), so using the
+	// number of cacheEntries to allocate the map and ID slice is a high upper
+	// bound, but guarantees we wont need to grow and reallocate either.
+	repos := make(map[string]pathInfo, len(cacheEntries))
+	repoIDs := make([]string, 0, len(cacheEntries))
+
+	var totalSize uint64
+
+	// Collect the paths grouped by their repo ID
+	// We assume the first 64 characters of a file or directory name are the
+	// repository ID because we use a sha256 sum of the repository config to
+	// create the ID (64 hex chars)
+	for _, entry := range cacheEntries {
+		eInfo, err := entry.Info()
+		if err != nil {
+			// skip it
+			continue
+		}
+
+		fname := entry.Name()
+		if len(fname) < 64 {
+			// unknown file in cache; ignore
+			continue
+		}
+		repoID := fname[:64]
+		repo, ok := repos[repoID]
+		if !ok {
+			// new repo ID
+			repoIDs = append(repoIDs, repoID)
+		}
+		mtime := eInfo.ModTime()
+		ePath := filepath.Join(r.root, entry.Name())
+
+		// calculate and add entry size
+		size, err := dirSize(ePath)
+		if err != nil {
+			// skip it
+			continue
+		}
+		repo.size += size
+		totalSize += size
+
+		// add path
+		repo.paths = append(repo.paths, ePath)
+
+		// if for some reason the mtimes of the various entries of a single
+		// repository are out of sync, use the most recent one
+		if repo.mtime.Before(mtime) {
+			repo.mtime = mtime
+		}
+
+		// update the collection
+		repos[repoID] = repo
+	}
+	sortFunc := func(idx, jdx int) bool {
+		ir := repos[repoIDs[idx]]
+		jr := repos[repoIDs[jdx]]
+		return ir.mtime.Before(jr.mtime)
+	}
+
+	// sort IDs by mtime (oldest first)
+	sort.Slice(repoIDs, sortFunc)
+
+	r.size = totalSize
+	r.repoElements = repos
+	r.repoRecency = repoIDs
+}
+
+func (r *rpmCache) shrink() error {
+	r.locker.Lock()
+	defer r.locker.Unlock()
+
+	// start deleting until we drop below r.maxSize
+	nDeleted := 0
+	for idx := 0; idx < len(r.repoRecency) && r.size >= r.maxSize; idx++ {
+		repoID := r.repoRecency[idx]
+		nDeleted++
+		repo, ok := r.repoElements[repoID]
+		if !ok {
+			// cache inconsistency?
+			// ignore and let the ID be removed from the recency list
+			continue
+		}
+		for _, gPath := range repo.paths {
+			if err := os.RemoveAll(gPath); err != nil {
+				return err
+			}
+		}
+		r.size -= repo.size
+		delete(r.repoElements, repoID)
+	}
+
+	// update recency list
+	r.repoRecency = r.repoRecency[nDeleted:]
+	return nil
+}
+
+// Update file atime and mtime on the filesystem to time t for all files in the
+// root of the cache that match the repo ID.  This should be called whenever a
+// repository is used.
+// This function does not update the internal cache info.  A call to
+// updateInfo() should be made after touching one or more repositories.
+func (r *rpmCache) touchRepo(repoID string, t time.Time) error {
+	repoGlob, err := glob.Compile(fmt.Sprintf("%s*", repoID))
+	if err != nil {
+		return err
+	}
+
+	// we only touch the top-level directories and files of the cache
+	cacheEntries, err := os.ReadDir(r.root)
+	if err != nil {
+		return err
+	}
+
+	for _, cacheEntry := range cacheEntries {
+		if repoGlob.Match(cacheEntry.Name()) {
+			path := filepath.Join(r.root, cacheEntry.Name())
+			if err := os.Chtimes(path, t, t); err != nil {
+				return err
+			}
+		}
+	}
+	return nil
+}
+
 func dirSize(path string) (uint64, error) {
 	var size uint64
 	sizer := func(path string, info fs.FileInfo, err error) error {
@@ -18,14 +207,3 @@ func dirSize(path string) (uint64, error) {
 	err := filepath.Walk(path, sizer)
 	return size, err
 }
-
-func (bs *BaseSolver) CleanCache() error {
-	curSize, err := dirSize(bs.cacheDir)
-	if err != nil {
-		return err
-	}
-	if curSize > bs.maxCacheSize {
-		return os.RemoveAll(bs.cacheDir)
-	}
-	return nil
-}
diff --git a/internal/dnfjson/dnfjson.go b/internal/dnfjson/dnfjson.go
index 223b55c7a..5d29a4319 100644
--- a/internal/dnfjson/dnfjson.go
+++ b/internal/dnfjson/dnfjson.go
@@ -21,21 +21,24 @@ import (
 	"os"
 	"os/exec"
 	"sort"
+	"time"
 
 	"github.com/osbuild/osbuild-composer/internal/rhsm"
 	"github.com/osbuild/osbuild-composer/internal/rpmmd"
 )
 
+// BaseSolver defines the basic solver configuration without platform
+// information. It can be used to create configured Solver instances with the
+// NewWithConfig() method. A BaseSolver maintains the global repository cache
+// directory and system subscription information.
 type BaseSolver struct {
 	subscriptions *rhsm.Subscriptions
 
-	// Cache directory for the DNF metadata
-	cacheDir string
+	// Cache information
+	cache *rpmCache
 
 	// Path to the dnf-json binary and optional args (default: "/usr/libexec/osbuild-composer/dnf-json")
 	dnfJsonCmd []string
-
-	maxCacheSize uint64
 }
 
 // Create a new unconfigured BaseSolver (without platform information). It can
@@ -44,15 +47,17 @@ type BaseSolver struct {
 func NewBaseSolver(cacheDir string) *BaseSolver {
 	subscriptions, _ := rhsm.LoadSystemSubscriptions()
 	return &BaseSolver{
-		cacheDir:      cacheDir,
+		cache:         newRPMCache(cacheDir, 524288000), // 500 MiB
 		subscriptions: subscriptions,
 		dnfJsonCmd:    []string{"/usr/libexec/osbuild-composer/dnf-json"},
-		maxCacheSize:  524288000, // 500 MiB
 	}
 }
 
+// SetMaxCacheSize sets the maximum size for the global repository metadata
+// cache. This is the maximum size of the cache after a CleanCache()
+// call. Cache cleanup is never performed automatically.
 func (s *BaseSolver) SetMaxCacheSize(size uint64) {
-	s.maxCacheSize = size
+	s.cache.maxSize = size
 }
 
 // SetDNFJSONPath sets the path to the dnf-json binary and optionally any command line arguments.
@@ -71,6 +76,13 @@ func (bs *BaseSolver) NewWithConfig(modulePlatformID string, releaseVer string,
 	return s
 }
 
+// CleanCache deletes the least recently used repository metadata caches until
+// the total size of the cache falls below the configured maximum size (see
+// SetMaxCacheSize()).
+func (bs *BaseSolver) CleanCache() error {
+	return bs.cache.shrink()
+}
+
 // Solver is configured with system information in order to resolve
 // dependencies for RPM packages using DNF.
 type Solver struct {
@@ -103,10 +115,22 @@ func (s *Solver) Depsolve(pkgSets []rpmmd.PackageSet) ([]rpmmd.PackageSpec, erro
 		return nil, err
 	}
 
+	// get non-exclusive read lock
+	s.cache.locker.RLock()
+	defer s.cache.locker.RUnlock()
+
 	output, err := run(s.dnfJsonCmd, req)
 	if err != nil {
 		return nil, err
 	}
+	// touch repos to now
+	now := time.Now().Local()
+	for _, r := range repoMap {
+		// ignore errors
+		_ = s.cache.touchRepo(r.Hash(), now)
+	}
+	s.cache.updateInfo()
+
 	var result packageSpecs
 	if err := json.Unmarshal(output, &result); err != nil {
 		return nil, err
@@ -115,16 +139,31 @@ func (s *Solver) Depsolve(pkgSets []rpmmd.PackageSet) ([]rpmmd.PackageSpec, erro
 	return result.toRPMMD(repoMap), nil
 }
 
+// FetchMetadata returns the list of all the available packages in repos and
+// their info.
 func (s *Solver) FetchMetadata(repos []rpmmd.RepoConfig) (rpmmd.PackageList, error) {
 	req, err := s.makeDumpRequest(repos)
 	if err != nil {
 		return nil, err
 	}
+
+	// get non-exclusive read lock
+	s.cache.locker.RLock()
+	defer s.cache.locker.RUnlock()
+
 	result, err := run(s.dnfJsonCmd, req)
 	if err != nil {
 		return nil, err
 	}
 
+	// touch repos to now
+	now := time.Now().Local()
+	for _, r := range repos {
+		// ignore errors
+		_ = s.cache.touchRepo(r.Hash(), now)
+	}
+	s.cache.updateInfo()
+
 	var pkgs rpmmd.PackageList
 	if err := json.Unmarshal(result, &pkgs); err != nil {
 		return nil, err
@@ -254,7 +293,7 @@ func (s *Solver) makeDepsolveRequest(pkgSets []rpmmd.PackageSet) (*Request, map[
 		Command:          "depsolve",
 		ModulePlatformID: s.modulePlatformID,
 		Arch:             s.arch,
-		CacheDir:         s.cacheDir,
+		CacheDir:         s.cache.root,
 		Arguments:        args,
 	}
 
@@ -271,7 +310,7 @@ func (s *Solver) makeDumpRequest(repos []rpmmd.RepoConfig) (*Request, error) {
 		Command:          "dump",
 		ModulePlatformID: s.modulePlatformID,
 		Arch:             s.arch,
-		CacheDir:         s.cacheDir,
+		CacheDir:         s.cache.root,
 		Arguments: arguments{
 			Repos: dnfRepos,
 		},


Test patch:
diff --git a/internal/dnfjson/cache_test.go b/internal/dnfjson/cache_test.go
new file mode 100644
index 000000000..5dfa05455
--- /dev/null
+++ b/internal/dnfjson/cache_test.go
@@ -0,0 +1,275 @@
+package dnfjson
+
+import (
+	"io/fs"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+	"time"
+
+	"github.com/stretchr/testify/assert"
+)
+
+func truncate(path string, size int64) {
+	fp, err := os.Create(path)
+	if err != nil {
+		panic(err)
+	}
+	defer fp.Close()
+	if err := fp.Truncate(size); err != nil {
+		panic(err)
+	}
+}
+
+// create a test cache based on the config, where the config keys are file
+// paths and the values are file sizes
+func createTestCache(root string, config testCache) uint64 {
+	var totalSize uint64
+	for path, fi := range config {
+		fullPath := filepath.Join(root, path)
+		parPath := filepath.Dir(fullPath)
+		if err := os.MkdirAll(parPath, 0770); err != nil {
+			panic(err)
+		}
+		truncate(fullPath, int64(fi.size))
+		mtime := time.Unix(fi.mtime, 0)
+		if err := os.Chtimes(fullPath, mtime, mtime); err != nil {
+			panic(err)
+		}
+
+		// if the path has multiple parts, touch the top level directory of the
+		// element
+		pathParts := strings.Split(path, "/")
+		if len(pathParts) > 1 {
+			top := pathParts[0]
+			if err := os.Chtimes(filepath.Join(root, top), mtime, mtime); err != nil {
+				panic(err)
+			}
+		}
+		if len(path) >= 64 {
+			// paths with shorter names will be ignored by the cache manager
+			totalSize += fi.size
+		}
+	}
+
+	// add directory sizes to total
+	sizer := func(path string, info fs.FileInfo, err error) error {
+		if err != nil {
+			return err
+		}
+		if path == root {
+			// don't count root
+			return nil
+		}
+		if info.IsDir() {
+			totalSize += uint64(info.Size())
+		}
+		return nil
+	}
+	if err := filepath.Walk(root, sizer); err != nil {
+		panic(err)
+	}
+
+	return totalSize
+}
+
+type fileInfo struct {
+	size  uint64
+	mtime int64
+}
+
+type testCache map[string]fileInfo
+
+var testCfgs = map[string]testCache{
+	"rhel84-aarch64": { // real repo metadata file names and sizes
+		"9adf133053f0691a0ec12e73cbf1875a90c9268b4f09162fc3387fd76ecb3bcc.solv":                                                                                                                      fileInfo{2095095, 100},
+		"9adf133053f0691a0ec12e73cbf1875a90c9268b4f09162fc3387fd76ecb3bcc-filenames.solvx":                                                                                                           fileInfo{14473401, 100},
+		"9adf133053f0691a0ec12e73cbf1875a90c9268b4f09162fc3387fd76ecb3bcc-33d346d177279673/repodata/gen/groups.xml":                                                                                  fileInfo{1419587, 100},
+		"9adf133053f0691a0ec12e73cbf1875a90c9268b4f09162fc3387fd76ecb3bcc-33d346d177279673/repodata/3eabd1122210e4def18ae4b96a18aa5bcc186abf2ec14e2e8f1c1bb1ab4d11da-modules.yaml.gz":                fileInfo{156314, 100},
+		"9adf133053f0691a0ec12e73cbf1875a90c9268b4f09162fc3387fd76ecb3bcc-33d346d177279673/repodata/90fd2e7463220a07457e76ae905e1bad754c29e22202bb3202c971a5ece28396-comps-AppStream.aarch64.xml.gz": fileInfo{199426, 100},
+		"9adf133053f0691a0ec12e73cbf1875a90c9268b4f09162fc3387fd76ecb3bcc-33d346d177279673/repodata/77a66c76b5f6ba51aaee6c0cf76d701601e8b622d1701d1781dabec434f27413-filelists.xml.gz":               fileInfo{14370201, 100},
+		"9adf133053f0691a0ec12e73cbf1875a90c9268b4f09162fc3387fd76ecb3bcc-33d346d177279673/repodata/1941c723c94218eed43eac3174aa94cefbe921e15547c39251a95895024207ca-primary.xml.gz":                 fileInfo{11439375, 100},
+		"9adf133053f0691a0ec12e73cbf1875a90c9268b4f09162fc3387fd76ecb3bcc-33d346d177279673/repodata/repomd.xml":                                                                                      fileInfo{13285, 100},
+		"df2665154150abf76f4d86156228a75c39f3f31a79d4a861d76b1edd89814b62.solv":                                                                                                                      fileInfo{1147863, 300},
+		"df2665154150abf76f4d86156228a75c39f3f31a79d4a861d76b1edd89814b62-filenames.solvx":                                                                                                           fileInfo{11133964, 300},
+		"df2665154150abf76f4d86156228a75c39f3f31a79d4a861d76b1edd89814b62-98177081b9162766/repodata/gen/groups.xml":                                                                                  fileInfo{1298102, 300},
+		"df2665154150abf76f4d86156228a75c39f3f31a79d4a861d76b1edd89814b62-98177081b9162766/repodata/d74783221709ab27d543c1cfc4c02562fde6edfaaaac33ac73a68ecf53188695-comps-BaseOS.aarch64.xml.gz":    fileInfo{174076, 300},
+		"df2665154150abf76f4d86156228a75c39f3f31a79d4a861d76b1edd89814b62-98177081b9162766/repodata/5ded48b4c9e238288130c6670d99f5febdb7273e4a31ac213836a15a2076514d-filelists.xml.gz":               fileInfo{11081612, 300},
+		"df2665154150abf76f4d86156228a75c39f3f31a79d4a861d76b1edd89814b62-98177081b9162766/repodata/8120caf8ebbb8c8b37f6f0dd027d866020ebe7acf9c9ce49ae9903b761986f0c-primary.xml.gz":                 fileInfo{1836471, 300},
+		"df2665154150abf76f4d86156228a75c39f3f31a79d4a861d76b1edd89814b62-98177081b9162766/repodata/repomd.xml":                                                                                      fileInfo{12817, 300},
+	},
+	"fake-real": { // fake but resembling real data
+		"1111111111111111111111111111111111111111111111111111111111111111.solv":           fileInfo{100, 0},
+		"1111111111111111111111111111111111111111111111111111111111111111-filenames.solv": fileInfo{200, 0},
+		"1111111111111111111111111111111111111111111111111111111111111111.whatever":       fileInfo{110, 0},
+		"1111111111111111111111111111111111111111111111111111111111111111/repodata/a":     fileInfo{1000, 0},
+		"1111111111111111111111111111111111111111111111111111111111111111/repodata/b":     fileInfo{3829, 0},
+		"1111111111111111111111111111111111111111111111111111111111111111/repodata/c":     fileInfo{831989, 0},
+		"2222222222222222222222222222222222222222222222222222222222222222.solv":           fileInfo{120, 2},
+		"2222222222222222222222222222222222222222222222222222222222222222-filenames.solv": fileInfo{232, 2},
+		"2222222222222222222222222222222222222222222222222222222222222222.whatever":       fileInfo{110, 2},
+		"2222222222222222222222222222222222222222222222222222222222222222/repodata/a":     fileInfo{1000, 2},
+		"2222222222222222222222222222222222222222222222222222222222222222/repodata/b":     fileInfo{3829, 2},
+		"2222222222222222222222222222222222222222222222222222222222222222/repodata/c":     fileInfo{831989, 2},
+		"3333333333333333333333333333333333333333333333333333333333333333.solv":           fileInfo{105, 4},
+		"3333333333333333333333333333333333333333333333333333333333333333-filenames.solv": fileInfo{200, 4},
+		"3333333333333333333333333333333333333333333333333333333333333333.whatever":       fileInfo{110, 4},
+		"3333333333333333333333333333333333333333333333333333333333333333/repodata/a":     fileInfo{2390, 4},
+		"3333333333333333333333333333333333333333333333333333333333333333/repodata/b":     fileInfo{1234890, 4},
+		"3333333333333333333333333333333333333333333333333333333333333333/repodata/c":     fileInfo{483, 4},
+	},
+	"completely-fake": { // just a mess of files (including files without a repo ID)
+		"somefile": fileInfo{192, 10291920},
+		"yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy-repofiley":  fileInfo{29384, 11},
+		"yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy-repofiley2": fileInfo{293, 31},
+		"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa-repofile":   fileInfo{29384, 30},
+		"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa-repofileb":  fileInfo{293, 45},
+	},
+}
+
+type testCase struct {
+	cache              testCache
+	maxSize            uint64
+	minSizeAfterShrink uint64
+	repoIDsAfterShrink []string
+}
+
+func getRepoIDs(ct testCache) []string {
+	idMap := make(map[string]bool)
+	ids := make([]string, 0)
+	for path := range ct {
+		if len(path) >= 64 {
+			id := path[:64]
+			if !idMap[id] {
+				idMap[id] = true
+				ids = append(ids, id)
+			}
+		}
+	}
+	return ids
+}
+
+func TestCacheRead(t *testing.T) {
+	assert := assert.New(t)
+	for name, cfg := range testCfgs {
+		t.Run(name, func(t *testing.T) {
+			testCacheRoot := t.TempDir()
+			s := createTestCache(testCacheRoot, cfg)
+
+			cache := newRPMCache(testCacheRoot, 1048576) // 1 MiB, but doesn't matter for this test
+
+			nrepos := len(getRepoIDs(cfg))
+			assert.Equal(s, cache.size)
+			assert.Equal(nrepos, len(cache.repoElements))
+			assert.Equal(nrepos, len(cache.repoRecency))
+		})
+	}
+}
+
+func strSliceContains(slc []string, elem string) bool {
+	for _, s := range slc {
+		if elem == s {
+			return true
+		}
+	}
+	return false
+}
+
+func sizeSum(cfg testCache, repoIDFilter ...string) uint64 {
+	var sum uint64
+	for path, info := range cfg {
+		if len(path) < 64 {
+			continue
+		}
+		rid := path[:64]
+		if len(repoIDFilter) == 0 || (len(repoIDFilter) > 0 && strSliceContains(repoIDFilter, rid)) {
+			sum += info.size
+		}
+	}
+	return sum
+}
+
+func TestCacheCleanup(t *testing.T) {
+	rhelRecentRepoSize := sizeSum(testCfgs["rhel84-aarch64"], "df2665154150abf76f4d86156228a75c39f3f31a79d4a861d76b1edd89814b62")
+	rhelTotalRepoSize := sizeSum(testCfgs["rhel84-aarch64"])
+
+	fakeRealSize2 := sizeSum(testCfgs["fake-real"], "2222222222222222222222222222222222222222222222222222222222222222")
+	fakeRealSize3 := sizeSum(testCfgs["fake-real"], "3333333333333333333333333333333333333333333333333333333333333333")
+
+	testCases := map[string]testCase{
+		// max size 1 byte -> clean will delete everything
+		"fake-real-full-delete": {
+			cache:              testCfgs["fake-real"],
+			maxSize:            1,
+			minSizeAfterShrink: 0,
+		},
+		"rhel-full-delete": {
+			cache:              testCfgs["rhel84-aarch64"],
+			maxSize:            1,
+			minSizeAfterShrink: 0,
+		},
+		"completely-fake-full-delete": {
+			cache:              testCfgs["completely-fake"],
+			maxSize:            1,
+			minSizeAfterShrink: 0,
+		},
+		"completely-fake-full-delete-2": {
+			cache:              testCfgs["completely-fake"],
+			maxSize:            100,
+			minSizeAfterShrink: 0,
+		},
+		// max size a bit larger than most recent repo -> clean will delete older repos
+		"completely-fake-half-delete": {
+			cache:              testCfgs["completely-fake"],
+			maxSize:            29384 + 293 + 1,                                                              // one byte larger than the files of one repo
+			minSizeAfterShrink: 29384 + 293,                                                                  // size of files from one repo
+			repoIDsAfterShrink: []string{"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"}, // most recent repo timestamp (45)
+		},
+		"rhel-half-delete": {
+			cache:              testCfgs["rhel84-aarch64"],
+			maxSize:            rhelRecentRepoSize + 102400,                                                  // most recent repo file sizes + 100k buffer (for directories)
+			minSizeAfterShrink: rhelRecentRepoSize,                                                           // after shrink it should be at least as big as the most recent repo
+			repoIDsAfterShrink: []string{"df2665154150abf76f4d86156228a75c39f3f31a79d4a861d76b1edd89814b62"}, // most recent repo timestamp (45)
+		},
+		"fake-real-delete-1": {
+			cache:              testCfgs["fake-real"],
+			maxSize:            fakeRealSize3 + fakeRealSize2 + 102400,
+			minSizeAfterShrink: fakeRealSize3 + fakeRealSize2,
+			repoIDsAfterShrink: []string{"3333333333333333333333333333333333333333333333333333333333333333", "2222222222222222222222222222222222222222222222222222222222222222"},
+		},
+		"fake-real-delete-2": {
+			cache:              testCfgs["fake-real"],
+			maxSize:            fakeRealSize3 + 102400,
+			minSizeAfterShrink: fakeRealSize3,
+			repoIDsAfterShrink: []string{"3333333333333333333333333333333333333333333333333333333333333333"},
+		},
+		// max size is huge -> clean wont delete anything
+		"rhel-no-delete": {
+			cache:              testCfgs["rhel84-aarch64"],
+			maxSize:            45097156608, // 42 GiB
+			minSizeAfterShrink: rhelTotalRepoSize,
+			repoIDsAfterShrink: []string{"df2665154150abf76f4d86156228a75c39f3f31a79d4a861d76b1edd89814b62", "9adf133053f0691a0ec12e73cbf1875a90c9268b4f09162fc3387fd76ecb3bcc"},
+		},
+	}
+
+	for name, cfg := range testCases {
+		t.Run(name, func(t *testing.T) {
+			assert := assert.New(t)
+			testCacheRoot := t.TempDir()
+			createTestCache(testCacheRoot, cfg.cache)
+			cache := newRPMCache(testCacheRoot, cfg.maxSize)
+			err := cache.shrink()
+			assert.NoError(err)
+
+			// it's hard to predict the exact size after shrink because of directory sizes
+			// so let's just check that the new size is between min and max
+			assert.LessOrEqual(cfg.minSizeAfterShrink, cache.size)
+			assert.Greater(cfg.maxSize, cache.size)
+			assert.Equal(len(cfg.repoIDsAfterShrink), len(cache.repoElements))
+			for _, id := range cfg.repoIDsAfterShrink {
+				assert.Contains(cache.repoElements, id)
+			}
+		})
+	}
+}