Repository: osbuild/osbuild
Pull request #1187 (2022-11-28 18:50:20)

🗄Write and read metadata from the store and integrate `FsCache` into `ObjectStore`
This is a combination of two "patch series":

1) #1186
2) Integrate `FsCache` into `ObjectStore`

## 🗄 write and read metadata from the store

Enhance `Object` to be able to store arbitrary metadata alongside its tree and change the stage code to do so. When serialising the results, read the metadata from via the `Object` instead of getting it from `BuildResult`. When the object is later committed the metadata will also be saved. This ensures that all metadata is also present for the object when later builds are resumed from that object, or a previously built object is exported. 

## Integrate FsCache into ObjectStore

Integrate the recently added file system cache `FsCache` into our object store `ObjectStore`. NB: This changes the semantics of it: previously a call to `ObjectStore.commit` resulted in the object being in the cache (i/o errors aside). But `FsCache.store`, which is now the backing store for objects, will only commit objects if there is enough space left. Thus we cannot rely that objects are present for reading after a call to `FsCache.store`. To cope with this we now always copy the object into the cache, even for cases where we previously moved it: for the case where commit is called with `object_id` matching `Object.id`, which is the case for when `commit` is called for last stage in the pipeline. We could keep this optimization but then we would have to special case it and not call `commit` for these cases but only after we exported all objects; or in other words, after we are sure we will never read from any committed object again. The extra complexity seems not worth it for the little gain of the optimization.
Convert all the tests for the new semantic and also remove a lot of them that make no sense under this new paradigm.

Add a new command line option `--cache-max-size` which will set the maximum size of the cache, if specified.

gicmo: I realised embarrassingly late that the integration approach here is fundamentally flawed, because with the `FsCache` introduction the semantics of the `ObjectStore` changes fundamentally. Before, `ObjectStore` was indeed a store of objects, i.e. a database, in the sense that every `commit` of an `Object` would result in an `Object` with the given `object_id` being present in the store (i/o errors aside). Now with the introduction of the `FsCache` we `ObjectStore` turns into an object cache where `commit` might or might not result in the object being available in the store, depending on the configuration and available space. As a result we cannot move (`rename(2)`) the object  into the store if we later need to be able to rename it. Thus either 1) we have different code paths for snapshots of intermediate stages and the last stage of a pipeline or 2) we always copy and take a slight inefficient penalty for the first build.
NB: we already have slightly different code paths _within_ `commit` now since we always make a copy for intermediate paths before the rename, which is omitted for the last stage. But 1) would mean to not call `commit` for the last stage during the build but then calling it after the full build and exports of artefacts have completed.
I don't think that is worth it and it could lead to a simpler, potential more efficient API for `FsCache.store`, which could just take a `path` and copying and size calculation could be done in one.

Patch:
diff --git a/docs/osbuild.1.rst b/docs/osbuild.1.rst
index 019e3ac4..1cab5f85 100644
--- a/docs/osbuild.1.rst
+++ b/docs/osbuild.1.rst
@@ -43,6 +43,9 @@ is not listed here, **osbuild** will deny startup and exit with an error.
                                 are stored
 -l DIR, --libdir=DIR            directory containing stages, assemblers, and
                                 the osbuild library
+--cache-max-size=SIZE           maximum size of the cache (bytes) or 'unlimited'
+                                for no restriction (size may include an optional
+                                unit suffix, like kB, kiB, MB, MiB and so on)
 --checkpoint=CHECKPOINT         stage to commit to the object store during
                                 build (can be passed multiple times)
 --export=OBJECT                 object to export (can be passed multiple times)
diff --git a/osbuild/api.py b/osbuild/api.py
index d5f90fde..40daf8f6 100644
--- a/osbuild/api.py
+++ b/osbuild/api.py
@@ -142,15 +142,8 @@ class API(BaseAPI):
 
     def __init__(self, *, socket_address=None):
         super().__init__(socket_address)
-        self.metadata = {}
         self.error = None
 
-    def _set_metadata(self, message, fds):
-        fd = message["metadata"]
-        with os.fdopen(fds.steal(fd), encoding="utf8") as f:
-            data = json.load(f)
-        self.metadata.update(data)
-
     def _get_exception(self, message):
         self.error = {
             "type": "exception",
@@ -158,9 +151,7 @@ class API(BaseAPI):
         }
 
     def _message(self, msg, fds, sock):
-        if msg["method"] == 'add-metadata':
-            self._set_metadata(msg, fds)
-        elif msg["method"] == 'exception':
+        if msg["method"] == 'exception':
             self._get_exception(msg)
 
 
@@ -200,18 +191,8 @@ def arguments(path="/run/osbuild/api/arguments"):
     return data
 
 
-def metadata(data: Dict, path="/run/osbuild/api/osbuild"):
+def metadata(data: Dict, path="/run/osbuild/meta"):
     """Update metadata for the current module"""
 
-    def data_to_file():
-        with tempfile.TemporaryFile() as f:
-            f.write(json.dumps(data).encode('utf8'))
-            # re-open the file to get a read-only file descriptor
-            return open(f"/proc/self/fd/{f.fileno()}", "r", encoding="utf8")
-
-    with jsoncomm.Socket.new_client(path) as client, data_to_file() as f:
-        msg = {
-            "method": "add-metadata",
-            "metadata": 0
-        }
-        client.send(msg, fds=[f.fileno()])
+    with open(path, "w", encoding="utf8") as f:
+        json.dump(data, f, indent=2)
diff --git a/osbuild/formats/v1.py b/osbuild/formats/v1.py
index 5943fa54..76e9d5c8 100644
--- a/osbuild/formats/v1.py
+++ b/osbuild/formats/v1.py
@@ -195,17 +195,17 @@ def load(description: Dict, index: Index) -> Manifest:
     return manifest
 
 
-def output(manifest: Manifest, res: Dict) -> Dict:
+def output(manifest: Manifest, res: Dict, store=None) -> Dict:
     """Convert a result into the v1 format"""
 
-    def result_for_stage(result: BuildResult):
+    def result_for_stage(result: BuildResult, obj):
         return {
             "id": result.id,
             "type": result.name,
             "success": result.success,
             "error": result.error,
             "output": result.output,
-            "metadata": result.metadata,
+            "metadata": obj and obj.meta.get(result.id),
         }
 
     def result_for_pipeline(pipeline):
@@ -223,10 +223,12 @@ def output(manifest: Manifest, res: Dict) -> Dict:
             retval["build"] = result_for_pipeline(build)
             retval["success"] = retval["build"]["success"]
 
+        obj = store and pipeline.id and store.get(pipeline.id)
+
         stages = current.get("stages")
         if stages:
             retval["stages"] = [
-                result_for_stage(r) for r in stages
+                result_for_stage(r, obj) for r in stages
             ]
         return retval
 
@@ -244,8 +246,9 @@ def output(manifest: Manifest, res: Dict) -> Dict:
 
     # The assembler pipeline must have exactly one stage
     # which is the v1 assembler
+    obj = store and store.get(assembler.id)
     stage = current["stages"][0]
-    result["assembler"] = result_for_stage(stage)
+    result["assembler"] = result_for_stage(stage, obj)
     if not result["assembler"]["success"]:
         result["success"] = False
 
diff --git a/osbuild/formats/v2.py b/osbuild/formats/v2.py
index a70d6297..5640b6ac 100644
--- a/osbuild/formats/v2.py
+++ b/osbuild/formats/v2.py
@@ -7,6 +7,7 @@ from typing import Any, Dict
 from osbuild.meta import Index, ModuleInfo, ValidationResult
 
 from ..inputs import Input
+from ..objectstore import ObjectStore
 from ..pipeline import Manifest, Pipeline, Runner, Stage
 from ..sources import Source
 
@@ -391,9 +392,28 @@ def load(description: Dict, index: Index) -> Manifest:
 
 
 #pylint: disable=too-many-branches
-def output(manifest: Manifest, res: Dict) -> Dict:
+def output(manifest: Manifest, res: Dict, store: ObjectStore = None) -> Dict:
     """Convert a result into the v2 format"""
 
+    def collect_metadata(p: Pipeline) -> Dict[str, Any]:
+        data: Dict[str, Any] = {}
+
+        if not store:  # for testing
+            return data
+
+        obj = store.get(p.id)
+        if not obj:
+            return data
+
+        for stage in p.stages:
+            md = obj.meta.get(stage.id)
+            if not md:
+                continue
+            val = data.setdefault(stage.name, {})
+            val.update(md)
+
+        return data
+
     result: Dict[str, Any] = {}
 
     if not res["success"]:
@@ -424,16 +444,7 @@ def output(manifest: Manifest, res: Dict) -> Dict:
 
         # gather all the metadata
         for p in manifest.pipelines.values():
-            data: Dict[str, Any] = {}
-            r = res.get(p.id, {})
-            for stage in r.get("stages", []):
-                md = stage.metadata
-                if not md:
-                    continue
-                name = stage.name
-                val = data.setdefault(name, {})
-                val.update(md)
-
+            data: Dict[str, Any] = collect_metadata(p)
             if data:
                 result["metadata"][p.name] = data
 
diff --git a/osbuild/main_cli.py b/osbuild/main_cli.py
index 195d01fd..d1ebfdb0 100644
--- a/osbuild/main_cli.py
+++ b/osbuild/main_cli.py
@@ -15,6 +15,7 @@ import osbuild
 import osbuild.meta
 import osbuild.monitor
 from osbuild.objectstore import ObjectStore
+from osbuild.util.parsing import parse_size
 from osbuild.util.term import fmt as vt
 
 
@@ -66,6 +67,8 @@ def parse_arguments(sys_argv):
                         help="directory where intermediary os trees are stored")
     parser.add_argument("-l", "--libdir", metavar="DIRECTORY", type=os.path.abspath, default="/usr/lib/osbuild",
                         help="directory containing stages, assemblers, and the osbuild library")
+    parser.add_argument("--cache-max-size", metavar="SIZE", type=parse_size, default=None,
+                        help="maximum size of the cache (bytes) or 'unlimited' for no restriction")
     parser.add_argument("--checkpoint", metavar="ID", action="append", type=str, default=None,
                         help="stage to commit to the object store during build (can be passed multiple times)")
     parser.add_argument("--export", metavar="ID", action="append", type=str, default=[],
@@ -150,6 +153,9 @@ def osbuild_cli():
 
     try:
         with ObjectStore(args.store) as object_store:
+            if args.cache_max_size is not None:
+                object_store.maximum_size = args.cache_max_size
+
             stage_timeout = args.stage_timeout
 
             pipelines = manifest.depsolve(object_store, exports)
@@ -168,21 +174,21 @@ def osbuild_cli():
                 for pid in exports:
                     export(pid, output_directory, object_store, manifest)
 
+            if args.json:
+                r = fmt.output(manifest, r, object_store)
+                json.dump(r, sys.stdout)
+                sys.stdout.write("\n")
+            else:
+                if r["success"]:
+                    for name, pl in manifest.pipelines.items():
+                        print(f"{name + ':': <10}\t{pl.id}")
+                else:
+                    print()
+                    print(f"{vt.reset}{vt.bold}{vt.red}Failed{vt.reset}")
+
+            return 0 if r["success"] else 1
+
     except KeyboardInterrupt:
         print()
         print(f"{vt.reset}{vt.bold}{vt.red}Aborted{vt.reset}")
         return 130
-
-    if args.json:
-        r = fmt.output(manifest, r)
-        json.dump(r, sys.stdout)
-        sys.stdout.write("\n")
-    else:
-        if r["success"]:
-            for name, pl in manifest.pipelines.items():
-                print(f"{name + ':': <10}\t{pl.id}")
-        else:
-            print()
-            print(f"{vt.reset}{vt.bold}{vt.red}Failed{vt.reset}")
-
-    return 0 if r["success"] else 1
diff --git a/osbuild/objectstore.py b/osbuild/objectstore.py
index e8cdb36e..606e79e3 100644
--- a/osbuild/objectstore.py
+++ b/osbuild/objectstore.py
@@ -1,12 +1,13 @@
 import contextlib
 import enum
+import json
 import os
 import subprocess
 import tempfile
-import uuid
-from typing import Optional, Set
+from typing import Any, Optional, Set, Union
 
-from osbuild.util import jsoncomm, rmrf
+from osbuild.util import jsoncomm
+from osbuild.util.fscache import FsCache, FsCacheInfo
 from osbuild.util.mnt import mount, umount
 from osbuild.util.types import PathLike
 
@@ -17,27 +18,137 @@ __all__ = [
 ]
 
 
-class Object:
+class PathAdapter:
+    """Expose an object attribute as `os.PathLike`"""
+
+    def __init__(self, obj: Any, attr: str) -> None:
+        self.obj = obj
+        self.attr = attr
+
+    def __fspath__(self):
+        return getattr(self.obj, self.attr)
+
 
+class Object:
     class Mode(enum.Enum):
         READ = 0
         WRITE = 1
 
-    def __init__(self, store: "ObjectStore", uid: str, mode: Mode):
+    class Metadata:
+        """store and retrieve metadata for an object"""
+
+        def __init__(self, base, folder: Optional[str] = None) -> None:
+            self.base = base
+            self.folder = folder
+            os.makedirs(self.path, exist_ok=True)
+
+        def _path_for_key(self, key) -> str:
+            assert key
+            name = f"{key}.json"
+            return os.path.join(self.path, name)
+
+        @property
+        def path(self):
+            if not self.folder:
+                return self.base
+            return os.path.join(self.base, self.folder)
+
+        @contextlib.contextmanager
+        def write(self, key):
+
+            tmp = tempfile.NamedTemporaryFile(
+                mode="w",
+                encoding="utf8",
+                dir=self.path,
+                prefix=".",
+                suffix=".tmp.json",
+                delete=True,
+            )
+
+            with tmp as f:
+                yield f
+
+                f.flush()
+
+                # if nothing was written to the file
+                si = os.stat(tmp.name)
+                if si.st_size == 0:
+                    return
+
+                dest = self._path_for_key(key)
+                # ensure it is proper json?
+                os.link(tmp.name, dest)
+
+        @contextlib.contextmanager
+        def read(self, key):
+            dest = self._path_for_key(key)
+            try:
+                with open(dest, "r", encoding="utf8") as f:
+                    yield f
+            except FileNotFoundError:
+                raise KeyError(f"No metadata for '{key}'") from None
+
+        def set(self, key: str, data):
+
+            if data is None:
+                return
+
+            with self.write(key) as f:
+                json.dump(data, f, indent=2)
+
+        def get(self, key: str):
+            with contextlib.suppress(KeyError):
+                with self.read(key) as f:
+                    return json.load(f)
+            return None
+
+        def __fspath__(self):
+            return self.path
+
+    def __init__(self, cache: FsCache, uid: str, mode: Mode):
+        self._cache = cache
         self._mode = mode
-        self._workdir = None
         self._id = uid
-        self.store = store
+        self._path = None
+        self._meta: Optional[Object.Metadata] = None
+        self._stack: Optional[contextlib.ExitStack] = None
+
+    def _open_for_reading(self):
+        name = self._stack.enter_context(
+            self._cache.load(self.id)
+        )
+        self._path = os.path.join(self._cache, name)
+
+    def _open_for_writing(self):
+        name = self._stack.enter_context(
+            self._cache.stage()
+        )
+        self._path = os.path.join(self._cache, name)
+        os.makedirs(os.path.join(self._path, "tree"))
 
+    def __enter__(self):
+        assert not self.active
+        self._stack = contextlib.ExitStack()
         if self.mode == Object.Mode.READ:
-            path = self.store.resolve_ref(uid)
-            assert path is not None
-            self._path = path
+            self._open_for_reading()
         else:
-            workdir = self.tempdir("workdir")
-            self._path = os.path.join(workdir.name, "object")
-            os.makedirs(self._path)
-            self._workdir = workdir
+            self._open_for_writing()
+
+        # Expose our base path as `os.PathLike` via `PathAdater`
+        # so any changes to it, e.g. via `store_tree`, will be
+        # automatically picked up by `Metadata`.
+        wrapped = PathAdapter(self, "_path")
+        self._meta = self.Metadata(wrapped, folder="meta")
+
+        return self
+
+    def __exit__(self, exc_type, exc_value, exc_tb):
+        assert self.active
+        self.cleanup()
+
+    @property
+    def active(self) -> bool:
+        return self._stack is not None
 
     @property
     def id(self) -> Optional[str]:
@@ -50,32 +161,21 @@ class Object:
     def init(self, base: "Object"):
         """Initialize the object with the base object"""
         self._check_mode(Object.Mode.WRITE)
+        assert self.active
+        assert self._path
         base.clone(self._path)
 
     @property
     def tree(self) -> str:
-        return self._path
-
-    def store_tree(self):
-        """Store the tree with a fresh name and close it
-
-        Moves the tree atomically by using rename(2), to a
-        randomly generated unique name.
-
-        This puts the object into the READ state.
-        """
-        self._check_mode(Object.Mode.WRITE)
-
-        name = str(uuid.uuid4())
-
-        destination = os.path.join(self.store.objects, name)
-        os.rename(self._path, destination)
-        self._path = destination
-
-        self.finalize()
-        self.cleanup()
+        assert self.active
+        assert self._path
+        return os.path.join(self._path, "tree")
 
-        return name
+    @property
+    def meta(self) -> Metadata:
+        assert self.active
+        assert self._meta
+        return self._meta
 
     def finalize(self):
         if self.mode != Object.Mode.WRITE:
@@ -85,27 +185,15 @@ class Object:
         self._mode = Object.Mode.READ
 
     def cleanup(self):
-        workdir = self._workdir
-        if workdir:
-            # manually remove the tree, it might contain
-            # files with immutable flag set, which will
-            # throw off standard Python 3 tempdir cleanup
-            rmrf.rmtree(os.path.join(workdir.name, "object"))
-
-            workdir.cleanup()
-            self._workdir = None
+        if self._stack:
+            self._stack.close()
+            self._stack = None
 
     def _check_mode(self, want: Mode):
         """Internal: Raise a ValueError if we are not in the desired mode"""
         if self.mode != want:
             raise ValueError(f"Wrong object mode: {self.mode}, want {want}")
 
-    def tempdir(self, suffix=None):
-        if suffix:
-            suffix = "-" + suffix
-        name = f"object-{self._id[:7]}-"
-        return self.store.tempdir(prefix=name, suffix=suffix)
-
     def export(self, to_directory: PathLike):
         """Copy object into an external directory"""
         subprocess.run(
@@ -188,16 +276,14 @@ class HostTree:
 
 class ObjectStore(contextlib.AbstractContextManager):
     def __init__(self, store: PathLike):
-        self.store = store
-        self.objects = os.path.join(store, "objects")
-        self.refs = os.path.join(store, "refs")
+        self.cache = FsCache("osbuild", store)
         self.tmp = os.path.join(store, "tmp")
         os.makedirs(self.store, exist_ok=True)
         os.makedirs(self.objects, exist_ok=True)
-        os.makedirs(self.refs, exist_ok=True)
         os.makedirs(self.tmp, exist_ok=True)
         self._objs: Set[Object] = set()
         self._host_tree: Optional[HostTree] = None
+        self._stack = contextlib.ExitStack()
 
     def _get_floating(self, object_id: str) -> Optional[Object]:
         """Internal: get a non-committed object"""
@@ -206,8 +292,33 @@ class ObjectStore(contextlib.AbstractContextManager):
                 return obj
         return None
 
+    @property
+    def maximum_size(self) -> Optional[Union[int, str]]:
+        info = self.cache.info
+        return info.maximum_size
+
+    @maximum_size.setter
+    def maximum_size(self, size: Union[int, str]):
+        info = FsCacheInfo(maximum_size=size)
+        self.cache.info = info
+
+    @property
+    def active(self) -> bool:
+        #pylint: disable=protected-access
+        return self.cache._is_active()
+
+    @property
+    def store(self):
+        return os.fspath(self.cache)
+
+    @property
+    def objects(self):
+        return os.path.join(self.cache, "objects")
+
     @property
     def host_tree(self) -> HostTree:
+        assert self.active
+
         if not self._host_tree:
             self._host_tree = HostTree(self)
         return self._host_tree
@@ -219,13 +330,11 @@ class ObjectStore(contextlib.AbstractContextManager):
         if self._get_floating(object_id):
             return True
 
-        return os.access(self.resolve_ref(object_id), os.F_OK)
-
-    def resolve_ref(self, object_id: Optional[str]) -> Optional[str]:
-        """Returns the path to the given object_id"""
-        if not object_id:
-            return None
-        return os.path.join(self.refs, object_id)
+        try:
+            with self.cache.load(object_id):
+                return True
+        except FsCache.MissError:
+            return False
 
     def tempdir(self, prefix=None, suffix=None):
         """Return a tempfile.TemporaryDirectory within the store"""
@@ -234,75 +343,51 @@ class ObjectStore(contextlib.AbstractContextManager):
                                            suffix=suffix)
 
     def get(self, object_id):
+        assert self.active
+
         obj = self._get_floating(object_id)
         if obj:
             return obj
 
-        if not self.contains(object_id):
+        try:
+            obj = Object(self.cache, object_id, Object.Mode.READ)
+            self._stack.enter_context(obj)
+            return obj
+        except FsCache.MissError:
             return None
 
-        return Object(self, object_id, Object.Mode.READ)
-
     def new(self, object_id: str):
         """Creates a new `Object` and open it for writing.
 
-        It returns a temporary instance of `Object`, the base
-        optionally set to `base_id`. It can be used to interact
-        with the store.
-        If changes to the object's content were made (by calling
-        `Object.write`), these must manually be committed to the
-        store via `commit()`.
+        It returns a instance of `Object` that can be used to
+        write tree and metadata. Use `commit` to attempt to
+        store the object in the cache.
         """
+        assert self.active
 
-        obj = Object(self, object_id, Object.Mode.WRITE)
+        obj = Object(self.cache, object_id, Object.Mode.WRITE)
+        self._stack.enter_context(obj)
 
         self._objs.add(obj)
 
         return obj
 
-    def commit(self, obj: Object, object_id: str) -> str:
-        """Commits a Object to the object store
+    def commit(self, obj: Object, object_id: str):
+        """Commits the Object to the object cache as `object_id`.
 
-        Move the contents of the obj (Object) to object directory
-        of the store with a universally unique name. Creates a
-        symlink to that ('objects/{hash}') in the references
-        directory with the object_id as the name ('refs/{object_id}).
-        If the link already exists, it will be atomically replaced.
-
-        If object_id is different from the id of the object, a copy
-        of the object will be stored.
-
-        Returns: The name of the object
+        Attempts to store the contents of `obj` and its metadata
+        in the object cache. Whether anything is actually stored
+        depends on the configuration of the cache, i.e. its size
+        and how much free space is left or can be made available.
+        Therefore the caller should not assume that the stored
+        object can be retrived at all.
         """
 
-        # The supplied object_id is not the object's final id, so
-        # we have to make a copy first
-        if obj.id != object_id:
-            tmp = self.new(object_id)
-            tmp.init(obj)
-            obj = tmp
-
-        # The object is stored in the objects directory using its unique
-        # name. This means that each commit will always result in a new
-        # object in the store, even if an identical one exists.
-        object_name = obj.store_tree()
-
-        # symlink the object_id (config hash) in the refs directory to the
-        # object name in the objects directory. If a symlink by that name
-        # already exists, atomically replace it, but leave the backing object
-        # in place (it may be in use).
-        with self.tempdir() as tmp:
-            link = f"{tmp}/link"
-            os.symlink(f"../objects/{object_name}", link)
+        assert self.active
 
-            ref = self.resolve_ref(object_id)
-
-            if not ref:
-                raise RuntimeError("commit with unresolvable ref")
-
-            os.replace(link, ref)
-
-        return object_name
+        with self.cache.store(object_id) as name:
+            path = os.path.join(self.cache, name)
+            obj.clone(path)
 
     def cleanup(self):
         """Cleanup all created Objects that are still alive"""
@@ -310,10 +395,19 @@ class ObjectStore(contextlib.AbstractContextManager):
             self._host_tree.cleanup()
             self._host_tree = None
 
-        for obj in self._objs:
-            obj.cleanup()
+        self._stack.close()
+        self._objs = set()
+
+    def __fspath__(self):
+        return os.fspath(self.store)
+
+    def __enter__(self):
+        assert not self.active
+        self._stack.enter_context(self.cache)
+        return self
 
     def __exit__(self, exc_type, exc_val, exc_tb):
+        assert self.active
         self.cleanup()
 
 
diff --git a/osbuild/pipeline.py b/osbuild/pipeline.py
index a0bdf923..5495244d 100644
--- a/osbuild/pipeline.py
+++ b/osbuild/pipeline.py
@@ -43,12 +43,11 @@ def cleanup(*objs):
 
 
 class BuildResult:
-    def __init__(self, origin, returncode, output, metadata, error):
+    def __init__(self, origin, returncode, output, error):
         self.name = origin.name
         self.id = origin.id
         self.success = returncode == 0
         self.output = output
-        self.metadata = metadata
         self.error = error
 
     def as_dict(self):
@@ -186,6 +185,10 @@ class Stage:
                 "mounts": mounts,
             }
 
+            meta = cm.enter_context(
+                tree.meta.write(self.id)
+            )
+
             ro_binds = [
                 f"{self.info.path}:/run/osbuild/bin/{self.name}",
                 f"{inputs_tmpdir}:{inputs_mapped}",
@@ -194,6 +197,7 @@ class Stage:
 
             binds = [
                 os.fspath(tree) + ":/run/osbuild/tree",
+                meta.name + ":/run/osbuild/meta",
                 f"{mounts_tmpdir}:{mounts_mapped}"
             ]
 
@@ -236,7 +240,7 @@ class Stage:
                                readonly_binds=ro_binds,
                                extra_env=extra_env)
 
-        return BuildResult(self, r.returncode, r.output, api.metadata, api.error)
+        return BuildResult(self, r.returncode, r.output, api.error)
 
 
 class Runner:
diff --git a/osbuild/util/parsing.py b/osbuild/util/parsing.py
new file mode 100644
index 00000000..303fbefa
--- /dev/null
+++ b/osbuild/util/parsing.py
@@ -0,0 +1,33 @@
+"""Helpers related to parsing"""
+
+import re
+from typing import Union
+
+
+def parse_size(s: str) -> Union[int, str]:
+    """Parse a size string into a number or 'unlimited'.
+
+    Supported suffixes: kB, kiB, MB, MiB, GB, GiB, TB, TiB
+    """
+    units = [
+        (r'^\s*(\d+)\s*kB$', 1000, 1),
+        (r'^\s*(\d+)\s*KiB$', 1024, 1),
+        (r'^\s*(\d+)\s*MB$', 1000, 2),
+        (r'^\s*(\d+)\s*MiB$', 1024, 2),
+        (r'^\s*(\d+)\s*GB$', 1000, 3),
+        (r'^\s*(\d+)\s*GiB$', 1024, 3),
+        (r'^\s*(\d+)\s*TB$', 1000, 4),
+        (r'^\s*(\d+)\s*TiB$', 1024, 4),
+        (r'^\s*(\d+)$', 1, 1),
+        (r'^unlimited$', "unlimited", 1),
+    ]
+
+    for pat, base, power in units:
+        m = re.fullmatch(pat, s)
+        if m:
+            if isinstance(base, int):
+                return int(m.group(1)) * base ** power
+            if base == "unlimited":
+                return "unlimited"
+
+    raise TypeError(f"invalid size value: '{s}'")


Test patch:
diff --git a/schutzbot/manifest_tests.sh b/schutzbot/manifest_tests.sh
index e7905f8c..f277cbf2 100755
--- a/schutzbot/manifest_tests.sh
+++ b/schutzbot/manifest_tests.sh
@@ -16,6 +16,9 @@ git checkout "$MANIFEST_DB_COMMIT"
 OSBUILD_LABEL=$(matchpathcon -n /usr/bin/osbuild)
 chcon $OSBUILD_LABEL tools/image-info
 
+# set the maximum cache size to unlimited
+echo "{}" | sudo osbuild --cache-max-size unlimited -
+
 # run the tests from the manifest-db for this arch+distro
 echo "Running the osbuild-image-test for arch $ARCH and ditribution $DISTRO_CODE"
 sudo tools/osbuild-image-test --arch=$ARCH --distro=$DISTRO_CODE --image-info-path=tools/image-info
diff --git a/test/mod/test_api.py b/test/mod/test_api.py
index 98afe453..6b029311 100644
--- a/test/mod/test_api.py
+++ b/test/mod/test_api.py
@@ -2,8 +2,10 @@
 # Test for API infrastructure
 #
 
+import json
 import multiprocessing as mp
 import os
+import pathlib
 import tempfile
 import unittest
 
@@ -99,19 +101,14 @@ class TestAPI(unittest.TestCase):
         # Check that `api.metadata` leads to `API.metadata` being
         # set correctly
         tmpdir = self.tmp.name
-        path = os.path.join(tmpdir, "osbuild-api")
+        path = pathlib.Path(tmpdir, "metadata")
+        path.touch()
 
-        def metadata(path):
-            data = {"meta": "42"}
-            osbuild.api.metadata(data, path=path)
-            return 0
+        data = {"meta": "42"}
+        osbuild.api.metadata(data, path=path)
+
+        with open(path, "r", encoding="utf8") as f:
+            metadata = json.load(f)
 
-        api = osbuild.api.API(socket_address=path)
-        with api:
-            p = mp.Process(target=metadata, args=(path, ))
-            p.start()
-            p.join()
-            self.assertEqual(p.exitcode, 0)
-        metadata = api.metadata  # pylint: disable=no-member
         assert metadata
-        self.assertEqual(metadata, {"meta": "42"})
+        self.assertEqual(metadata, data)
diff --git a/test/mod/test_objectstore.py b/test/mod/test_objectstore.py
index 8a2a4fb2..c68e7ce0 100644
--- a/test/mod/test_objectstore.py
+++ b/test/mod/test_objectstore.py
@@ -15,10 +15,8 @@ from .. import test
 
 
 def store_path(store: objectstore.ObjectStore, ref: str, path: str) -> bool:
-    if not store.contains(ref):
-        return False
-    obj = store.resolve_ref(ref)
-    if not obj or not os.path.exists(obj):
+    obj = store.get(ref)
+    if not obj:
         return False
     return os.path.exists(os.path.join(obj, path))
 
@@ -35,8 +33,9 @@ class TestObjectStore(unittest.TestCase):
     def test_basic(self):
         # always use a temporary store so item counting works
         with objectstore.ObjectStore(self.store) as object_store:
+            object_store.maximum_size = 1024*1024*1024
+
             # No objects or references should be in the store
-            assert len(os.listdir(object_store.refs)) == 0
             assert len(os.listdir(object_store.objects)) == 0
 
             tree = object_store.new("a")
@@ -47,157 +46,133 @@ class TestObjectStore(unittest.TestCase):
             p = Path(tree, "A")
             p.touch()
 
-            # consumes the object, puts it into read mode
-            object_store.commit(tree, "a")
-
+            tree.finalize()  # put the object into READ mode
             assert tree.mode == objectstore.Object.Mode.READ
 
-            assert object_store.contains("a")
+            # commit makes a copy, if space
+            object_store.commit(tree, "a")
             assert store_path(object_store, "a", "A")
 
-            assert len(os.listdir(object_store.refs)) == 1
-            assert len(os.listdir(object_store.objects)) == 1
+            # second object, based on the first one
+            obj2 = object_store.new("b")
+            obj2.init(tree)
 
-            tree = object_store.new("b")
-            p = Path(tree, "A")
-            p.touch()
-            p = Path(tree, "B")
+            p = Path(obj2, "B")
             p.touch()
 
-            # consumes the object, puts it into read mode
+            obj2.finalize()  # put the object into READ mode
+            assert obj2.mode == objectstore.Object.Mode.READ
+
+            # commit always makes a copy, if space
             object_store.commit(tree, "b")
 
             assert object_store.contains("b")
+            assert store_path(object_store, "b", "A")
             assert store_path(object_store, "b", "B")
 
-            assert len(os.listdir(object_store.refs)) == 2
             assert len(os.listdir(object_store.objects)) == 2
-            # assert len(os.listdir(f"{object_store.refs}/b/")) == 2
-
-            self.assertEqual(object_store.resolve_ref(None), None)
-            self.assertEqual(object_store.resolve_ref("a"),
-                             f"{object_store.refs}/a")
 
+            # object should exist and should be in read mode
             tree = object_store.get("b")
             assert tree is not None
             assert tree.mode == objectstore.Object.Mode.READ
 
     def test_cleanup(self):
         # always use a temporary store so item counting works
-        with tempfile.TemporaryDirectory(dir="/var/tmp") as tmp:
-            with objectstore.ObjectStore(tmp) as object_store:
-                tree = object_store.new("a")
-                self.assertEqual(len(os.listdir(object_store.tmp)), 1)
-                p = Path(tree, "A")
-                p.touch()
-
-            # there should be no temporary Objects dirs anymore
-            self.assertEqual(len(os.listdir(object_store.tmp)), 0)
-
-    def test_commit_clone(self):
-        # operate with a clean object store
-        with tempfile.TemporaryDirectory(dir="/var/tmp") as tmp:
-            # sample data to be used for read, write checks
-            data = "23"
-
-            with objectstore.ObjectStore(tmp) as store:
-                assert len(os.listdir(store.refs)) == 0
-
-                tree = store.new("a")
-                with open(os.path.join(tree, "data"), "w",
-                          encoding="utf-8") as f:
-                    f.write(data)
-                    st = os.fstat(f.fileno())
-                    data_inode = st.st_ino
-
-                # commit the object as "x", making a copy
-                store.commit(tree, "x")
-
-                # check that "data" got indeed copied
-                tree = store.get("x")
-                assert tree is not None
-
-                with open(os.path.join(tree, "data"), "r",
-                          encoding="utf-8") as f:
-                    st = os.fstat(f.fileno())
-                    self.assertNotEqual(st.st_ino, data_inode)
-                    data_read = f.read()
-                    self.assertEqual(data, data_read)
-
-    def test_commit_consume(self):
-        # operate with a clean object store
-        with tempfile.TemporaryDirectory(dir="/var/tmp") as tmp:
-            # sample data to be used for read, write checks
-            data = "23"
-
-            with objectstore.ObjectStore(tmp) as store:
-                assert len(os.listdir(store.refs)) == 0
-
-                tree = store.new("a")
-                with open(os.path.join(tree, "data"), "w", encoding="utf8") as f:
-                    f.write(data)
-                    st = os.fstat(f.fileno())
-                    data_inode = st.st_ino
-
-                # commit the object as "a"
-                store.commit(tree, "a")
-                assert len(os.listdir(store.refs)) == 1
-
-                # check that "data" is still the very
-                # same file after committing
-                with open(os.path.join(tree, "data"), "r", encoding="utf8") as f:
-                    st = os.fstat(f.fileno())
-                    self.assertEqual(st.st_ino, data_inode)
-                    data_read = f.read()
-                    self.assertEqual(data, data_read)
-
-    def test_object_base(self):
-        with objectstore.ObjectStore(self.store) as store:
-            assert len(os.listdir(store.refs)) == 0
-            assert len(os.listdir(store.objects)) == 0
+        with objectstore.ObjectStore(self.store) as object_store:
+            object_store.maximum_size = 1024*1024*1024
 
-            base = store.new("a")
-            p = Path(base, "A")
+            stage = os.path.join(object_store, "stage")
+            tree = object_store.new("a")
+            self.assertEqual(len(os.listdir(stage)), 1)
+            p = Path(tree, "A")
             p.touch()
-            store.commit(base, "a")
 
-            assert store.contains("a")
-            assert store_path(store, "a", "A")
+        # there should be no temporary Objects dirs anymore
+        with objectstore.ObjectStore(self.store) as object_store:
+            assert object_store.get("A") is None
+
+    def test_metadata(self):
 
-            tree = store.new("b")
-            tree.init(base)
+        # test metadata object directly first
+        with tempfile.TemporaryDirectory() as tmp:
+            md = objectstore.Object.Metadata(tmp)
 
-            p = Path(tree, "B")
-            p.touch()
+            assert os.fspath(md) == tmp
 
-            tree.finalize()
+            with self.assertRaises(KeyError):
+                with md.read("a"):
+                    pass
 
-            assert os.path.exists(os.path.join(tree, "A"))
-            assert os.path.exists(os.path.join(tree, "B"))
+            # do not write anything to the file, it should not get stored
+            with md.write("a"):
+                pass
+
+            assert len(list(os.scandir(tmp))) == 0
+
+            # also we should not write anything if an exception was raised
+            with self.assertRaises(AssertionError):
+                with md.write("a") as f:
+                    f.write("{}")
+                    raise AssertionError
+
+            with md.write("a") as f:
+                f.write("{}")
+
+            assert len(list(os.scandir(tmp))) == 1
+
+            with md.read("a") as f:
+                assert f.read() == "{}"
+
+        data = {
+            "boolean": True,
+            "number": 42,
+            "string": "yes, please"
+        }
+
+        extra = {
+            "extra": "data"
+        }
+
+        with tempfile.TemporaryDirectory() as tmp:
+            md = objectstore.Object.Metadata(tmp)
+
+            d = md.get("a")
+            assert d is None
+
+            md.set("a", None)
+            with self.assertRaises(KeyError):
+                with md.read("a"):
+                    pass
+
+            md.set("a", data)
+            assert md.get("a") == data
 
-    def test_snapshot(self):
         with objectstore.ObjectStore(self.store) as store:
-            tree = store.new("b")
-            p = Path(tree, "A")
+            store.maximum_size = 1024*1024*1024
+            obj = store.new("a")
+            p = Path(obj, "A")
             p.touch()
 
-            assert not store.contains("a")
-            store.commit(tree, "a")  # store via "a", creates a clone
-            assert store.contains("a")
+            obj.meta.set("md", data)
+            assert obj.meta.get("md") == data
 
-            p = Path(tree, "B")
-            p.touch()
-            store.commit(tree, "b")
+            store.commit(obj, "x")
+            obj.meta.set("extra", extra)
+            assert obj.meta.get("extra") == extra
+
+            store.commit(obj, "a")
+
+        with objectstore.ObjectStore(self.store) as store:
+            obj = store.get("a")
+
+            assert obj.meta.get("md") == data
+            assert obj.meta.get("extra") == extra
 
-            # check the references exist
-            assert os.path.exists(f"{store.refs}/a")
-            assert os.path.exists(f"{store.refs}/b")
+            ext = store.get("x")
 
-            # check the contents of the trees
-            assert store_path(store, "a", "A")
-            assert not store_path(store, "a", "B")
-            assert store_path(store, "b", "A")
-            assert store_path(store, "b", "B")
+            assert ext.meta.get("md") == data
+            assert ext.meta.get("extra") is None
 
     def test_host_tree(self):
         with objectstore.ObjectStore(self.store) as store:
@@ -222,7 +197,7 @@ class TestObjectStore(unittest.TestCase):
         with contextlib.ExitStack() as stack:
 
             store = objectstore.ObjectStore(self.store)
-            stack.enter_context(stack)
+            stack.enter_context(store)
 
             tmpdir = tempfile.TemporaryDirectory()
             tmpdir = stack.enter_context(tmpdir)
diff --git a/test/mod/test_osbuild.py b/test/mod/test_osbuild.py
index 89f81cff..e01a2f4c 100644
--- a/test/mod/test_osbuild.py
+++ b/test/mod/test_osbuild.py
@@ -40,16 +40,16 @@ class TestDescriptions(unittest.TestCase):
 
         with tempfile.TemporaryDirectory() as tmpdir:
 
-            data = pathlib.Path(tmpdir, "data")
             storedir = pathlib.Path(tmpdir, "store")
             root = pathlib.Path("/")
             runner = Runner(index.detect_host_runner())
             monitor = NullMonitor(sys.stderr.fileno())
             libdir = os.path.abspath(os.curdir)
             store = ObjectStore(storedir)
-            data.mkdir()
 
-            res = stage.run(data, runner, root, store, monitor, libdir)
+            with ObjectStore(storedir) as store:
+                data = store.new(stage.id)
+                res = stage.run(data, runner, root, store, monitor, libdir)
 
         self.assertEqual(res.success, True)
         self.assertEqual(res.id, stage.id)
diff --git a/test/mod/test_util_parsing.py b/test/mod/test_util_parsing.py
new file mode 100644
index 00000000..faaa31a5
--- /dev/null
+++ b/test/mod/test_util_parsing.py
@@ -0,0 +1,39 @@
+"""Unit tests for osbuild.util.parsing"""
+
+import pytest
+
+from osbuild.util import parsing
+
+
+def test_parse_size():
+    cases = [
+        ("123", True, 123),
+        ("123 kB", True, 123000),
+        ("123 KiB", True, 123 * 1024),
+        ("123 MB", True, 123 * 1000 * 1000),
+        ("123 MiB", True, 123 * 1024 * 1024),
+        ("123 GB", True, 123 * 1000 * 1000 * 1000),
+        ("123 GiB", True, 123 * 1024 * 1024 * 1024),
+        ("123 TB", True, 123 * 1000 * 1000 * 1000 * 1000),
+        ("123 TiB", True, 123 * 1024 * 1024 * 1024 * 1024),
+        ("123kB", True, 123000),
+        ("123KiB", True, 123 * 1024),
+        (" 123", True, 123),
+        ("  123kB", True, 123000),
+        ("  123KiB", True, 123 * 1024),
+        ("unlimited", True, "unlimited"),
+        ("string", False, 0),
+        ("123 KB", False, 0),
+        ("123 mb", False, 0),
+        ("123 PB", False, 0),
+        ("123 PiB", False, 0),
+
+    ]
+
+    for s, success, num in cases:
+        if not success:
+            with pytest.raises(TypeError):
+                parsing.parse_size(s)
+        else:
+            res = parsing.parse_size(s)
+            assert res == num, f"{s} parsed as {res} (wanted {num})"
diff --git a/test/test.py b/test/test.py
index 92664573..98be29b1 100644
--- a/test/test.py
+++ b/test/test.py
@@ -12,6 +12,7 @@ import tempfile
 import unittest
 
 import osbuild.meta
+from osbuild.objectstore import ObjectStore
 from osbuild.util import linux
 
 
@@ -277,6 +278,8 @@ class OSBuild(contextlib.AbstractContextManager):
     _exitstack = None
     _cachedir = None
 
+    maximum_cache_size = 20 * 1024 * 1024 * 1024  # 20 GB
+
     def __init__(self, *, cache_from=None):
         self._cache_from = cache_from
 
@@ -297,6 +300,9 @@ class OSBuild(contextlib.AbstractContextManager):
                     self._cachedir
                 ], check=True)
 
+            with ObjectStore(self._cachedir) as store:
+                store.maximum_size = self.maximum_cache_size
+
             # Keep our ExitStack for `__exit__()`.
             self._exitstack = self._exitstack.pop_all()