Repository: osbuild/osbuild
Pull request #219 (2020-01-28 16:41:52)

osbuild: ability to validate stage options
Add a `--validate` option to the osbuild command that will validate the options of the stages, including the build pipeline and the assembler, against the json schema of the corresponding stages.

Will abort if any errors are found and print the corresponding messages, optionally in form of json if `--json` was passed as an command line option.

I was thinking of moving the actual validation code, including the `StageInfo` class to a new `osbuild/meta.py`. Opinions?

bcl: Shouldn't validation be mandatory? Or at least the default? I'm not sure if there are any cases where you wouldn't want to validate the json, wouldn't that just lead to harder to debug problems later in the pipeline execution?

gicmo: > Shouldn't validation be mandatory? Or at least the default? I'm not sure if there are any cases where you wouldn't want to validate the json, wouldn't that just lead to harder to debug problems later in the pipeline execution?

Good question. We could make it mandatory but then we gain a hard dependency on `jsonschema`. Apropos, even though the import is option, pylint is not happy about that.
gicmo: I wan to refactor the code to use the `ast` package to "just" parse the source of the stages, instead of actually instantiating the module.
gicmo: Need to properly honour `--libdir` here as well.
bcl: > I wan to refactor the code to use the `ast` package to "just" parse the source of the stages, instead of actually instantiating the module.

As long as the module import doesn't have any side-effects (which it shouldn't) why go to the extra work?
gicmo: For the record errors look like this:
```
org.osbuild.fstab: 895f90b59c61d8117d38792cd5c28e14eeb1436830b20894f4daff165a13b146
ERROR: '1' is not of type 'number'

Failed validating 'type' in schema['properties']['filesystems']['items']['properties']['freq']:
    {'default': 0,
     'description': 'dump(8) period in days',
     'type': 'number'}

On instance['filesystems'][0]['freq']:
    '1'
```
dvdhrm: The individual stage descriptions seem to be out-of-date. At least some tests are failing due to that. I guess that's a good thing? ;)
gicmo: > Thanks for picking this up!

:heart: I am not 100% done yet: I think, all the stages should have `"additionalProperties": false` as well.

Also, the pipeline itself is not validate. @dvdhrm did the schema for it, but it is outside. What do you guys think of installing the schema as well. In `/usr/share/osbuild/schemas`, although `/usr/lib/osbuild/schemas` would be more convenient from a coding POV because `--libdir` would work for the schema as well then. Once the schemata are installed, a small tool could be crafted to export the schemata from the stages into `schemata/{stages, assemblers}/org.osbuild.<name>` and then that information could be used in the `Index` class (`get_schema(item)`). Which also means that in the future if the stages would be non-python binaries it would still work. Also, the `STAGE_DESC` could be exported to man pages. What do you guys thing?
gicmo: Ok this is ready for review. There a few changes, major ones are:
  - Install the `osbuild1.json`
  - Validate the whole manifest, not only stage/assembler options
  - return value is `2` in case of validation errors
  - JSON validation result is a dict with `{"success": False, "type": "http://<id>", "errors": [...] }` (see below) Format is based on [RFC 7807](https://tools.ietf.org/rfc/rfc7807.txt)
  - New `--inspect` option which returns the description of the pipeline in JSON format, including the ids, but **not** build the pipeline. Will validate the pipeline though and return errors in JSON.
  - Fix up all invalid schemata information and pipelines


Example validation error in JSON:
```json
{
  "type": "https://osbuild.org/validation-error",
  "title": "JSON Schema validation failed",
  "success": false,
  "errors": [
    {
      "message": "Additional properties are not allowed ('foo' was unexpected)",
      "path": [
        "pipeline",
        "build",
        "pipeline"
      ]
    }
  ]
}
```

Example validation error in human readable form:
```
samples/base-qcow2.json has errors:

.pipeline.build.pipeline:
  Additional properties are not allowed ('foo' was unexpected)
```

gicmo: I reworked the `Schema` initialization that should hopefully make it clearer what is going on. Additionally, I added a `Schema.check()` function to verify the schema itself and added tests for it. I also realized that the `definitions` inside stage options need to be located at the root node of the schema otherwise it would be found. Curiously, the schema itself did pass validation without that change, but then would fail during validation.

Patch:
diff --git a/assemblers/org.osbuild.noop b/assemblers/org.osbuild.noop
index 09061305..615ee5f7 100755
--- a/assemblers/org.osbuild.noop
+++ b/assemblers/org.osbuild.noop
@@ -8,7 +8,9 @@ STAGE_INFO = """
 No-op assembler. Produces no output, just prints a JSON dump of its options
 and then exits.
 """
-STAGE_OPTS = ""
+STAGE_OPTS = """
+"additionalProperties": false
+"""
 def main(_tree, _output_dir, options):
     print("Not doing anything with these options:", json.dumps(options))
 
diff --git a/assemblers/org.osbuild.ostree.commit b/assemblers/org.osbuild.ostree.commit
index 48b0dcb7..1c640d6b 100755
--- a/assemblers/org.osbuild.ostree.commit
+++ b/assemblers/org.osbuild.ostree.commit
@@ -22,6 +22,7 @@ the commit id and the compose information respectively.
 [1] https://ostree.readthedocs.io/en/stable/manual/adapting-existing/
 """
 STAGE_OPTS = """
+"additionalProperties": false,
 "required": ["ref"],
 "properties": {
   "ref": {
diff --git a/assemblers/org.osbuild.qemu b/assemblers/org.osbuild.qemu
index d734ac61..41e94dfb 100755
--- a/assemblers/org.osbuild.qemu
+++ b/assemblers/org.osbuild.qemu
@@ -30,6 +30,7 @@ Buildhost commands used: `truncate`, `mount`, `umount`, `sfdisk`,
 `grub2-mkimage`, `mkfs.ext4` or `mkfs.xfs`, `qemu-img`.
 """
 STAGE_OPTS = """
+"additionalProperties": false,
 "required": ["format", "filename", "ptuuid", "size"],
 "oneOf": [{
   "required": ["root_fs_uuid"]
@@ -51,7 +52,7 @@ STAGE_OPTS = """
   "format": {
     "description": "Image file format to use",
     "type": "string",
-    "enum": ["raw", "qcow2", "vdi", "vmdk", "vpc", "vhdx"]
+    "enum": ["raw", "raw.xz", "qcow2", "vdi", "vmdk", "vpc", "vhdx"]
   },
   "filename": {
     "description": "Image filename",
diff --git a/assemblers/org.osbuild.rawfs b/assemblers/org.osbuild.rawfs
index ae07a059..b8dc5b75 100755
--- a/assemblers/org.osbuild.rawfs
+++ b/assemblers/org.osbuild.rawfs
@@ -26,6 +26,7 @@ generate with uuid.uuid4() in Python, `uuidgen(1)` in a shell script, or
 read from `/proc/sys/kernel/random/uuid` if your kernel provides it.
 """
 STAGE_OPTS = """
+"additionalProperties": false,
 "required": ["filename", "root_fs_uuid", "size"],
 "properties": {
   "filename": {
diff --git a/assemblers/org.osbuild.tar b/assemblers/org.osbuild.tar
index 860e7256..17088510 100755
--- a/assemblers/org.osbuild.tar
+++ b/assemblers/org.osbuild.tar
@@ -22,6 +22,7 @@ caller is responsible for making sure that `compression` and `filename` match.
 Buildhost commands used: `tar` and any named `compression` program.
 """
 STAGE_OPTS = """
+"additionalProperties": false,
 "required": ["filename"],
 "properties": {
   "filename": {
diff --git a/assemblers/osbuild b/assemblers/osbuild
deleted file mode 120000
index d41e248d..00000000
--- a/assemblers/osbuild
+++ /dev/null
@@ -1,1 +0,0 @@
-../osbuild
\ No newline at end of file
diff --git a/osbuild.spec b/osbuild.spec
index f4d0dff0..5e1af788 100644
--- a/osbuild.spec
+++ b/osbuild.spec
@@ -1,6 +1,6 @@
 %global         forgeurl https://github.com/osbuild/osbuild
 
-Version:        12
+Version:        13
 
 %forgemeta
 
@@ -75,23 +75,24 @@ make man
 
 mkdir -p %{buildroot}%{pkgdir}/stages
 install -p -m 0755 $(find stages/* -not -path "*/osbuild") %{buildroot}%{pkgdir}/stages/
-ln -s ../osbuild %{buildroot}%{pkgdir}/stages/osbuild
 
 mkdir -p %{buildroot}%{pkgdir}/assemblers
 install -p -m 0755 $(find assemblers/* -not -path "*/osbuild") %{buildroot}%{pkgdir}/assemblers/
-ln -s ../osbuild %{buildroot}%{pkgdir}/assemblers/osbuild
 
 mkdir -p %{buildroot}%{pkgdir}/runners
 install -p -m 0755 $(find runners/* -not -path "*/osbuild") %{buildroot}%{pkgdir}/runners
-ln -s ../osbuild %{buildroot}%{pkgdir}/runners/osbuild
 
 mkdir -p %{buildroot}%{pkgdir}/sources
 install -p -m 0755 $(find sources/* -not -path "*/osbuild") %{buildroot}%{pkgdir}/sources
-ln -s ../osbuild %{buildroot}%{pkgdir}/sources/osbuild
 
 # mount point for bind mounting the osbuild library
 mkdir -p %{buildroot}%{pkgdir}/osbuild
 
+# schemata
+mkdir -p %{buildroot}%{_datadir}/osbuild/schemas
+install -p -m 0755 $(find schemas/*.json) %{buildroot}%{_datadir}/osbuild/schemas
+ln -s %{_datadir}/osbuild/schemas %{buildroot}%{pkgdir}/schemas
+
 # documentation
 mkdir -p %{buildroot}%{_mandir}/man1
 mkdir -p %{buildroot}%{_mandir}/man5
@@ -108,6 +109,7 @@ exit 0
 %{_bindir}/osbuild
 %{_mandir}/man1/%{name}.1*
 %{_mandir}/man5/%{name}-manifest.5*
+%{_datadir}/osbuild/schemas
 %{pkgdir}
 # the following files are in the ostree sub-package
 %exclude %{pkgdir}/assemblers/org.osbuild.ostree.commit
diff --git a/osbuild/buildroot.py b/osbuild/buildroot.py
index 7b80348f..fe2f12a1 100644
--- a/osbuild/buildroot.py
+++ b/osbuild/buildroot.py
@@ -84,13 +84,6 @@ class BuildRoot(contextlib.AbstractContextManager):
         # pylint suggests to epxlicitly pass `check` to subprocess.run()
         check = kwargs.pop("check", False)
 
-        # we need read-write access to loopback devices
-        loopback_allow = "rw"
-        if platform.machine() == "s390x":
-            # on s390x, the bootloader installation program (zipl)
-            # wants to be able create devices nodes, so allow that
-            loopback_allow += "m"
-
         # make osbuild API-calls accessible to the container
         nspawn_ro_binds.append(f"{self.api}:/run/osbuild/api")
 
@@ -122,8 +115,8 @@ class BuildRoot(contextlib.AbstractContextManager):
             "--keep-unit",
             "--as-pid2",
             "--link-journal=no",
-            f"--property=DeviceAllow=block-loop {loopback_allow}",
             f"--directory={self.root}",
+            f"--setenv=PYTHONPATH=/run/osbuild/lib",
             *[f"--bind-ro={b}" for b in nspawn_ro_binds],
             *[f"--bind={b}" for b in (binds or [])],
             *[f"--bind-ro={b}" for b in (readonly_binds or [])],
diff --git a/osbuild/main_cli.py b/osbuild/main_cli.py
index fb27728a..7b65bff2 100644
--- a/osbuild/main_cli.py
+++ b/osbuild/main_cli.py
@@ -10,12 +10,15 @@ import argparse
 import json
 import os
 import sys
+
 import osbuild
+import osbuild.meta
 
 
 RESET = "\033[0m"
 BOLD = "\033[1m"
 RED = "\033[31m"
+GREEN = "\033[32m"
 
 
 def mark_checkpoints(pipeline, checkpoints):
@@ -36,6 +39,7 @@ def mark_checkpoints(pipeline, checkpoints):
     mark_pipeline(pipeline)
     return points
 
+
 def parse_manifest(path):
     if path == "-":
         manifest = json.load(sys.stdin)
@@ -45,6 +49,25 @@ def parse_manifest(path):
 
     return manifest
 
+
+def show_validation(result, name):
+    if name == "-":
+        name = "<stdin>"
+
+    print(f"{BOLD}{name}{RESET} ", end='')
+
+    if result:
+        print(f"is {BOLD}{GREEN}valid{RESET}")
+        return
+
+    print(f"has {BOLD}{RED}errors{RESET}:")
+    print("")
+
+    for error in result:
+        print(f"{BOLD}{error.id}{RESET}:")
+        print(f"  {error.message}\n")
+
+
 def parse_arguments(sys_argv):
     parser = argparse.ArgumentParser(description="Build operating system images")
 
@@ -67,13 +90,28 @@ def parse_arguments(sys_argv):
                         help="output results in JSON format")
     parser.add_argument("--output-directory", metavar="DIRECTORY", type=os.path.abspath,
                         help="directory where result objects are stored")
+    parser.add_argument("--inspect", action="store_true",
+                        help="return the manifest in JSON format including all the ids")
 
     return parser.parse_args(sys_argv[1:])
 
+
+# pylint: disable=too-many-branches
 def osbuild_cli(*, sys_argv=[]):
     args = parse_arguments(sys_argv)
     manifest = parse_manifest(args.manifest_path)
 
+    # first thing after parsing is validation of the input
+    index = osbuild.meta.Index(args.libdir)
+    res = osbuild.meta.validate(manifest, index)
+    if not res:
+        if args.json or args.inspect:
+            json.dump(res.as_dict(), sys.stdout)
+            sys.stdout.write("\n")
+        else:
+            show_validation(res, args.manifest_path)
+        return 2
+
     pipeline = manifest.get("pipeline", {})
     sources_options = manifest.get("sources", {})
 
@@ -101,6 +139,14 @@ def osbuild_cli(*, sys_argv=[]):
             print(f"{RESET}{BOLD}{RED}Failed{RESET}")
             return 1
 
+    if args.inspect:
+        result = {"pipeline": pipeline.description(with_id=True)}
+        if sources_options:
+            result["sources"] = sources_options
+        json.dump(result, sys.stdout)
+        sys.stdout.write("\n")
+        return 0
+
     try:
         r = pipeline.run(
             args.store,
diff --git a/osbuild/meta.py b/osbuild/meta.py
new file mode 100644
index 00000000..584b38ae
--- /dev/null
+++ b/osbuild/meta.py
@@ -0,0 +1,448 @@
+"""Introspection and validation for osbuild
+
+This module contains utilities that help to introspect parts
+that constitute the inner parts of osbuild, i.e. its stages
+and assembler (which is also considered a type of stage in
+this context). Additionally, it provides classes and functions
+to do schema validation of OSBuild manifests and stage options.
+
+A central `Index` class can be used to obtain stage and schema
+information. For the former a `StageInfo` class is returned via
+`Index.get_stage_info`, which contains meta-information about
+the individual stages. Schemata, obtained via `Index.get_schema`
+is represented via a `Schema` class that can in turn be used
+to validate the individual components.
+
+The high level `validate` function can be used to check a given
+manifest (parsed form JSON input in dictionary form) against all
+available schemata. The result is a `ValidationResult` which
+contains a single `ValidationError` for each error detected in
+the manifest. See the individual documentation for details.
+"""
+import ast
+import contextlib
+import copy
+import os
+import json
+from collections import deque
+from typing import Dict, Iterable, Optional
+
+import jsonschema
+
+
+FAILED_TITLE = "JSON Schema validation failed"
+FAILED_TYPEURI = "https://osbuild.org/validation-error"
+
+
+class ValidationError:
+    """Describes a single failed validation
+
+    Consists of a `message` member describing the error
+    that occurred and a `path` that points to the element
+    that caused the error.
+    Implements hashing, equality and less-than and thus
+    can be sorted and used in sets and dictionaries.
+    """
+
+    def __init__(self, message: str):
+        self.message = message
+        self.path = deque()
+
+    @classmethod
+    def from_exception(cls, ex):
+        err = cls(ex.message)
+        err.path = ex.absolute_path
+        return err
+
+    @property
+    def id(self):
+        if not self.path:
+            return "."
+
+        result = ""
+        for p in self.path:
+            if isinstance(p, str):
+                if " " in p:
+                    p = f"'{p}'"
+                result += "." + p
+            elif isinstance(p, int):
+                result += f"[{p}]"
+            else:
+                assert "new type"
+
+        return result
+
+    def as_dict(self):
+        """Serializes this object as a dictionary
+
+        The `path` member will be serialized as a list of
+        components (string or integer) and `message` the
+        human readable message string.
+        """
+        return {
+            "message": self.message,
+            "path": list(self.path)
+        }
+
+    def rebase(self, path: Iterable[str]):
+        """Prepend the `path` to `self.path`"""
+        rev = reversed(path)
+        self.path.extendleft(rev)
+
+    def __hash__(self):
+        return hash((self.id, self.message))
+
+    def __eq__(self, other: "ValidationError"):
+        if not isinstance(other, ValidationError):
+            raise ValueError("Need ValidationError")
+
+        if self.id != other.id:
+            return False
+        return self.message == other.message
+
+    def __lt__(self, other: "ValidationError"):
+        if not isinstance(other, ValidationError):
+            raise ValueError("Need ValidationError")
+
+        return self.id < other.id
+
+    def __str__(self):
+        return f"ValidationError: {self.message} [{self.id}]"
+
+
+class ValidationResult:
+    """Result of a JSON Schema validation"""
+
+    def __init__(self, origin: Optional[str]):
+        self.origin = origin
+        self.errors = set()
+
+    def fail(self, msg: str) -> ValidationError:
+        """Add a new `ValidationError` with `msg` as message"""
+        err = ValidationError(msg)
+        self.errors.add(err)
+        return err
+
+    def add(self, err: ValidationError):
+        """Add a `ValidationError` to the set of errors"""
+        self.errors.add(err)
+        return self
+
+    def merge(self, result: "ValidationResult", *, path=[]):
+        """Merge all errors of `result` into this
+
+        Merge all the errors of in `result` into this,
+        adjusting their the paths be pre-pending the
+        supplied `path`.
+        """
+        for err in result:
+            err = copy.deepcopy(err)
+            err.rebase(path)
+            self.errors.add(err)
+
+    def as_dict(self):
+        """Represent this result as a dictionary
+
+        If there are not errors, returns an empty dict;
+        otherwise it will contain a `type`, `title` and
+        `errors` field. The `title` is a human readable
+        description, the `type` is a URI identifying
+        the validation error type and errors is a list
+        of `ValueErrors`, in turn serialized as dict.
+        Additionally, a `success` member is provided to
+        be compatible with pipeline build results.
+        """
+        errors = [e.as_dict() for e in self]
+        if not errors:
+            return {}
+
+        return {
+            "type": FAILED_TYPEURI,
+            "title": FAILED_TITLE,
+            "success": False,
+            "errors": errors
+        }
+
+    @property
+    def valid(self):
+        """Returns `True` if there are zero errors"""
+        return len(self) == 0
+
+    def __iadd__(self, error: ValidationError):
+        return self.add(error)
+
+    def __bool__(self):
+        return self.valid
+
+    def __len__(self):
+        return len(self.errors)
+
+    def __iter__(self):
+        return iter(sorted(self.errors))
+
+    def __str__(self):
+        return f"ValidationResult: {len(self)} error(s)"
+
+    def __getitem__(self, key):
+        if not isinstance(key, str):
+            raise ValueError("Only string keys allowed")
+
+        lst = list(filter(lambda e: e.id == key, self))
+        if not lst:
+            raise IndexError(f"{key} not found")
+
+        return lst
+
+
+class Schema:
+    """JSON Schema representation
+
+    Class that represents a JSON schema. The `data` attribute
+    contains the actual schema data itself. The `klass` and
+    (optional) `name` refer to entity this schema belongs to.
+    The schema information can be used to validate data via
+    the `validate` method.
+
+    The class can be created with empty schema data. In that
+    case it represents missing schema information. Any call
+    to `validate` will then result in a failure.
+
+    The truth value of this objects corresponds to it having
+    schema data.
+    """
+
+    def __init__(self, schema: str, name: Optional[str] = None):
+        self.data = schema
+        self.name = name
+        self._validator = None
+
+    def check(self) -> ValidationResult:
+        """Validate the `schema` data itself"""
+        res = ValidationResult(self.name)
+
+        # validator is assigned if and only if the schema
+        # itself passes validation (see below). Therefore
+        # this can be taken as an indicator for a valid
+        # schema and thus we can and should short-circuit
+        if self._validator:
+            return res
+
+        if not self.data:
+            res.fail("missing schema information")
+            return res
+
+        try:
+            Validator = jsonschema.Draft7Validator
+            Validator.check_schema(self.data)
+            self._validator = Validator(self.data)
+        except jsonschema.exceptions.SchemaError as err:
+            res += ValidationError.from_exception(err)
+
+        return res
+
+    def validate(self, target) -> ValidationResult:
+        """Validate the `target` against this schema
+
+        If the schema information itself is missing, it
+        will return a `ValidationResult` in failed state,
+        with 'missing schema information' as the reason.
+        """
+        res = self.check()
+        if not res:
+            return res
+
+        for error in self._validator.iter_errors(target):
+            res += ValidationError.from_exception(error)
+
+        return res
+
+    def __bool__(self):
+        return self.check().valid
+
+
+class StageInfo:
+    """Meta information about a stage
+
+    Represents the information about a osbuild pipeline
+    stage or assembler (here also considered to be a stage).
+    Contains the short description (`desc`), a longer
+    description (`info`) and the JSON schema of valid options
+    (`opts`). The `validate` method will check a the options
+    of a stage instance against the JSON schema.
+
+    Normally this class is instantiated via its `load` method.
+    """
+
+    def __init__(self, klass: str, name: str, info: str):
+        self.name = name
+        self.type = klass
+
+        opts = info.get("STAGE_OPTS") or ""
+        self.info = info.get("STAGE_INFO")
+        self.desc = info.get("STAGE_DESC")
+        self.opts = info.get("STAGE_OPTS")
+        self.opts = json.loads("{" + opts + "}")
+
+    @property
+    def schema(self):
+        schema = {
+            "title": f"Pipeline {self.type}",
+            "type": "object",
+            "additionalProperties": False,
+            "properties": {
+                "name": {"type": "string"},
+                "options": {
+                    "type": "object",
+                    **self.opts
+                }
+            },
+            "required": ["name"]
+        }
+
+        # if there are is a definitions node, it needs to be at
+        # the top level schema node, since the schema inside the
+        # stages is written as-if they were the root node and
+        # so are the references
+        definitions = self.opts.get("definitions")
+        if definitions:
+            schema["definitions"] = definitions
+            del schema["properties"]["options"]["definitions"]
+
+        return schema
+
+    @classmethod
+    def load(cls, root, klass, name) -> Optional["StageInfo"]:
+        names = ['STAGE_INFO', 'STAGE_DESC', 'STAGE_OPTS']
+
+        def value(a):
+            v = a.value
+            if isinstance(v, ast.Str):
+                return v.s
+            return ""
+
+        def filter_type(lst, target):
+            return [x for x in lst if isinstance(x, target)]
+
+        def targets(a):
+            return [t.id for t in filter_type(a.targets, ast.Name)]
+
+        mapping = {
+            "Stage": "stages",
+            "Assembler": "assemblers"
+        }
+
+        if klass not in mapping:
+            raise ValueError(f"Unsupported type: {klass}")
+
+        base = mapping[klass]
+
+        path = os.path.join(root, base, name)
+        try:
+            with open(path) as f:
+                data = f.read()
+        except FileNotFoundError:
+            return None
+
+        tree = ast.parse(data, name)
+        assigns = filter_type(tree.body, ast.Assign)
+        targets = [(t, a) for a in assigns for t in targets(a)]
+        info = {k: value(v) for k, v in targets if k in names}
+        return cls(klass, name, info)
+
+
+class Index:
+    """Index of stages and assemblers
+
+    Class that can be used to get the meta information about
+    osbuild stages and assemblers as well as JSON schemata.
+    """
+
+    def __init__(self, path: str):
+        self.path = path
+        self._stage_info = {}
+        self._schemata = {}
+
+    def get_stage_info(self, klass, name) -> Optional[StageInfo]:
+        """Obtain `StageInfo` for a given stage or assembler"""
+
+        if (klass, name) not in self._stage_info:
+
+            info = StageInfo.load(self.path, klass, name)
+            self._stage_info[(klass, name)] = info
+
+        return self._stage_info[(klass, name)]
+
+    def get_schema(self, klass, name=None) -> Schema:
+        """Obtain a `Schema` for `klass` and `name` (optional)
+
+        Returns a `Schema` for the entity identified via `klass`
+        and `name` (if given). Always returns a `Schema` even if
+        no schema information could be found for the entity. In
+        that case the actual schema data for `Schema` will be
+        `None` and any validation will fail.
+        """
+        schema = self._schemata.get((klass, name))
+        if schema is not None:
+            return schema
+
+        if klass == "Manifest":
+            path = f"{self.path}/schemas/osbuild1.json"
+            with contextlib.suppress(FileNotFoundError):
+                with open(path, "r") as f:
+                    schema = json.load(f)
+        elif klass in ["Stage", "Assembler"]:
+            info = self.get_stage_info(klass, name)
+            if info:
+                schema = info.schema
+        else:
+            raise ValueError(f"Unknown klass: {klass}")
+
+        schema = Schema(schema, name or klass)
+        self._schemata[(klass, name)] = schema
+
+        return schema
+
+
+def validate(manifest: Dict, index: Index) -> ValidationResult:
+    """Validate a OSBuild manifest
+
+    This function will validate a OSBuild manifest, including
+    all its stages and assembler and build manifests. It will
+    try to validate as much as possible and not stop on errors.
+    The result is a `ValidationResult` object that can be used
+    to check the overall validation status and iterate all the
+    individual validation errors.
+    """
+
+    schema = index.get_schema("Manifest")
+    result = schema.validate(manifest)
+
+    # main pipeline
+    pipeline = manifest.get("pipeline", {})
+
+    # recursively validate the build pipeline  as a "normal"
+    # pipeline in order to validate its stages and assembler
+    # options; for this it is being re-parented in a new plain
+    # {"pipeline": ...} dictionary. NB: Any nested structural
+    # errors might be detected twice, but de-duplicated by the
+    # `ValidationResult.merge` call
+    build = pipeline.get("build", {}).get("pipeline")
+    if build:
+        res = validate({"pipeline": build}, index=index)
+        result.merge(res, path=["pipeline", "build"])
+
+    stages = pipeline.get("stages", [])
+    for i, stage in enumerate(stages):
+        name = stage["name"]
+        schema = index.get_schema("Stage", name)
+        res = schema.validate(stage)
+        result.merge(res, path=["pipeline", "stages", i])
+
+    asm = pipeline.get("assembler", {})
+    if asm:
+        name = asm["name"]
+        schema = index.get_schema("Assembler", name)
+        res = schema.validate(asm)
+        result.merge(res, path=["pipeline", "assembler"])
+
+    return result
diff --git a/osbuild/pipeline.py b/osbuild/pipeline.py
index 1f844540..94cc30c4 100644
--- a/osbuild/pipeline.py
+++ b/osbuild/pipeline.py
@@ -58,11 +58,12 @@ class Stage:
         m.update(json.dumps(self.options, sort_keys=True).encode())
         return m.hexdigest()
 
-    def description(self):
-        description = {}
-        description["name"] = self.name
+    def description(self, *, with_id=False):
+        description = {"name": self.name}
         if self.options:
             description["options"] = self.options
+        if with_id:
+            description["id"] = self.id
         return description
 
     def run(self,
@@ -121,11 +122,12 @@ class Assembler:
         m.update(json.dumps(self.options, sort_keys=True).encode())
         return m.hexdigest()
 
-    def description(self):
-        description = {}
-        description["name"] = self.name
+    def description(self, *, with_id=False):
+        description = {"name": self.name}
         if self.options:
             description["options"] = self.options
+        if with_id:
+            description["id"] = self.id
         return description
 
     def run(self, tree, runner, build_tree, output_dir=None, interactive=False, libdir=None, var="/var/tmp"):
@@ -190,17 +192,28 @@ class Pipeline:
         pipeline.build = build_pipeline
         pipeline.runner = runner
 
-    def description(self):
+    def description(self, *, with_id=False):
         description = {}
         if self.build:
             description["build"] = {
-                "pipeline": self.build.description(),
+                "pipeline": self.build.description(with_id=with_id),
                 "runner": self.runner
             }
         if self.stages:
-            description["stages"] = [s.description() for s in self.stages]
+            stages = [s.description(with_id=with_id) for s in self.stages]
+            description["stages"] = stages
         if self.assembler:
-            description["assembler"] = self.assembler.description()
+            assembler = self.assembler.description(with_id=with_id)
+            description["assembler"] = assembler
+
+        if with_id:
+            tree_id = self.tree_id
+            if tree_id:
+                description["tree_id"] = tree_id
+            output_id = self.output_id
+            if output_id:
+                description["output_id"] = tree_id
+
         return description
 
     def build_stages(self, object_store, interactive, libdir, secrets):
diff --git a/requirements.txt b/requirements.txt
new file mode 100644
index 00000000..d89304b1
--- /dev/null
+++ b/requirements.txt
@@ -0,0 +1,1 @@
+jsonschema
diff --git a/runners/osbuild b/runners/osbuild
deleted file mode 120000
index d41e248d..00000000
--- a/runners/osbuild
+++ /dev/null
@@ -1,1 +0,0 @@
-../osbuild
\ No newline at end of file
diff --git a/samples/base-qcow2.json b/samples/base-qcow2.json
index f6b5f83c..5599e2f5 100644
--- a/samples/base-qcow2.json
+++ b/samples/base-qcow2.json
@@ -914,8 +914,8 @@
               "uuid": "76a22bf4-f153-4541-b6c7-0332c0dfaeac",
               "vfs_type": "ext4",
               "path": "/",
-              "freq": "1",
-              "passno": "1"
+              "freq": 1,
+              "passno": 1
             }
           ]
         }
diff --git a/samples/f30-aarch64.json b/samples/f30-aarch64.json
index a06d0ad3..b1cdfaf9 100644
--- a/samples/f30-aarch64.json
+++ b/samples/f30-aarch64.json
@@ -337,7 +337,6 @@
     }
   },
   "pipeline": {
-    "runner": "org.osbuild.fedora30",
     "stages": [
       {
         "name": "org.osbuild.rpm",
diff --git a/samples/f30-base-uefi.json b/samples/f30-base-uefi.json
index a5c84d36..280fee67 100644
--- a/samples/f30-base-uefi.json
+++ b/samples/f30-base-uefi.json
@@ -339,7 +339,6 @@
     }
   },
   "pipeline": {
-    "runner": "org.osbuild.fedora30",
     "stages": [
       {
         "name": "org.osbuild.rpm",
diff --git a/samples/f30-bootpart-hybrid.json b/samples/f30-bootpart-hybrid.json
index 3935d72f..a64fcc46 100644
--- a/samples/f30-bootpart-hybrid.json
+++ b/samples/f30-bootpart-hybrid.json
@@ -390,7 +390,6 @@
     }
   },
   "pipeline": {
-    "runner": "org.osbuild.fedora30",
     "stages": [
       {
         "name": "org.osbuild.rpm",
diff --git a/samples/f30-qcow2-hybrid.json b/samples/f30-qcow2-hybrid.json
index cbc1326b..8e9a3148 100644
--- a/samples/f30-qcow2-hybrid.json
+++ b/samples/f30-qcow2-hybrid.json
@@ -390,7 +390,6 @@
     }
   },
   "pipeline": {
-    "runner": "org.osbuild.fedora30",
     "stages": [
       {
         "name": "org.osbuild.rpm",
diff --git a/schemas/osbuild1.json b/schemas/osbuild1.json
index 6ece58fc..1e333239 100644
--- a/schemas/osbuild1.json
+++ b/schemas/osbuild1.json
@@ -1,5 +1,5 @@
 {
-  "$schema": "https://json-schema.org/schema",
+  "$schema": "http://json-schema.org/draft-07/schema#",
   "$id": "https://osbuild.org/schemas/osbuild1.json",
 
   "title": "OSBuild Manifest",
diff --git a/setup.py b/setup.py
index c870903f..0612d119 100644
--- a/setup.py
+++ b/setup.py
@@ -2,10 +2,13 @@ import setuptools
 
 setuptools.setup(
     name="osbuild",
-    version="12",
+    version="13",
     description="A build system for OS images",
     packages=["osbuild", "osbuild.util"],
     license='Apache-2.0',
+    install_requires=[
+        "jsonschema",
+    ],
     entry_points={
         "console_scripts": [
             "osbuild = osbuild.main_cli:main_cli"
diff --git a/sources/osbuild b/sources/osbuild
deleted file mode 120000
index d41e248d..00000000
--- a/sources/osbuild
+++ /dev/null
@@ -1,1 +0,0 @@
-../osbuild
\ No newline at end of file
diff --git a/stages/org.osbuild.chrony b/stages/org.osbuild.chrony
index f065729a..d159966a 100755
--- a/stages/org.osbuild.chrony
+++ b/stages/org.osbuild.chrony
@@ -11,6 +11,8 @@ Modifies /etc/chrony.conf, removing all "server" or "pool" lines and adding
 a "server" line for each server listed in `timeservers`.
 """
 STAGE_OPTS = """
+"additionalProperties": false,
+"required": ["timeservers"],
 "properties": {
   "timeservers": {
     "type": "array",
diff --git a/stages/org.osbuild.debug-shell b/stages/org.osbuild.debug-shell
index f8047fea..56241c0d 100755
--- a/stages/org.osbuild.debug-shell
+++ b/stages/org.osbuild.debug-shell
@@ -12,6 +12,7 @@ which starts an early-boot root shell on the given `tty`.
 Also symlinks the service file into /etc/systemd/system/sysinit.target.wants/.
 """
 STAGE_OPTS = """
+"additionalProperties": false,
 "required": ["tty"],
 "properties": {
   "tty": {
diff --git a/stages/org.osbuild.error b/stages/org.osbuild.error
index a615063c..88041dbb 100755
--- a/stages/org.osbuild.error
+++ b/stages/org.osbuild.error
@@ -9,6 +9,7 @@ Error stage. Return the given error. Useful for testing, debugging, and
 wasting time.
 """
 STAGE_OPTS = """
+"additionalProperties": false,
 "properties": {
   "returncode": {
     "description": "What to return code to use",
diff --git a/stages/org.osbuild.firewall b/stages/org.osbuild.firewall
index 225f2516..d5d156cf 100755
--- a/stages/org.osbuild.firewall
+++ b/stages/org.osbuild.firewall
@@ -29,6 +29,7 @@ target tree, which means it may fail unexpectedly when the buildhost and target
 are different arches or OSes.
 """
 STAGE_OPTS = """
+"additionalProperties": false,
 "properties": {
   "ports": {
     "description": "Ports (or port ranges) to open",
diff --git a/stages/org.osbuild.first-boot b/stages/org.osbuild.first-boot
index f3e2fb90..c071eb23 100755
--- a/stages/org.osbuild.first-boot
+++ b/stages/org.osbuild.first-boot
@@ -19,6 +19,7 @@ If the flag-file cannot be removed, the service fails without executing
 any further first-boot commands.
 """
 STAGE_OPTS = """
+"additionalProperties": false,
 "required": ["commands"],
 "properties": {
   "commands": {
@@ -29,7 +30,7 @@ STAGE_OPTS = """
     }
   },
   "wait_for_network": {
-    "type": "bool",
+    "type": "boolean",
     "description": "Wait for the network to be up before executing",
     "default": false
   }
diff --git a/stages/org.osbuild.fix-bls b/stages/org.osbuild.fix-bls
index 26b8f09c..2233248b 100755
--- a/stages/org.osbuild.fix-bls
+++ b/stages/org.osbuild.fix-bls
@@ -13,9 +13,24 @@ This happens because some boot loader config tools (e.g. grub2-mkrelpath)
 examine /proc/self/mountinfo to find the "real" path to /boot, and find the
 path to the osbuild tree - which won't be valid at boot time for this image.
 
+The paths in the Bootloader Specification are relative to the partition
+they are located on, i.e. `/boot/loader/...` if `/boot` is on the root
+file-system partition. If `/boot` is on a separate partition, the correct
+path would be `/loader/.../` The `prefix` can be used to adjust for that.
+By default it is `/boot`, i.e. assumes `/boot` is on the root file-system.
+
 This stage reads and (re)writes all .conf files in /boot/loader/entries.
 """
-STAGE_OPTS = ""
+STAGE_OPTS = """
+"additionalProperties": false,
+"properties": {
+  "prefix": {
+    "description": "Prefix to use, normally `/boot`",
+    "type": "string",
+    "default": "/boot"
+  }
+}
+"""
 
 
 def main(tree, options):
diff --git a/stages/org.osbuild.fstab b/stages/org.osbuild.fstab
index 50d75e3b..2df985d7 100755
--- a/stages/org.osbuild.fstab
+++ b/stages/org.osbuild.fstab
@@ -13,6 +13,7 @@ a `path` (mount point).
 This stage replaces /etc/fstab, removing any existing entries.
 """
 STAGE_OPTS = """
+"additionalProperties": false,
 "required": ["filesystems"],
 "properties": {
   "filesystems": {
@@ -74,8 +75,8 @@ def main(tree, options):
             label = filesystem.get("label")
             vfs_type = filesystem.get("vfs_type", "none")
             options = filesystem.get("options", "defaults")
-            freq = filesystem.get("freq", "0")
-            passno = filesystem.get("passno", "0")
+            freq = filesystem.get("freq", 0)
+            passno = filesystem.get("passno", 0)
 
             if uuid:
                 fs_spec = f"UUID={uuid}"
diff --git a/stages/org.osbuild.groups b/stages/org.osbuild.groups
index 9be1afd3..dd17170e 100755
--- a/stages/org.osbuild.groups
+++ b/stages/org.osbuild.groups
@@ -13,6 +13,7 @@ If no `gid` is given, `groupadd` will choose one.
 If the specified group name or GID is already in use, this stage will fail.
 """
 STAGE_OPTS = """
+"additionalProperties": false,
 "properties": {
   "groups": {
     "type": "object",
diff --git a/stages/org.osbuild.grub2 b/stages/org.osbuild.grub2
index 6af2c0a9..60594312 100755
--- a/stages/org.osbuild.grub2
+++ b/stages/org.osbuild.grub2
@@ -42,6 +42,7 @@ and accompanying data can be installed from the built root via `uefi.install`.
 Both UEFI and Legacy can be specified at the same time.
 """
 STAGE_OPTS = """
+"additionalProperties": false,
 "oneOf": [{
   "required": ["root_fs_uuid"]
 }, {
diff --git a/stages/org.osbuild.hostname b/stages/org.osbuild.hostname
index d6c9914b..2e169aa4 100755
--- a/stages/org.osbuild.hostname
+++ b/stages/org.osbuild.hostname
@@ -14,6 +14,7 @@ buildhost with `--hostname={hostname}`, which checks the validity of the
 hostname and writes it to /etc/hostname.
 """
 STAGE_OPTS = """
+"additionalProperties": false,
 "required": ["hostname"],
 "properties": {
   "hostname": {
diff --git a/stages/org.osbuild.kernel-cmdline b/stages/org.osbuild.kernel-cmdline
index 5573974d..f1ba73db 100755
--- a/stages/org.osbuild.kernel-cmdline
+++ b/stages/org.osbuild.kernel-cmdline
@@ -11,6 +11,7 @@ command line.
 https://www.kernel.org/doc/html/latest/admin-guide/kernel-parameters.html
 """
 STAGE_OPTS = """
+"additionalProperties": false,
 "properties": {
   "root_fs_uuid": {
     "description": "UUID of the root filesystem image",
diff --git a/stages/org.osbuild.keymap b/stages/org.osbuild.keymap
index 2e31c22c..89f8c1e0 100755
--- a/stages/org.osbuild.keymap
+++ b/stages/org.osbuild.keymap
@@ -15,6 +15,7 @@ Removes any existing /etc/vconsole.conf, then runs `systemd-firstboot` with the
 Valid keymaps are generally found in /lib/kbd/keymaps.
 """
 STAGE_OPTS = """
+"additionalProperties": false,
 "required": ["keymap"],
 "properties": {
   "keymap": {
diff --git a/stages/org.osbuild.locale b/stages/org.osbuild.locale
index 41745c15..d65a2dd8 100755
--- a/stages/org.osbuild.locale
+++ b/stages/org.osbuild.locale
@@ -15,6 +15,7 @@ with the `--locale` flag, which will write a new `/etc/locale.conf` in the
 target system with `LANG={language}`.
 """
 STAGE_OPTS = """
+"additionalProperties": false,
 "required": ["language"],
 "properties": {
   "language": {
diff --git a/stages/org.osbuild.noop b/stages/org.osbuild.noop
index 69c13065..e77c3439 100755
--- a/stages/org.osbuild.noop
+++ b/stages/org.osbuild.noop
@@ -8,7 +8,9 @@ STAGE_INFO = """
 No-op stage. Prints a JSON dump of the options passed into this stage and
 leaves the tree untouched. Useful for testing, debugging, and wasting time.
 """
-STAGE_OPTS = ""
+STAGE_OPTS = """
+"additionalProperties": false
+"""
 
 def main(_tree, options):
     print("Not doing anything with these options:", json.dumps(options))
diff --git a/stages/org.osbuild.ostree b/stages/org.osbuild.ostree
index 9a888ab4..2e886ec4 100755
--- a/stages/org.osbuild.ostree
+++ b/stages/org.osbuild.ostree
@@ -30,8 +30,12 @@ the sysroot and the deployments. Additional kernel options can be passed via
 [1] https://ostree.readthedocs.io/en/latest/manual/deployment/
 """
 STAGE_OPTS = """
-"required": ["rootfs"],
+"required": ["commit", "osname", "rootfs"],
 "properties": {
+  "commit": {
+    "description": "checksum of the OSTree commit",
+    "type": "string"
+  },
   "mounts": {
     "description": "Mount points of the final file system",
     "type": "array",
@@ -52,9 +56,17 @@ STAGE_OPTS = """
       }
     }
   },
+  "osname": {
+    "description": "Name of the stateroot to be used in the deployment",
+    "type": "string"
+  },
   "kernel_opts": {
     "description": "Additional kernel command line options",
-    "type": "string"
+    "type": "array",
+    "items": {
+      "description": "A single kernel command line option",
+      "type": "string"
+    }
   },
   "ref": {
     "description": "OStree ref to create and use for deployment",
diff --git a/stages/org.osbuild.rpm b/stages/org.osbuild.rpm
index f9d27ceb..af6caf6e 100755
--- a/stages/org.osbuild.rpm
+++ b/stages/org.osbuild.rpm
@@ -34,6 +34,7 @@ Uses the following binaries from the host:
     * `rpm` to install packages into the target tree
 """
 STAGE_OPTS = """
+"additionalProperties": false,
 "properties": {
   "gpgkeys": {
     "description": "Array of GPG key contents to import",
diff --git a/stages/org.osbuild.rpm-ostree b/stages/org.osbuild.rpm-ostree
index 6de83435..d2e99429 100755
--- a/stages/org.osbuild.rpm-ostree
+++ b/stages/org.osbuild.rpm-ostree
@@ -38,7 +38,7 @@ human users need to be part of.
 [2] https://rpm-ostree.readthedocs.io/en/latest/manual/treefile/
 """
 STAGE_OPTS = """
-"required": ["enabled_services"],
+"additionalProperties": false,
 "properties": {
   "etc_group_members": {
     "description": "Array of group names to still keep in /etc/group",
diff --git a/stages/org.osbuild.script b/stages/org.osbuild.script
index 062fad03..bbd9f733 100755
--- a/stages/org.osbuild.script
+++ b/stages/org.osbuild.script
@@ -25,6 +25,7 @@ through `/bin/sh` in that case, so it might still work, but that behavior is
 not guaranteed.
 """
 STAGE_OPTS = """
+"additionalProperties": false,
 "required": ["script"],
 "properties": {
   "script": {
diff --git a/stages/org.osbuild.selinux b/stages/org.osbuild.selinux
index ea675409..4cb5c569 100755
--- a/stages/org.osbuild.selinux
+++ b/stages/org.osbuild.selinux
@@ -24,6 +24,7 @@ labels for newly-created files are determined by the host's SELinux policy and
 may not match the tree's policy.
 """
 STAGE_OPTS = """
+"additionalProperties": false,
 "required": ["file_contexts"],
 "properties": {
   "file_contexts": {
diff --git a/stages/org.osbuild.systemd b/stages/org.osbuild.systemd
index 8bdbe6ee..0ddf79c5 100755
--- a/stages/org.osbuild.systemd
+++ b/stages/org.osbuild.systemd
@@ -16,6 +16,7 @@ items, which will delete _all_ symlinks to the named services.
 Uses `systemctl` from the buildhost.
 """
 STAGE_OPTS = """
+"additionalProperties": false,
 "required": ["enabled_services"],
 "properties": {
   "enabled_services": {
diff --git a/stages/org.osbuild.timezone b/stages/org.osbuild.timezone
index c1b92486..4ac01e2c 100755
--- a/stages/org.osbuild.timezone
+++ b/stages/org.osbuild.timezone
@@ -14,6 +14,7 @@ Removes `/etc/localtime`, then runs the host's `systemd-firstboot` binary with
 the `--timezone` option, which will re-create `/etc/localtime`.
 """
 STAGE_OPTS = """
+"additionalProperties": false,
 "required": ["zone"],
 "properties": {
   "zone": {
diff --git a/stages/org.osbuild.users b/stages/org.osbuild.users
index 4c652c72..4c30af4f 100755
--- a/stages/org.osbuild.users
+++ b/stages/org.osbuild.users
@@ -15,6 +15,7 @@ misbehave if the `usermod`/`useradd` binary inside the tree makes incorrect
 assumptions about its host system.
 """
 STAGE_OPTS = """
+"additionalProperties": false,
 "properties": {
   "users": {
     "type": "object",
diff --git a/stages/org.osbuild.zipl b/stages/org.osbuild.zipl
index 03bcbce6..efed206c 100755
--- a/stages/org.osbuild.zipl
+++ b/stages/org.osbuild.zipl
@@ -10,6 +10,13 @@ Configures `zipl` with a minimal config so it can be used in
 the assembler to write the bootmap and bootloader code.
 """
 STAGE_OPTS = """
+"additionalProperties": false,
+"properties": {
+  "timeout": {
+    "description": "Boot loader timeout value",
+    "type": "number"
+  }
+}
 """
 
 
diff --git a/stages/osbuild b/stages/osbuild
deleted file mode 120000
index d41e248d..00000000
--- a/stages/osbuild
+++ /dev/null
@@ -1,1 +0,0 @@
-../osbuild
\ No newline at end of file


Test patch:
diff --git a/.github/workflows/runtime-tests.yml b/.github/workflows/runtime-tests.yml
index f00340e4..5109cb51 100644
--- a/.github/workflows/runtime-tests.yml
+++ b/.github/workflows/runtime-tests.yml
@@ -31,6 +31,8 @@ jobs:
       uses: actions/setup-python@v1
       with:
         python-version: 3.7
+    - name: Install Python Packages
+      run: pip install jsonschema
     - name: "Run Pipeline Tests"
       run: sudo env "PATH=$PATH" make test-runtime
 
@@ -49,6 +51,8 @@ jobs:
       uses: actions/setup-python@v1
       with:
         python-version: 3.7
+    - name: Install Python Packages
+      run: pip install jsonschema
     - name: "Run Noop-Pipeline Tests"
       run: |
         for i in {0..2} ; do
@@ -75,6 +79,8 @@ jobs:
       uses: actions/setup-python@v1
       with:
         python-version: 3.7
+    - name: Install Python Packages
+      run: pip install jsonschema
     - name: "Run Assembler Tests"
       run: sudo env "PATH=$PATH" python3 -m unittest -v test.test_assemblers
 
@@ -95,5 +101,7 @@ jobs:
       uses: actions/setup-python@v1
       with:
         python-version: 3.7
+    - name: Install Python Packages
+      run: pip install jsonschema
     - name: "Run Stage Tests"
       run: sudo env "PATH=$PATH" python3 -m unittest -v test.test_stages
diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml
index 97537a43..b450d58a 100644
--- a/.github/workflows/tests.yml
+++ b/.github/workflows/tests.yml
@@ -9,7 +9,7 @@ jobs:
     container: docker.io/library/python:3.7
     steps:
       - name: Install pylint
-        run: pip install pylint==2.4.1
+        run: pip install pylint==2.4.1 jsonschema
       - name: Clone repository
         uses: actions/checkout@v2
       - name: Run pylint
@@ -67,6 +67,9 @@ jobs:
         with:
           path: osbuild
 
+      - name: Install Dependencies
+        run: pip install jsonschema
+
       - name: Run test_osbuild
         run: |
           cd osbuild
@@ -77,6 +80,28 @@ jobs:
           cd osbuild
           python3 -m unittest -v test.test_objectstore
 
+  sample_validation:
+    name: "sample validation"
+    runs-on: ubuntu-latest
+    container:
+      image: docker.io/library/python:3.7
+    steps:
+      - name: Clone repository
+        uses: actions/checkout@v2
+        with:
+          path: osbuild
+
+      - name: Install Dependencies
+        run: |
+          pip install jsonschema
+
+      - name: Validate the samples
+        run: |
+          cd osbuild
+          for f in samples/*; do
+            python3 -m osbuild --libdir . --inspect "$f" > /dev/null
+          done
+
   rpm_build:
     name: "📦 RPM"
     runs-on: ubuntu-latest
diff --git a/jenkins/run_tests.sh b/jenkins/run_tests.sh
index b72da219..d139f67a 100644
--- a/jenkins/run_tests.sh
+++ b/jenkins/run_tests.sh
@@ -5,14 +5,14 @@ set -euxo pipefail
 source /etc/os-release
 
 # Restart systemd to work around some Fedora issues in cloud images.
-systemctl restart systemd-journald
+sudo systemctl restart systemd-journald
 
 # Get the current journald cursor.
-export JOURNALD_CURSOR=$(journalctl --quiet -n 1 --show-cursor | tail -n 1 | grep -oP 's\=.*$')
+export JOURNALD_CURSOR=$(sudo journalctl --quiet -n 1 --show-cursor | tail -n 1 | grep -oP 's\=.*$')
 
 # Add a function to preserve the system journal if something goes wrong.
 preserve_journal() {
-  journalctl --after-cursor=${JOURNALD_CURSOR} > systemd-journald.log
+  sudo journalctl --after-cursor=${JOURNALD_CURSOR} > systemd-journald.log
   exit 1
 }
 trap "preserve_journal" ERR
@@ -39,6 +39,7 @@ ansible-playbook \
   -i hosts.ini \
   -e osbuild_repo=${WORKSPACE} \
   -e osbuild_version=$(git rev-parse HEAD) \
+  -e cleanup_composer_directories=yes \
   ansible-osbuild/playbook.yml
 
 # Run the tests only on Fedora 31 for now.
@@ -52,4 +53,4 @@ if [[ $NAME == "Fedora" ]] && [[ $VERSION_ID == "31" ]]; then
 fi
 
 # Collect the systemd journal anyway if we made it all the way to the end.
-journalctl --after-cursor=${JOURNALD_CURSOR} > systemd-journald.log
+sudo journalctl --after-cursor=${JOURNALD_CURSOR} > systemd-journald.log
diff --git a/stages/org.osbuild.test b/stages/org.osbuild.test
index 81fce175..7a4a9f88 100755
--- a/stages/org.osbuild.test
+++ b/stages/org.osbuild.test
@@ -13,6 +13,7 @@ Creates `/etc/systemd/system/osbuild-test.service`, and a symlink to it in
 `/etc/systemd/system/multi-user.target.wants/`.
 """
 STAGE_OPTS = """
+"additionalProperties": false,
 "required": ["script"],
 "properties": {
   "script": {
diff --git a/test/pipelines/f30-boot.json b/test/pipelines/f30-boot.json
index b0826eb8..1b276555 100644
--- a/test/pipelines/f30-boot.json
+++ b/test/pipelines/f30-boot.json
@@ -914,8 +914,8 @@
               "uuid": "76a22bf4-f153-4541-b6c7-0332c0dfaeac",
               "vfs_type": "ext4",
               "path": "/",
-              "freq": "1",
-              "passno": "1"
+              "freq": 1,
+              "passno": 1
             }
           ]
         }
diff --git a/test/stages_tests/fstab/b.json b/test/stages_tests/fstab/b.json
index a9b55484..020db6df 100644
--- a/test/stages_tests/fstab/b.json
+++ b/test/stages_tests/fstab/b.json
@@ -828,8 +828,8 @@
               "uuid": "76a22bf4-f153-4541-b6c7-0332c0dfaeac",
               "vfs_type": "ext4",
               "path": "/",
-              "freq": "1",
-              "passno": "1"
+              "freq": 1,
+              "passno": 1
             }
           ]
         }
diff --git a/test/stages_tests/groups/b.json b/test/stages_tests/groups/b.json
index 1311356e..5dd151b2 100644
--- a/test/stages_tests/groups/b.json
+++ b/test/stages_tests/groups/b.json
@@ -825,7 +825,7 @@
         "options": {
           "groups": {
             "testgroup": {
-              "gid": "42042"
+              "gid": 42042
             }
           }
         }
diff --git a/test/test_assemblers.py b/test/test_assemblers.py
index 9e8bc45e..3f6c75d9 100644
--- a/test/test_assemblers.py
+++ b/test/test_assemblers.py
@@ -107,10 +107,9 @@ class TestAssemblers(osbuildtest.TestCase):
             ("tree.tar.gz", "gzip", ["application/x-gzip", "application/gzip"])
         ]
         for filename, compression, expected_mimetypes in cases:
-            options = {
-                "filename": filename,
-                "compression": compression
-            }
+            options = {"filename": filename}
+            if compression:
+                options["compression"] = compression
             _, output_id = self.run_assembler("org.osbuild.tar", options)
             image = f"{self.store}/refs/{output_id}/{filename}"
             output = subprocess.check_output(["file", "--mime-type", image], encoding="utf-8")
diff --git a/test/test_osbuild.py b/test/test_osbuild.py
index b40856ca..f811826e 100644
--- a/test/test_osbuild.py
+++ b/test/test_osbuild.py
@@ -1,9 +1,10 @@
-import importlib.util
 import json
 import os
 import unittest
 
 import osbuild
+import osbuild.meta
+
 
 class TestDescriptions(unittest.TestCase):
     def test_canonical(self):
@@ -77,28 +78,150 @@ class TestDescriptions(unittest.TestCase):
             })
 
     def test_stageinfo(self):
-        def list_stages(base):
-            return [(base, f) for f in os.listdir(base) if f.startswith("org.osbuild")]
-
-        def load_module(base, name):
-            loader = importlib.machinery.SourceFileLoader(name, f"{base}/{name}")
-            spec = importlib.util.spec_from_loader(loader.name, loader)
-            mod = importlib.util.module_from_spec(spec)
-            loader.exec_module(mod)
-            return mod
+        def list_stages(base, klass):
+            return [(klass, f) for f in os.listdir(base) if f.startswith("org.osbuild")]
 
-        stages = list_stages("stages")
-        stages += list_stages("assemblers")
+        stages = list_stages("stages", "Stage")
+        stages += list_stages("assemblers", "Assembler")
 
         for stage in stages:
-            base, name = stage
-            m = load_module(base, name)
+            klass, name = stage
             try:
-                json.loads("{" + m.STAGE_OPTS + "}")
+                info = osbuild.meta.StageInfo.load(os.curdir, klass, name)
+                schema = osbuild.meta.Schema(info.schema, name)
+                res = schema.check()
+                if not res:
+                    err = "\n  ".join(str(e) for e in res)
+                    self.fail(str(res) + "\n  " + err)
             except json.decoder.JSONDecodeError as e:
-                msg = f"Stage '{base}/{name}' has invalid STAGE_OPTS\n\t" + str(e)
+                msg = f"{klass} '{name}' has invalid STAGE_OPTS\n\t" + str(e)
                 self.fail(msg)
 
+    def test_schema(self):
+        schema = osbuild.meta.Schema(None)
+        self.assertFalse(schema)
+
+        schema = osbuild.meta.Schema({"type": "bool"})  # should be 'boolean'
+        self.assertFalse(schema.check().valid)
+        self.assertFalse(schema)
+
+        schema = osbuild.meta.Schema({"type": "array", "minItems": 3})
+        self.assertTrue(schema.check().valid)
+        self.assertTrue(schema)
+
+        res = schema.validate([1, 2])
+        self.assertFalse(res)
+        res = schema.validate([1, 2, 3])
+        self.assertTrue(res)
+
+    def test_validation(self):
+        index = osbuild.meta.Index(os.curdir)
+
+        # an empty manifest is OK
+        res = osbuild.meta.validate({}, index)
+        self.assertEqual(res.valid, True)
+
+        # something totally invalid (by Ondřej Budai)
+        totally_invalid = {
+            "osbuild": {
+                "state": "awesome",
+                "but": {
+                    "input-validation": 1
+                }
+            }
+        }
+
+        res = osbuild.meta.validate(totally_invalid, index)
+        self.assertEqual(res.valid, False)
+        # The top-level 'osbuild' is an additional property
+        self.assertEqual(len(res), 1)
+
+        # This is missing the runner
+        no_runner = {
+            "pipeline": {
+                "build": {
+                    "pipeline": {}
+                }
+            }
+        }
+
+        res = osbuild.meta.validate(no_runner, index)
+        self.assertEqual(res.valid, False)
+        self.assertEqual(len(res), 1)  # missing runner
+        lst = res[".pipeline.build"]
+        self.assertEqual(len(lst), 1)
+
+        # de-dup issues: the manifest checking will report
+        # the extra element and the recursive build pipeline
+        # check will also report that same error; make sure
+        # they get properly de-duplicated
+        no_runner_extra = {
+            "pipeline": {
+                "build": {  # missing runner
+                    "pipeline": {
+                        "extra": True,  # should not be there
+                        "stages": [{
+                            "name": "org.osbuild.chrony",
+                            "options": {
+                                "timeservers": "string"  # should be an array
+                            }
+                        }]
+                    }
+                }
+            }
+        }
+
+        res = osbuild.meta.validate(no_runner_extra, index)
+        self.assertEqual(res.valid, False)
+        self.assertEqual(len(res), 3)
+        lst = res[".pipeline.build.pipeline"]
+        self.assertEqual(len(lst), 1)  # should only have one
+        lst = res[".pipeline.build.pipeline.stages[0].options.timeservers"]
+        self.assertEqual(len(lst), 1)  # should only have one
+
+        # stage issues
+        stage_check = {
+            "pipeline": {
+                "stages": [{
+                    "name": "org.osbuild.grub2",
+                    "options": {
+                        "uefi": {
+                            "install": False,
+                            # missing "vendor"
+                        },
+                        # missing rootfs or root_fs_uuid
+                    }
+                }]
+            }
+        }
+
+        res = osbuild.meta.validate(stage_check, index)
+        self.assertEqual(res.valid, False)
+        self.assertEqual(len(res), 2)
+        lst = res[".pipeline.stages[0].options"]
+        self.assertEqual(len(lst), 1)  #  missing rootfs
+        lst = res[".pipeline.stages[0].options.uefi"]
+        self.assertEqual(len(lst), 1)  #  missing "osname"
+
+        assembler_check = {
+            "pipeline": {
+                "assembler": {
+                    "name": "org.osbuild.tar",
+                    "options": {
+                        "compression": "foobar"
+                    }
+                }
+            }
+        }
+
+        res = osbuild.meta.validate(assembler_check, index)
+        self.assertEqual(res.valid, False)
+        self.assertEqual(len(res), 2)
+        lst = res[".pipeline.assembler.options"]
+        self.assertEqual(len(lst), 1)  #  missing "filename"
+        lst = res[".pipeline.assembler.options.compression"]
+        self.assertEqual(len(lst), 1)  #  wrong compression method
+
 
 if __name__ == "__main__":
     unittest.main()