Repository: osbuild/osbuild
Pull request #378 (2020-05-06 10:34:44)

Continue Unittest Conversion
This continues the ongoing attempt to integrate all our tests with the `unittest` framework. This series converts `pylint` over, and starts providing accessors to the workdir and test-data paths. If we agree on the model, I will gladly convert the other tests as well.

To recap, the idea is to make all our tests execute through `python -m unittest`. This way, we make sure we can run tests locally as well as in a CI. Furthermore, we will be able to install tests into a system to run them against the installed osbuild package.

This series continues to extend our test infrastructure with accessors to the checkout as well as to test-data. Both paths are currently taken relative to the test sources, but this will not be the case when we run with the test-module installed.

dvdhrm: The Jenkins failure is in:
`TASK [Ensure python requirements are installed]`
...failing due to connection-issues to the fedora mirrors.

I consider this unrelated to this PR.
dvdhrm: > I have some reservations about the magic way the tests figure out if they are installed or not. I would have loved for this to be more explicit (somehow mutate the tests upon installation?). I would also have loved if we could avoid the environment variable to point at the test data, ideally the tests would know they are installed and then look in the canonical place for the data.

Suggestions?
teg: Not sure, to be honest.
dvdhrm: Alternatively, I will just make the functions assume we always have checkout/data access. Once we decide how to install them, we can adjust the functions.

This way, we would keep the status-quo, but avoid hard-coding all the paths everywhere. And we would get proper test-skipping for free, once we adjust this.
teg: That makes sense to me!
dvdhrm: I changed the code to always assume it runs from a checkout. Also rebased on master.

Patch:
diff --git a/Makefile b/Makefile
index 924896f4..792d7f00 100644
--- a/Makefile
+++ b/Makefile
@@ -17,7 +17,6 @@
 BUILDDIR ?= .
 SRCDIR ?= .
 
-PYLINT ?= pylint
 PYTHON3 ?= python3
 RST2MAN ?= rst2man
 
@@ -80,9 +79,9 @@ help:
 	@echo "    man:                Generate all man-pages"
 	@echo
 	@echo "    test-all:           Run all tests"
-	@echo "    test-pylint:        Run pylint on all sources"
 	@echo "    test-module:        Run all module unit-tests"
 	@echo "    test-runtime:       Run all osbuild pipeline tests"
+	@echo "    test-src:           Run all osbuild source tests"
 
 $(BUILDDIR)/:
 	mkdir -p "$@"
@@ -113,7 +112,8 @@ man: $(MANPAGES_TROFF)
 #
 # We use the python `unittest` module for all tests. All the test-sources are
 # located in the `./test/` top-level directory, with `./test/mod/` for module
-# unittests and `./test/run/` for osbuild pipeline runtime tests.
+# unittests, `./test/run/` for osbuild pipeline runtime tests, and `./test/src/`
+# for linters and other tests on the source code.
 #
 
 .PHONY: test-units
@@ -133,13 +133,16 @@ test-runtime:
 			--top-level-directory=$(SRCDIR) \
 			-v
 
-.PHONY: test-pylint
-test-pylint:
-	@find . -type f -name "*.py" | xargs $(PYLINT)
-	@$(PYLINT) runners/* assemblers/* stages/* sources/*
+.PHONY: test-src
+test-src:
+	@$(PYTHON3) -m unittest \
+		discover \
+			--start=$(SRCDIR)/test/src \
+			--top-level-directory=$(SRCDIR) \
+			-v
 
 .PHONY: test
-test-all: test-pylint
+test-all:
 	@$(PYTHON3) -m unittest \
 		discover \
 			--start=$(SRCDIR)/test \
diff --git a/osbuild/buildroot.py b/osbuild/buildroot.py
index fe2f12a1..b99d256c 100644
--- a/osbuild/buildroot.py
+++ b/osbuild/buildroot.py
@@ -73,7 +73,7 @@ class BuildRoot(contextlib.AbstractContextManager):
             shutil.rmtree(self.var)
             self.var = None
 
-    def run(self, argv, binds=None, readonly_binds=None, **kwargs):
+    def run(self, argv, binds=None, readonly_binds=None):
         """Runs a command in the buildroot.
 
         Its arguments mean the same as those for subprocess.run().
@@ -81,9 +81,6 @@ class BuildRoot(contextlib.AbstractContextManager):
 
         nspawn_ro_binds = []
 
-        # pylint suggests to epxlicitly pass `check` to subprocess.run()
-        check = kwargs.pop("check", False)
-
         # make osbuild API-calls accessible to the container
         nspawn_ro_binds.append(f"{self.api}:/run/osbuild/api")
 
@@ -121,7 +118,7 @@ class BuildRoot(contextlib.AbstractContextManager):
             *[f"--bind={b}" for b in (binds or [])],
             *[f"--bind-ro={b}" for b in (readonly_binds or [])],
             f"/run/osbuild/lib/runners/{self.runner}"
-            ] + argv, check=check, **kwargs)
+            ] + argv, check=False, stdin=subprocess.DEVNULL)
 
     def __del__(self):
         self.unmount()
diff --git a/osbuild/meta.py b/osbuild/meta.py
index 584b38ae..2a2f159e 100644
--- a/osbuild/meta.py
+++ b/osbuild/meta.py
@@ -232,7 +232,7 @@ class Schema:
             return res
 
         try:
-            Validator = jsonschema.Draft7Validator
+            Validator = jsonschema.Draft4Validator
             Validator.check_schema(self.data)
             self._validator = Validator(self.data)
         except jsonschema.exceptions.SchemaError as err:
diff --git a/osbuild/objectstore.py b/osbuild/objectstore.py
index 8624a670..6fe58eca 100644
--- a/osbuild/objectstore.py
+++ b/osbuild/objectstore.py
@@ -7,7 +7,7 @@ import tempfile
 import weakref
 from typing import Optional
 
-import osbuild.util.rmrf as rmrf
+from osbuild.util import ctx, rmrf
 from . import treesum
 
 
@@ -16,19 +16,6 @@ __all__ = [
 ]
 
 
-@contextlib.contextmanager
-def suppress_oserror(*errnos):
-    """A context manager that suppresses any OSError with an errno in `errnos`.
-
-    Like contextlib.suppress, but can differentiate between OSErrors.
-    """
-    try:
-        yield
-    except OSError as e:
-        if e.errno not in errnos:
-            raise e
-
-
 def mount(source, target, bind=True, ro=True, private=True, mode="0755"):
     options = []
     if bind:
@@ -145,7 +132,7 @@ class Object:
         self._check_readers()
         self._check_writer()
         self.init()
-        with suppress_oserror(errno.ENOTEMPTY, errno.EEXIST):
+        with ctx.suppress_oserror(errno.ENOTEMPTY, errno.EEXIST):
             os.rename(self._tree, destination)
         self.reset()
 
diff --git a/osbuild/pipeline.py b/osbuild/pipeline.py
index 4473e231..0b8c48b6 100644
--- a/osbuild/pipeline.py
+++ b/osbuild/pipeline.py
@@ -101,7 +101,6 @@ class Stage:
                     [f"/run/osbuild/lib/stages/{self.name}"],
                     binds=[f"{tree}:/run/osbuild/tree"],
                     readonly_binds=ro_binds,
-                    stdin=subprocess.DEVNULL,
                 )
                 return BuildResult(self, r.returncode, api.output)
 
@@ -154,7 +153,6 @@ class Assembler:
                     [f"/run/osbuild/lib/assemblers/{self.name}"],
                     binds=binds,
                     readonly_binds=ro_binds,
-                    stdin=subprocess.DEVNULL,
                 )
                 return BuildResult(self, r.returncode, api.output)
 
diff --git a/osbuild/tmpfs.py b/osbuild/tmpfs.py
deleted file mode 100644
index d55b7f2d..00000000
--- a/osbuild/tmpfs.py
+++ /dev/null
@@ -1,36 +0,0 @@
-
-import os
-import subprocess
-import tempfile
-
-
-__all__ = [
-    "TmpFs",
-]
-
-
-class TmpFs:
-    def __init__(self, path="/run/osbuild"):
-        self.path = path
-        self.root = None
-        self.mounted = False
-
-    def __enter__(self):
-        self.root = tempfile.mkdtemp(prefix="osbuild-tmpfs-", dir=self.path)
-        try:
-            subprocess.run(["mount", "-t", "tmpfs", "-o", "mode=0755", "tmpfs", self.root], check=True)
-            self.mounted = True
-        except subprocess.CalledProcessError:
-            os.rmdir(self.root)
-            self.root = None
-            raise
-        return self.root
-
-    def __exit__(self, exc_type, exc_value, exc_tb):
-        if not self.root:
-            return
-        if self.mounted:
-            subprocess.run(["umount", "--lazy", self.root], check=True)
-            self.mounted = False
-        os.rmdir(self.root)
-        self.root = None
diff --git a/osbuild/util/ctx.py b/osbuild/util/ctx.py
new file mode 100644
index 00000000..68c8ff45
--- /dev/null
+++ b/osbuild/util/ctx.py
@@ -0,0 +1,35 @@
+"""ContextManager Utilities
+
+This module implements helpers around python context-managers, with-statements,
+and RAII. It is meant as a supplement to `contextlib` from the python standard
+library.
+"""
+
+import contextlib
+
+
+__all__ = [
+    "suppress_oserror",
+]
+
+
+@contextlib.contextmanager
+def suppress_oserror(*errnos):
+    """Suppress OSError Exceptions
+
+    This is an extension to `contextlib.suppress()` from the python standard
+    library. It catches any `OSError` exceptions and suppresses them. However,
+    it only catches the exceptions that match the specified error numbers.
+
+    Parameters
+    ----------
+    errnos
+        A list of error numbers to match on. If none are specified, this
+        function has no effect.
+    """
+
+    try:
+        yield
+    except OSError as e:
+        if e.errno not in errnos:
+            raise e
diff --git a/schemas/osbuild1.json b/schemas/osbuild1.json
index 1e333239..76f86992 100644
--- a/schemas/osbuild1.json
+++ b/schemas/osbuild1.json
@@ -1,5 +1,5 @@
 {
-  "$schema": "http://json-schema.org/draft-07/schema#",
+  "$schema": "http://json-schema.org/draft-04/schema#",
   "$id": "https://osbuild.org/schemas/osbuild1.json",
 
   "title": "OSBuild Manifest",
@@ -10,7 +10,6 @@
     "pipeline": { "$ref": "#/definitions/pipeline" },
     "sources": { "$ref": "#/definitions/sources" }
   },
-  "required": [],
 
   "definitions": {
     "assembler": {
@@ -49,8 +48,7 @@
         "assembler": { "$ref": "#/definitions/assembler" },
         "build": { "$ref": "#/definitions/build" },
         "stages": { "$ref": "#/definitions/stages" }
-      },
-      "required": []
+      }
     },
 
     "source": {
diff --git a/schutzbot/Jenkinsfile b/schutzbot/Jenkinsfile
new file mode 100644
index 00000000..68fd19cc
--- /dev/null
+++ b/schutzbot/Jenkinsfile
@@ -0,0 +1,29 @@
+pipeline {
+    agent none
+
+    options {
+        timestamps()
+        ansiColor('xterm')
+    }
+
+    stages {
+        stage("Functional Testing") {
+            parallel {
+                stage('Fedora 31 image') {
+                    agent { label "fedora31" }
+                    environment { TEST_TYPE = "image" }
+                    steps {
+                        sh "schutzbot/run_tests.sh"
+                    }
+                }
+                stage('Fedora 32 image') {
+                    agent { label "fedora32" }
+                    environment { TEST_TYPE = "image" }
+                    steps {
+                        sh "schutzbot/run_tests.sh"
+                    }
+                }
+            }
+        }
+    }
+}


Test patch:
diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml
index b450d58a..0dfe7e3f 100644
--- a/.github/workflows/tests.yml
+++ b/.github/workflows/tests.yml
@@ -4,7 +4,7 @@ on: [pull_request, push]
 
 jobs:
   pylint:
-    name: "pylint"
+    name: "Source Tests"
     runs-on: ubuntu-latest
     container: docker.io/library/python:3.7
     steps:
@@ -12,8 +12,8 @@ jobs:
         run: pip install pylint==2.4.1 jsonschema
       - name: Clone repository
         uses: actions/checkout@v2
-      - name: Run pylint
-        run: make test-pylint
+      - name: Run Source Tests
+        run: make test-src
 
   module:
     name: "Module Unittests"
diff --git a/schutzbot/run_tests.sh b/schutzbot/run_tests.sh
new file mode 100755
index 00000000..c5bc945d
--- /dev/null
+++ b/schutzbot/run_tests.sh
@@ -0,0 +1,67 @@
+#!/bin/bash
+set -euxo pipefail
+
+# Read variables about the OS.
+source /etc/os-release
+
+# Create temporary directories for Ansible.
+sudo mkdir -vp /opt/ansible_{local,remote}
+sudo chmod -R 777 /opt/ansible_{local,remote}
+
+# Remove Fedora modular repositories to speed up dnf-json.
+sudo rm -rfv /etc/yum.repos.d/fedora*modular*
+
+# Ensure /tmp is mounted on tmpfs.
+sudo systemctl enable tmp.mount || \
+  sudo systemctl unmask tmp.mount && sudo systemctl start tmp.mount
+
+# Restart systemd to work around some Fedora issues in cloud images.
+sudo systemctl restart systemd-journald
+
+# Get the current journald cursor.
+export JOURNALD_CURSOR=$(sudo journalctl --quiet -n 1 --show-cursor | tail -n 1 | grep -oP 's\=.*$')
+
+# Add a function to preserve the system journal if something goes wrong.
+preserve_journal() {
+  sudo journalctl --after-cursor=${JOURNALD_CURSOR} > systemd-journald.log
+  exit 1
+}
+trap "preserve_journal" ERR
+
+# Ensure Ansible is installed.
+if ! rpm -q ansible; then
+  sudo dnf -y install ansible
+fi
+
+# Write a simple hosts file for Ansible.
+echo -e "[test_instances]\nlocalhost ansible_connection=local" > hosts.ini
+
+# Set Ansible's config file location.
+export ANSIBLE_CONFIG=ansible-osbuild/ansible.cfg
+
+# Clone the latest version of ansible-osbuild.
+
+# Get the current SHA of osbuild.
+OSBUILD_VERSION=$(git rev-parse HEAD)
+
+# Deploy the software.
+git clone https://github.com/osbuild/ansible-osbuild.git ansible-osbuild
+ansible-playbook \
+  -i hosts.ini \
+  -e osbuild_repo=${WORKSPACE} \
+  -e osbuild_version=$(git rev-parse HEAD) \
+  -e cleanup_composer_directories=yes \
+  ansible-osbuild/playbook.yml
+
+# Run the tests only on Fedora 31 for now.
+if [[ $NAME == "Fedora" ]] && [[ $VERSION_ID == "31" ]]; then
+  ansible-playbook \
+    -e workspace=${WORKSPACE} \
+    -e journald_cursor="${JOURNALD_CURSOR}" \
+    -e test_type=${TEST_TYPE:-image} \
+    -i hosts.ini \
+    ansible-osbuild/repos/osbuild-composer/jenkins/test.yml
+fi
+
+# Collect the systemd journal anyway if we made it all the way to the end.
+sudo journalctl --after-cursor=${JOURNALD_CURSOR} > systemd-journald.log
diff --git a/test/os-release/arch b/test/data/os-release/arch
similarity index 100%
rename from test/os-release/arch
rename to test/data/os-release/arch
diff --git a/test/os-release/fedora30 b/test/data/os-release/fedora30
similarity index 100%
rename from test/os-release/fedora30
rename to test/data/os-release/fedora30
diff --git a/test/os-release/fedora31 b/test/data/os-release/fedora31
similarity index 100%
rename from test/os-release/fedora31
rename to test/data/os-release/fedora31
diff --git a/test/os-release/fedora32 b/test/data/os-release/fedora32
similarity index 100%
rename from test/os-release/fedora32
rename to test/data/os-release/fedora32
diff --git a/test/os-release/fedora33 b/test/data/os-release/fedora33
similarity index 100%
rename from test/os-release/fedora33
rename to test/data/os-release/fedora33
diff --git a/test/os-release/linux b/test/data/os-release/linux
similarity index 100%
rename from test/os-release/linux
rename to test/data/os-release/linux
diff --git a/test/os-release/rhel81 b/test/data/os-release/rhel81
similarity index 100%
rename from test/os-release/rhel81
rename to test/data/os-release/rhel81
diff --git a/test/os-release/rhel82 b/test/data/os-release/rhel82
similarity index 100%
rename from test/os-release/rhel82
rename to test/data/os-release/rhel82
diff --git a/test/os-release/ubuntu1804 b/test/data/os-release/ubuntu1804
similarity index 100%
rename from test/os-release/ubuntu1804
rename to test/data/os-release/ubuntu1804
diff --git a/test/mod/test_util_ctx.py b/test/mod/test_util_ctx.py
new file mode 100644
index 00000000..0bc2e3cc
--- /dev/null
+++ b/test/mod/test_util_ctx.py
@@ -0,0 +1,43 @@
+#
+# Tests for the 'osbuild.util.ctx' module.
+#
+
+import errno
+import unittest
+
+from osbuild.util import ctx
+
+
+class TestUtilCtx(unittest.TestCase):
+    def test_suppress_oserror(self):
+        #
+        # Verify the `suppress_oserror()` function.
+        #
+
+        # Empty list and empty statement is a no-op.
+        with ctx.suppress_oserror():
+            pass
+
+        # Single errno matches raised errno.
+        with ctx.suppress_oserror(errno.EPERM):
+            raise OSError(errno.EPERM, "Operation not permitted")
+
+        # Many errnos match raised errno regardless of their order.
+        with ctx.suppress_oserror(errno.EPERM, errno.ENOENT, errno.ESRCH):
+            raise OSError(errno.EPERM, "Operation not permitted")
+        with ctx.suppress_oserror(errno.ENOENT, errno.EPERM, errno.ESRCH):
+            raise OSError(errno.EPERM, "Operation not permitted")
+        with ctx.suppress_oserror(errno.ENOENT, errno.ESRCH, errno.EPERM):
+            raise OSError(errno.EPERM, "Operation not permitted")
+
+        # Empty list re-raises exceptions.
+        with self.assertRaises(OSError):
+            with ctx.suppress_oserror():
+                raise OSError(errno.EPERM, "Operation not permitted")
+
+        # Non-matching lists re-raise exceptions.
+        with self.assertRaises(OSError):
+            with ctx.suppress_oserror(errno.ENOENT):
+                raise OSError(errno.EPERM, "Operation not permitted")
+            with ctx.suppress_oserror(errno.ENOENT, errno.ESRCH):
+                raise OSError(errno.EPERM, "Operation not permitted")
diff --git a/test/mod/test_util_linux.py b/test/mod/test_util_linux.py
index c10e5639..54e7a926 100644
--- a/test/mod/test_util_linux.py
+++ b/test/mod/test_util_linux.py
@@ -9,18 +9,7 @@ import unittest
 
 from osbuild.util import linux
 
-
-def can_set_immutable():
-    with tempfile.TemporaryDirectory(dir="/var/tmp") as tmp:
-        try:
-            os.makedirs(f"{tmp}/f")
-            # fist they give it ...
-            subprocess.run(["chattr", "+i", f"{tmp}/f"], check=True)
-            # ... then they take it away
-            subprocess.run(["chattr", "-i", f"{tmp}/f"], check=True)
-        except (subprocess.CalledProcessError, FileNotFoundError):
-            return False
-        return True
+from .. import test
 
 
 class TestUtilLinux(unittest.TestCase):
@@ -30,6 +19,7 @@ class TestUtilLinux(unittest.TestCase):
     def tearDown(self):
         self.vartmpdir.cleanup()
 
+    @unittest.skipUnless(test.TestBase.can_modify_immutable("/var/tmp"), "root-only")
     def test_ioctl_get_immutable(self):
         #
         # Test the `ioctl_get_immutable()` helper and make sure it works
@@ -39,7 +29,7 @@ class TestUtilLinux(unittest.TestCase):
         with open(f"{self.vartmpdir.name}/immutable", "x") as f:
             assert not linux.ioctl_get_immutable(f.fileno())
 
-    @unittest.skipUnless(can_set_immutable(), "root-only")
+    @unittest.skipUnless(test.TestBase.can_modify_immutable("/var/tmp"), "root-only")
     def test_ioctl_toggle_immutable(self):
         #
         # Test the `ioctl_toggle_immutable()` helper and make sure it works
diff --git a/test/mod/test_util_osrelease.py b/test/mod/test_util_osrelease.py
index 6f1f387e..756c7965 100644
--- a/test/mod/test_util_osrelease.py
+++ b/test/mod/test_util_osrelease.py
@@ -7,16 +7,19 @@ import unittest
 
 from osbuild.util import osrelease
 
+from .. import test
 
-class TestUtilOSRelease(unittest.TestCase):
+
+class TestUtilOSRelease(test.TestBase, unittest.TestCase):
     def test_non_existant(self):
         """Verify default os-release value, if no files are given."""
         self.assertEqual(osrelease.describe_os(), "linux")
 
+    @unittest.skipUnless(test.TestBase.have_test_data(), "no test-data access")
     def test_describe_os(self):
         """Test host os detection. test/os-release contains the os-release files
         for all supported runners.
         """
-        for entry in os.scandir("test/os-release"):
+        for entry in os.scandir(os.path.join(self.locate_test_data(), "os-release")):
             with self.subTest(entry.name):
                 self.assertEqual(osrelease.describe_os(entry.path), entry.name)
diff --git a/test/mod/test_util_ostree.py b/test/mod/test_util_ostree.py
index 70cad3b4..381ce377 100644
--- a/test/mod/test_util_ostree.py
+++ b/test/mod/test_util_ostree.py
@@ -8,6 +8,8 @@ import subprocess
 
 from osbuild.util import ostree
 
+from .. import test
+
 
 def run(*args, check=True, encoding="utf-8", **kwargs):
     res = subprocess.run(*args,
@@ -17,19 +19,10 @@ def run(*args, check=True, encoding="utf-8", **kwargs):
     return res
 
 
-def have_rpm_ostree():
-    try:
-        r = run(["rpm-ostree", "--version"],
-                capture_output=True, check=False)
-    except FileNotFoundError:
-        return False
-    return r.returncode == 0 and "compose" in r.stdout
-
-
 class TestObjectStore(unittest.TestCase):
 
     # pylint: disable=no-self-use
-    @unittest.skipIf(not have_rpm_ostree(), "rpm-ostree missing")
+    @unittest.skipUnless(test.TestBase.have_rpm_ostree(), "rpm-ostree missing")
     def test_treefile_empty(self):
         # check we produce a valid treefile from an empty object
         tf = ostree.Treefile()
@@ -64,7 +57,7 @@ class TestObjectStore(unittest.TestCase):
                 self.assertEqual(js["ref"], test_ref)
                 self.assertEqual(tf["ref"], test_ref)
 
-    @unittest.skipIf(not have_rpm_ostree(), "rpm-ostree missing")
+    @unittest.skipUnless(test.TestBase.have_rpm_ostree(), "rpm-ostree missing")
     def test_treefile_full(self):
         params = {
             "ref": "osbuild/ostree/devel",
diff --git a/test/mod/test_util_rmrf.py b/test/mod/test_util_rmrf.py
index 24b1b735..e2ed9a85 100644
--- a/test/mod/test_util_rmrf.py
+++ b/test/mod/test_util_rmrf.py
@@ -11,22 +11,11 @@ import unittest
 
 from osbuild.util import rmrf
 
-
-def can_set_immutable():
-    with tempfile.TemporaryDirectory(dir="/var/tmp") as tmp:
-        try:
-            os.makedirs(f"{tmp}/f")
-            # fist they give it ...
-            subprocess.run(["chattr", "+i", f"{tmp}/f"], check=True)
-            # ... then they take it away
-            subprocess.run(["chattr", "-i", f"{tmp}/f"], check=True)
-        except (subprocess.CalledProcessError, FileNotFoundError):
-            return False
-        return True
+from .. import test
 
 
 class TestUtilLinux(unittest.TestCase):
-    @unittest.skipUnless(can_set_immutable(), "Need root permissions")
+    @unittest.skipUnless(test.TestBase.can_modify_immutable("/var/tmp"), "root-only")
     def test_rmtree_immutable(self):
         #
         # Test the `rmrf.rmtree()` helper and verify it can correctly unlink
diff --git a/test/src/__init__.py b/test/src/__init__.py
new file mode 100644
index 00000000..e69de29b
diff --git a/test/src/test_pylint.py b/test/src/test_pylint.py
new file mode 100644
index 00000000..45cbabf0
--- /dev/null
+++ b/test/src/test_pylint.py
@@ -0,0 +1,30 @@
+#
+# Run `pylint` on all python sources.
+#
+
+import subprocess
+import unittest
+
+from .. import test
+
+
+@unittest.skipUnless(test.TestBase.have_test_checkout(), "no test-checkout access")
+class TestPylint(test.TestBase, unittest.TestCase):
+    def test_pylint(self):
+        #
+        # Run `pylint` on all python sources. We simply use `find` to locate
+        # all `*.py` files, and then manually select the reverse-domain named
+        # modules we have.
+        #
+
+        path = self.locate_test_checkout()
+        options = "--errors-only"
+
+        subprocess.run(f"find {path} -type f -name '*.py' | xargs pylint {options}",
+                       shell=True, check=True)
+        subprocess.run(f"pylint {options}"
+                       + f" {path}/assemblers/*"
+                       + f" {path}/runners/*"
+                       + f" {path}/sources/*"
+                       + f" {path}/stages/*",
+                       shell=True, check=True)
diff --git a/test/test.py b/test/test.py
new file mode 100644
index 00000000..000cffa4
--- /dev/null
+++ b/test/test.py
@@ -0,0 +1,133 @@
+#
+# Test Infrastructure
+#
+
+import errno
+import os
+import subprocess
+import tempfile
+
+from osbuild.util import linux
+
+
+class TestBase():
+    """Base Class for Tests
+
+    This class serves as base for our test infrastructure and provides access
+    to common functionality.
+    """
+
+    @staticmethod
+    def have_test_checkout() -> bool:
+        """Check Test-Checkout Access
+
+        Check whether the current test-run has access to a repository checkout
+        of the project and tests. This is usually the guard around code that
+        requires `locate_test_checkout()`.
+
+        For now, we always require tests to be run from a checkout. Hence, this
+        function will always return `True`. This might change in the future,
+        though.
+        """
+
+        # Sanity test to verify we run from within a checkout.
+        assert os.access("setup.py", os.R_OK)
+        return True
+
+    @staticmethod
+    def locate_test_checkout() -> str:
+        """Locate Test-Checkout Path
+
+        This returns the path to the repository checkout we run against. This
+        will fail if `have_test_checkout()` returns false.
+        """
+
+        assert TestBase.have_test_checkout()
+        return os.getcwd()
+
+    @staticmethod
+    def have_test_data() -> bool:
+        """Check Test-Data Access
+
+        Check whether the current test-run has access to the test data. This
+        data is required to run elaborate tests. If it is not available, those
+        tests have to be skipped.
+
+        Test data, unlike test code, is not shipped as part of the `test`
+        python module, hence it needs to be located independently of the code.
+
+        For now, we only support taking test-data from a checkout (see
+        `locate_test_checkout()`). This might be extended in the future, though.
+        """
+
+        return TestBase.have_test_checkout()
+
+    @staticmethod
+    def locate_test_data() -> str:
+        """Locate Test-Data Path
+
+        This returns the path to the test-data directory. This will fail if
+        `have_test_data()` returns false.
+        """
+
+        return os.path.join(TestBase.locate_test_checkout(), "test/data")
+
+    @staticmethod
+    def can_modify_immutable(path: str = "/var/tmp") -> bool:
+        """Check Immutable-Flag Capability
+
+        This checks whether the calling process is allowed to toggle the
+        `FS_IMMUTABLE_FL` file flag. This is limited to `CAP_LINUX_IMMUTABLE`
+        in the initial user-namespace. Therefore, only highly privileged
+        processes can do this.
+
+        There is no reliable way to check whether we can do this. The only
+        possible check is to see whether we can temporarily toggle the flag
+        or not. Since this is highly dependent on the file-system that file
+        is on, you can optionally pass in the path where to test this. Since
+        shmem/tmpfs on linux does not support this, the default is `/var/tmp`.
+        """
+
+        with tempfile.TemporaryFile(dir=path) as f:
+            # First try whether `FS_IOC_GETFLAGS` is actually implemented
+            # for the filesystem we test on. If it is not, lets assume we
+            # cannot modify the flag and make callers skip their tests.
+            try:
+                b = linux.ioctl_get_immutable(f.fileno())
+            except OSError as e:
+                if e.errno in [errno.EACCES, errno.ENOTTY, errno.EPERM]:
+                    return False
+                raise
+
+            # Verify temporary files are not marked immutable by default.
+            assert not b
+
+            # Try toggling the immutable flag. Make sure we always reset it
+            # so the cleanup code can actually drop the temporary object.
+            try:
+                linux.ioctl_toggle_immutable(f.fileno(), True)
+                linux.ioctl_toggle_immutable(f.fileno(), False)
+            except OSError as e:
+                if e.errno in [errno.EACCES, errno.EPERM]:
+                    return False
+                raise
+
+        return True
+
+    @staticmethod
+    def have_rpm_ostree() -> bool:
+        """Check rpm-ostree Availability
+
+        This checks whether `rpm-ostree` is available in the current path and
+        can be called by this process.
+        """
+
+        try:
+            r = subprocess.run(["rpm-ostree", "--version"],
+                               encoding="utf-8",
+                               capture_output=True,
+                               check=False)
+        except FileNotFoundError:
+            return False
+
+        return r.returncode == 0 and "compose" in r.stdout