Repository: python/cpython
Pull request: gh-106368: Increase test coverage for Argument Clinic


<!-- gh-issue-number: gh-106368 -->
* Issue: gh-106368
<!-- /gh-issue-number -->


Problem statement:
Increase Argument Clinic test coverage
The Argument Clinic test suite consists of three parts:

- Lib/test/test_clinic.py: mostly parser tests
- Lib/test/clinic.test: generated code, checked by test_clinic.py (`ClinicExternalTest`)
- Modules/_testclinic.c: mostly converter tests (thanks, @colorfulappl!)

Test coverage is currently just above 80%, which is too low. I'd say we try to reach 90-something percent coverage.

Reminder: PRs for increased test coverage should be backported to all bugfix branches.

<!-- gh-linked-prs -->
### Linked PRs
* gh-106369
* gh-106370
* gh-106373
* gh-106374
* gh-106379
* gh-106381
* gh-106384
* gh-106387
* gh-106388
* gh-106389
* gh-106390
* gh-106391
* gh-106407
* gh-106409
* gh-106410
* gh-106415
* gh-106438
* gh-106439
* gh-106728
* gh-106730
* gh-106731
* gh-106759
* gh-106769
* gh-106770
* gh-106813
* gh-106833
* gh-106838
* gh-106839
* gh-106933
* gh-106943
* gh-106944
* gh-106979
* gh-106994
* gh-107002
* gh-107156
* gh-107189
* gh-107190
* gh-107277
* gh-107282
* gh-107326
* gh-107364
* gh-107365
* gh-107366
* gh-107387
* gh-107496
* gh-107499
* gh-107500
* gh-107514
* gh-107582
* gh-107606
* gh-107611
* gh-107641
* gh-107693
* gh-107731
* gh-107977
* gh-107985
<!-- /gh-linked-prs -->

Hints from issue discussion:
Also: a side effect of good coverage, is that it can help identify unreachable paths in the code.
cc. @sobolevn, who I know enjoys increasing test coverage 📈 
FTR, we're up in 86% coverage now.
> FTR, we're up in 86% coverage now.

A question, where can I find the test coverage? Thank you.
> A question, where can I find the test coverage? Thank you.

There is no published coverage, nor coverage measurement in CI. You'll have to measure it yourself, for now.
FTR, I got my coverage setup up and running again; coverage is at ~87% as of HEAD at 70dc00946938027d5a79bcb7b65038319040144e.

Commit message: gh-106368: Increase test coverage for Argument Clinic

Patch to review:
diff --git a/Lib/test/clinic.test.c b/Lib/test/clinic.test.c
index a660ccf8876e2d..ff384fe775e1b3 100644
--- a/Lib/test/clinic.test.c
+++ b/Lib/test/clinic.test.c
@@ -5004,3 +5004,269 @@ PyDoc_STRVAR(new_dest__doc__,
 "\n"
 "Only this docstring should be outputted to test1.");
 /*[clinic end generated code: output=9cac703f51d90e84 input=090db8df4945576d]*/
+
+
+/*[clinic input]
+mangled_c_keyword_identifier
+    i as int: int
+The 'int' param should be mangled as 'int_value'
+[clinic start generated code]*/
+
+PyDoc_STRVAR(mangled_c_keyword_identifier__doc__,
+"mangled_c_keyword_identifier($module, /, i)\n"
+"--\n"
+"\n"
+"The \'int\' param should be mangled as \'int_value\'");
+
+#define MANGLED_C_KEYWORD_IDENTIFIER_METHODDEF    \
+    {"mangled_c_keyword_identifier", _PyCFunction_CAST(mangled_c_keyword_identifier), METH_FASTCALL|METH_KEYWORDS, mangled_c_keyword_identifier__doc__},
+
+static PyObject *
+mangled_c_keyword_identifier_impl(PyObject *module, int int_value);
+
+static PyObject *
+mangled_c_keyword_identifier(PyObject *module, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames)
+{
+    PyObject *return_value = NULL;
+    #if defined(Py_BUILD_CORE) && !defined(Py_BUILD_CORE_MODULE)
+
+    #define NUM_KEYWORDS 1
+    static struct {
+        PyGC_Head _this_is_not_used;
+        PyObject_VAR_HEAD
+        PyObject *ob_item[NUM_KEYWORDS];
+    } _kwtuple = {
+        .ob_base = PyVarObject_HEAD_INIT(&PyTuple_Type, NUM_KEYWORDS)
+        .ob_item = { &_Py_ID(i), },
+    };
+    #undef NUM_KEYWORDS
+    #define KWTUPLE (&_kwtuple.ob_base.ob_base)
+
+    #else  // !Py_BUILD_CORE
+    #  define KWTUPLE NULL
+    #endif  // !Py_BUILD_CORE
+
+    static const char * const _keywords[] = {"i", NULL};
+    static _PyArg_Parser _parser = {
+        .keywords = _keywords,
+        .fname = "mangled_c_keyword_identifier",
+        .kwtuple = KWTUPLE,
+    };
+    #undef KWTUPLE
+    PyObject *argsbuf[1];
+    int int_value;
+
+    args = _PyArg_UnpackKeywords(args, nargs, NULL, kwnames, &_parser, 1, 1, 0, argsbuf);
+    if (!args) {
+        goto exit;
+    }
+    int_value = _PyLong_AsInt(args[0]);
+    if (int_value == -1 && PyErr_Occurred()) {
+        goto exit;
+    }
+    return_value = mangled_c_keyword_identifier_impl(module, int_value);
+
+exit:
+    return return_value;
+}
+
+static PyObject *
+mangled_c_keyword_identifier_impl(PyObject *module, int int_value)
+/*[clinic end generated code: output=c049d7d79be26cda input=060876448ab567a2]*/
+
+
+/*[clinic input]
+bool_return -> bool
+[clinic start generated code]*/
+
+PyDoc_STRVAR(bool_return__doc__,
+"bool_return($module, /)\n"
+"--\n"
+"\n");
+
+#define BOOL_RETURN_METHODDEF    \
+    {"bool_return", (PyCFunction)bool_return, METH_NOARGS, bool_return__doc__},
+
+static int
+bool_return_impl(PyObject *module);
+
+static PyObject *
+bool_return(PyObject *module, PyObject *Py_UNUSED(ignored))
+{
+    PyObject *return_value = NULL;
+    int _return_value;
+
+    _return_value = bool_return_impl(module);
+    if ((_return_value == -1) && PyErr_Occurred()) {
+        goto exit;
+    }
+    return_value = PyBool_FromLong((long)_return_value);
+
+exit:
+    return return_value;
+}
+
+static int
+bool_return_impl(PyObject *module)
+/*[clinic end generated code: output=3a65f07830e48e98 input=93ba95d39ee98f39]*/
+
+
+/*[clinic input]
+double_return -> double
+[clinic start generated code]*/
+
+PyDoc_STRVAR(double_return__doc__,
+"double_return($module, /)\n"
+"--\n"
+"\n");
+
+#define DOUBLE_RETURN_METHODDEF    \
+    {"double_return", (PyCFunction)double_return, METH_NOARGS, double_return__doc__},
+
+static double
+double_return_impl(PyObject *module);
+
+static PyObject *
+double_return(PyObject *module, PyObject *Py_UNUSED(ignored))
+{
+    PyObject *return_value = NULL;
+    double _return_value;
+
+    _return_value = double_return_impl(module);
+    if ((_return_value == -1.0) && PyErr_Occurred()) {
+        goto exit;
+    }
+    return_value = PyFloat_FromDouble(_return_value);
+
+exit:
+    return return_value;
+}
+
+static double
+double_return_impl(PyObject *module)
+/*[clinic end generated code: output=076dc72595d3f66d input=da11b6255e4cbfd7]*/
+
+
+/*[clinic input]
+Test.__init__
+    a: object
+    [
+    b: object
+    ]
+    /
+Should generate two PyArg_ParseTuple calls.
+
+For some reason, this test fails if the rest of the docstring is removed.
+[clinic start generated output]*/
+[clinic start generated code]*/
+
+PyDoc_STRVAR(Test___init____doc__,
+"Test(a, [b])\n"
+"Should generate two PyArg_ParseTuple calls.\n"
+"\n"
+"For some reason, this test fails if the rest of the docstring is removed.\n"
+"[clinic start generated output]*/");
+
+static int
+Test___init___impl(TestObj *self, PyObject *a, int group_right_1,
+                   PyObject *b);
+
+static int
+Test___init__(PyObject *self, PyObject *args, PyObject *kwargs)
+{
+    int return_value = -1;
+    PyTypeObject *base_tp = TestType;
+    PyObject *a;
+    int group_right_1 = 0;
+    PyObject *b = NULL;
+
+    if ((Py_IS_TYPE(self, base_tp) ||
+         Py_TYPE(self)->tp_new == base_tp->tp_new) &&
+        !_PyArg_NoKeywords("Test", kwargs)) {
+        goto exit;
+    }
+    switch (PyTuple_GET_SIZE(args)) {
+        case 1:
+            if (!PyArg_ParseTuple(args, "O:__init__", &a)) {
+                goto exit;
+            }
+            break;
+        case 2:
+            if (!PyArg_ParseTuple(args, "OO:__init__", &a, &b)) {
+                goto exit;
+            }
+            group_right_1 = 1;
+            break;
+        default:
+            PyErr_SetString(PyExc_TypeError, "Test.__init__ requires 1 to 2 arguments");
+            goto exit;
+    }
+    return_value = Test___init___impl((TestObj *)self, a, group_right_1, b);
+
+exit:
+    return return_value;
+}
+
+static int
+Test___init___impl(TestObj *self, PyObject *a, int group_right_1,
+                   PyObject *b)
+/*[clinic end generated code: output=a1697d0b3e523310 input=56ee43c117773c85]*/
+
+
+/*[clinic input]
+Test._pyarg_parsestackandkeywords
+    cls: defining_class
+    key: str(accept={str, robuffer}, zeroes=True)
+    /
+Check that _PyArg_ParseStackAndKeywords() is generated.
+[clinic start generated code]*/
+
+PyDoc_STRVAR(Test__pyarg_parsestackandkeywords__doc__,
+"_pyarg_parsestackandkeywords($self, key, /)\n"
+"--\n"
+"\n"
+"Check that _PyArg_ParseStackAndKeywords() is generated.");
+
+#define TEST__PYARG_PARSESTACKANDKEYWORDS_METHODDEF    \
+    {"_pyarg_parsestackandkeywords", _PyCFunction_CAST(Test__pyarg_parsestackandkeywords), METH_METHOD|METH_FASTCALL|METH_KEYWORDS, Test__pyarg_parsestackandkeywords__doc__},
+
+static PyObject *
+Test__pyarg_parsestackandkeywords_impl(TestObj *self, PyTypeObject *cls,
+                                       const char *key,
+                                       Py_ssize_t key_length);
+
+static PyObject *
+Test__pyarg_parsestackandkeywords(TestObj *self, PyTypeObject *cls, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames)
+{
+    PyObject *return_value = NULL;
+    #if defined(Py_BUILD_CORE) && !defined(Py_BUILD_CORE_MODULE)
+    #  define KWTUPLE (PyObject *)&_Py_SINGLETON(tuple_empty)
+    #else
+    #  define KWTUPLE NULL
+    #endif
+
+    static const char * const _keywords[] = {"", NULL};
+    static _PyArg_Parser _parser = {
+        .keywords = _keywords,
+        .format = "s#:_pyarg_parsestackandkeywords",
+        .kwtuple = KWTUPLE,
+    };
+    #undef KWTUPLE
+    const char *key;
+    Py_ssize_t key_length;
+
+    if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser,
+        &key, &key_length)) {
+        goto exit;
+    }
+    return_value = Test__pyarg_parsestackandkeywords_impl(self, cls, key, key_length);
+
+exit:
+    return return_value;
+}
+
+static PyObject *
+Test__pyarg_parsestackandkeywords_impl(TestObj *self, PyTypeObject *cls,
+                                       const char *key,
+                                       Py_ssize_t key_length)
+/*[clinic end generated code: output=4fda8a7f2547137c input=fc72ef4b4cfafabc]*/
diff --git a/Lib/test/test_clinic.py b/Lib/test/test_clinic.py
index 2f94f0168c9166..b733730eff7c9d 100644
--- a/Lib/test/test_clinic.py
+++ b/Lib/test/test_clinic.py
@@ -9,6 +9,7 @@
 from textwrap import dedent
 from unittest import TestCase
 import collections
+import contextlib
 import inspect
 import os.path
 import subprocess
@@ -338,6 +339,76 @@ def converter_init(self):
         )
         self.assertIn(msg, out)
 
+    @staticmethod
+    @contextlib.contextmanager
+    def _clinic_version(new_version):
+        """Helper for test_version_*() tests"""
+        _saved = clinic.version
+        clinic.version = new_version
+        try:
+            yield
+        finally:
+            clinic.version = _saved
+
+    def test_version_directive_lower(self):
+        with self._clinic_version('3'):
+            block = dedent("""
+                /*[clinic input]
+                version 2
+                [clinic start generated code]*/
+            """)
+            self.clinic.parse(block)
+
+    def test_version_directive_equal(self):
+        with self._clinic_version('5'):
+            block = dedent("""
+                /*[clinic input]
+                version 5
+                [clinic start generated code]*/
+            """)
+            self.clinic.parse(block)
+
+    def test_version_directive_with_modifier(self):
+        with self._clinic_version('1.2b0'):
+            block = dedent("""
+                /*[clinic input]
+                version 1.2a7
+                [clinic start generated code]*/
+            """)
+            self.clinic.parse(block)
+
+    def test_version_directive_insufficient_version(self):
+        with self._clinic_version('4'):
+            err = (
+                "Insufficient Clinic version!\n"
+                "  Version: 4\n"
+                "  Required: 5"
+            )
+            out = self.expect_failure("""
+                /*[clinic input]
+                version 5
+                [clinic start generated code]*/
+            """)
+            self.assertIn(err, out)
+
+    def test_version_directive_illegal_char(self):
+        err = "Illegal character 'v' in version string 'v5'"
+        out = self.expect_failure("""
+            /*[clinic input]
+            version v5
+            [clinic start generated code]*/
+        """)
+        self.assertIn(err, out)
+
+    def test_version_directive_unsupported_string(self):
+        err = "Unsupported version string: '.-'"
+        out = self.expect_failure("""
+            /*[clinic input]
+            version .-
+            [clinic start generated code]*/
+        """)
+        self.assertIn(err, out)
+
 
 class ClinicGroupPermuterTest(TestCase):
     def _test(self, l, m, r, output):
@@ -516,6 +587,22 @@ def test_clinic_1(self):
 
 
 class ClinicParserTest(_ParserBase):
+
+    def parse(self, text):
+        c = FakeClinic()
+        parser = DSLParser(c)
+        block = clinic.Block(text)
+        parser.parse(block)
+        return block
+
+    def parse_function(self, text, signatures_in_block=2, function_index=1):
+        block = self.parse(text)
+        s = block.signatures
+        self.assertEqual(len(s), signatures_in_block)
+        assert isinstance(s[0], clinic.Module)
+        assert isinstance(s[function_index], clinic.Function)
+        return s[function_index]
+
     def checkDocstring(self, fn, expected):
         self.assertTrue(hasattr(fn, "docstring"))
         self.assertEqual(fn.docstring.strip(),
@@ -1376,21 +1463,6 @@ def test_unused_param(self):
                 parser_decl = p.simple_declaration(in_parser=True)
                 self.assertNotIn("Py_UNUSED", parser_decl)
 
-    def parse(self, text):
-        c = FakeClinic()
-        parser = DSLParser(c)
-        block = clinic.Block(text)
-        parser.parse(block)
-        return block
-
-    def parse_function(self, text, signatures_in_block=2, function_index=1):
-        block = self.parse(text)
-        s = block.signatures
-        self.assertEqual(len(s), signatures_in_block)
-        assert isinstance(s[0], clinic.Module)
-        assert isinstance(s[function_index], clinic.Function)
-        return s[function_index]
-
     def test_scaffolding(self):
         # test repr on special values
         self.assertEqual(repr(clinic.unspecified), '<Unspecified>')
@@ -1408,6 +1480,26 @@ def test_scaffolding(self):
         actual = stdout.getvalue()
         self.assertEqual(actual, expected)
 
+    def test_non_ascii_character_in_docstring(self):
+        block = """
+            module test
+            test.fn
+            docstring fü bár baß
+        """
+        with support.captured_stdout() as stdout:
+            self.parse(block)
+        msg = "Non-ascii character appear in docstring"
+        self.assertIn(msg, stdout.getvalue())
+
+    def test_illegal_c_identifier(self):
+        err = "Illegal C identifier: 17a"
+        out = self.parse_function_should_fail("""
+            module test
+            test.fn
+                a as 17a: int
+        """)
+        self.assertIn(err, out)
+
 
 class ClinicExternalTest(TestCase):
     maxDiff = None


Code review comments:
diff_hunk: @@ -1408,6 +1480,26 @@ def test_scaffolding(self):
         actual = stdout.getvalue()
         self.assertEqual(actual, expected)
 
+    def test_non_ascii_character_in_docstring(self):
+        block = """
+            module test
+            test.fn
+            docstring fü bár baß
+        """
+        with support.captured_stdout() as stdout:
+            self.parse(block)
+        msg = "Non-ascii character appear in docstring"
path: Lib/test/test_clinic.py
text: @user1:
The error message here doesn't make grammatical sense!

https://github.com/python/cpython/blob/dfb55d9d7f9349582a2077f87e95c8f9d5e2ebd4/Tools/clinic/clinic.py#L788

We should fix that rather than asserting incorrect behaviour in a test :)

@author:
Tomorrow! :)

@author:
How about _"Non-ascii characters are not allowed in docstrings: {the offending char!r}"_

@user1:
> How about _"Non-ascii characters are not allowed in docstrings: {the offending char!r}"_

Sounds good!

@author:
Addressed in gh-107559 (issue) and gh-107560 (PR).

diff_hunk: @@ -5004,3 +5004,269 @@ PyDoc_STRVAR(new_dest__doc__,
 "\n"
 "Only this docstring should be outputted to test1.");
 /*[clinic end generated code: output=9cac703f51d90e84 input=090db8df4945576d]*/
+
+
+/*[clinic input]
+mangled_c_keyword_identifier
+    i as int: int
+The 'int' param should be mangled as 'int_value'
+[clinic start generated code]*/
+
+PyDoc_STRVAR(mangled_c_keyword_identifier__doc__,
+"mangled_c_keyword_identifier($module, /, i)\n"
+"--\n"
+"\n"
+"The \'int\' param should be mangled as \'int_value\'");
+
+#define MANGLED_C_KEYWORD_IDENTIFIER_METHODDEF    \
+    {"mangled_c_keyword_identifier", _PyCFunction_CAST(mangled_c_keyword_identifier), METH_FASTCALL|METH_KEYWORDS, mangled_c_keyword_identifier__doc__},
+
+static PyObject *
+mangled_c_keyword_identifier_impl(PyObject *module, int int_value);
+
+static PyObject *
+mangled_c_keyword_identifier(PyObject *module, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames)
+{
+    PyObject *return_value = NULL;
+    #if defined(Py_BUILD_CORE) && !defined(Py_BUILD_CORE_MODULE)
+
+    #define NUM_KEYWORDS 1
+    static struct {
+        PyGC_Head _this_is_not_used;
+        PyObject_VAR_HEAD
+        PyObject *ob_item[NUM_KEYWORDS];
+    } _kwtuple = {
+        .ob_base = PyVarObject_HEAD_INIT(&PyTuple_Type, NUM_KEYWORDS)
+        .ob_item = { &_Py_ID(i), },
+    };
+    #undef NUM_KEYWORDS
+    #define KWTUPLE (&_kwtuple.ob_base.ob_base)
+
+    #else  // !Py_BUILD_CORE
+    #  define KWTUPLE NULL
+    #endif  // !Py_BUILD_CORE
+
+    static const char * const _keywords[] = {"i", NULL};
+    static _PyArg_Parser _parser = {
+        .keywords = _keywords,
+        .fname = "mangled_c_keyword_identifier",
+        .kwtuple = KWTUPLE,
+    };
+    #undef KWTUPLE
+    PyObject *argsbuf[1];
+    int int_value;
+
+    args = _PyArg_UnpackKeywords(args, nargs, NULL, kwnames, &_parser, 1, 1, 0, argsbuf);
+    if (!args) {
+        goto exit;
+    }
+    int_value = _PyLong_AsInt(args[0]);
+    if (int_value == -1 && PyErr_Occurred()) {
+        goto exit;
+    }
+    return_value = mangled_c_keyword_identifier_impl(module, int_value);
+
+exit:
+    return return_value;
+}
+
+static PyObject *
+mangled_c_keyword_identifier_impl(PyObject *module, int int_value)
+/*[clinic end generated code: output=c049d7d79be26cda input=060876448ab567a2]*/
+
+
+/*[clinic input]
+bool_return -> bool
+[clinic start generated code]*/
+
+PyDoc_STRVAR(bool_return__doc__,
+"bool_return($module, /)\n"
+"--\n"
+"\n");
+
+#define BOOL_RETURN_METHODDEF    \
+    {"bool_return", (PyCFunction)bool_return, METH_NOARGS, bool_return__doc__},
+
+static int
+bool_return_impl(PyObject *module);
+
+static PyObject *
+bool_return(PyObject *module, PyObject *Py_UNUSED(ignored))
+{
+    PyObject *return_value = NULL;
+    int _return_value;
+
+    _return_value = bool_return_impl(module);
+    if ((_return_value == -1) && PyErr_Occurred()) {
+        goto exit;
+    }
+    return_value = PyBool_FromLong((long)_return_value);
+
+exit:
+    return return_value;
+}
+
+static int
+bool_return_impl(PyObject *module)
+/*[clinic end generated code: output=3a65f07830e48e98 input=93ba95d39ee98f39]*/
+
+
+/*[clinic input]
+double_return -> double
+[clinic start generated code]*/
+
+PyDoc_STRVAR(double_return__doc__,
+"double_return($module, /)\n"
+"--\n"
+"\n");
+
+#define DOUBLE_RETURN_METHODDEF    \
+    {"double_return", (PyCFunction)double_return, METH_NOARGS, double_return__doc__},
+
+static double
+double_return_impl(PyObject *module);
+
+static PyObject *
+double_return(PyObject *module, PyObject *Py_UNUSED(ignored))
+{
+    PyObject *return_value = NULL;
+    double _return_value;
+
+    _return_value = double_return_impl(module);
+    if ((_return_value == -1.0) && PyErr_Occurred()) {
+        goto exit;
+    }
+    return_value = PyFloat_FromDouble(_return_value);
+
+exit:
+    return return_value;
+}
+
+static double
+double_return_impl(PyObject *module)
+/*[clinic end generated code: output=076dc72595d3f66d input=da11b6255e4cbfd7]*/
+
+
+/*[clinic input]
+Test.__init__
+    a: object
+    [
+    b: object
+    ]
+    /
+Should generate two PyArg_ParseTuple calls.
+
+For some reason, this test fails if the rest of the docstring is removed.
path: Lib/test/clinic.test.c
text: @user1:
Looks like a `TODO` item to me :)

@author:
Yes; I'll see if I can fix it first.

@author:
PEBKAC: 7c06c3160191f2b04e4d553e93294106f0db5f5d

diff_hunk: @@ -338,6 +339,76 @@ def converter_init(self):
         )
         self.assertIn(msg, out)
 
+    @staticmethod
+    @contextlib.contextmanager
+    def _clinic_version(new_version):
+        """Helper for test_version_*() tests"""
+        _saved = clinic.version
+        clinic.version = new_version
+        try:
+            yield
+        finally:
+            clinic.version = _saved
+
+    def test_version_directive_lower(self):
+        with self._clinic_version('3'):
+            block = dedent("""
+                /*[clinic input]
+                version 2
path: Lib/test/test_clinic.py
text: @user1:
I think we can parametrize these tests.

@author:
00a9557f5d4f86e2d1ac3f8e39d15c8d6502845e

@user2:
> 00a9557

Hmm, I know it's annoying when reviewers disagree, but I think I preferred it before 00a9557. I completely take @user1's point that the tests were somewhat repetitive before 00a9557, and I agree that we should try to avoid repetitive tests where possible by using subtests. But I think tests that involve generating an expected output — rather than declaratively stating the expected output — are worse than repetitive tests. It becomes less clear what you're testing, and you risk bugs creeping into the test code.

I'm not convinced these tests in particular are particularly well suited for being parameterised.

@user2:
Actually, I guess I only object to `test_version_directive_insufficient_version()`. Generating the test _input_ (as you do in `test_version_directive()`) is fine; I'm just not keen on generating the expected _output_ of the test that we want to assert

@author:
I have no hard opinions on this matter; I partially reverted 00a9557f5d4f86e2d1ac3f8e39d15c8d6502845e in 0618a202bbd422fac73cc4cbcc04dbf98ab57284.