Repository: Materials-Consortia/optimade-python-tools
Pull request #502 (2020-09-15 22:41:33)

Enable JSON response from the validator
This PR adds a `-j`/`--json` flag to the validator that lowers verbosity to "-1" and only prints a JSON representation of the results to stdout.

codecov[bot]: # [Codecov](https://codecov.io/gh/Materials-Consortia/optimade-python-tools/pull/502?src=pr&el=h1) Report
> Merging [#502](https://codecov.io/gh/Materials-Consortia/optimade-python-tools/pull/502?src=pr&el=desc) into [master](https://codecov.io/gh/Materials-Consortia/optimade-python-tools/commit/e05e0b806e5918ab30b0e8bcbc26b66d3927e306?el=desc) will **decrease** coverage by `0.02%`.
> The diff coverage is `97.05%`.

[![Impacted file tree graph](https://codecov.io/gh/Materials-Consortia/optimade-python-tools/pull/502/graphs/tree.svg?width=650&height=150&src=pr&token=UJAtmqkZZO)](https://codecov.io/gh/Materials-Consortia/optimade-python-tools/pull/502?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master     #502      +/-   ##
==========================================
- Coverage   91.58%   91.56%   -0.03%     
==========================================
  Files          61       61              
  Lines        3103     3106       +3     
==========================================
+ Hits         2842     2844       +2     
- Misses        261      262       +1     
```

| Flag | Coverage Δ | |
|---|---|---|
| #project | `91.56% <97.05%> (-0.03%)` | :arrow_down: |
| #validator | `?` | |

Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.

| [Impacted Files](https://codecov.io/gh/Materials-Consortia/optimade-python-tools/pull/502?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [optimade/validator/\_\_init\_\_.py](https://codecov.io/gh/Materials-Consortia/optimade-python-tools/pull/502/diff?src=pr&el=tree#diff-b3B0aW1hZGUvdmFsaWRhdG9yL19faW5pdF9fLnB5) | `9.09% <0.00%> (-0.29%)` | :arrow_down: |
| [optimade/validator/utils.py](https://codecov.io/gh/Materials-Consortia/optimade-python-tools/pull/502/diff?src=pr&el=tree#diff-b3B0aW1hZGUvdmFsaWRhdG9yL3V0aWxzLnB5) | `90.29% <100.00%> (+0.14%)` | :arrow_up: |
| [optimade/validator/validator.py](https://codecov.io/gh/Materials-Consortia/optimade-python-tools/pull/502/diff?src=pr&el=tree#diff-b3B0aW1hZGUvdmFsaWRhdG9yL3ZhbGlkYXRvci5weQ==) | `82.18% <100.00%> (ø)` | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/Materials-Consortia/optimade-python-tools/pull/502?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/Materials-Consortia/optimade-python-tools/pull/502?src=pr&el=footer). Last update [e05e0b8...d0556bd](https://codecov.io/gh/Materials-Consortia/optimade-python-tools/pull/502?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

ml-evs: Now we have the same problem as with the other PR, mkdocs can't handle `dataclasses` or `NamedTuple` :(
CasperWA: > Now we have the same problem as with the other PR, mkdocs can't handle `dataclasses` or `NamedTuple` :(

I wonder if there is a comment syntax (like `# pylint: disable=some-rule`) to exclude a class from MkDocs?
Otherwise, we should contribute to MkDocs, or rather its doc-string parser :confused: 
CasperWA: > > Now we have the same problem as with the other PR, mkdocs can't handle `dataclasses` or `NamedTuple` :(
> 
> I wonder if there is a comment syntax (like `# pylint: disable=some-rule`) to exclude a class from MkDocs?
> Otherwise, we should contribute to MkDocs, or rather its doc-string parser confused

Looking into the error from the CI run, it seems it is not MkDocs, but rather mkdocstrings.
ml-evs: > > > Now we have the same problem as with the other PR, mkdocs can't handle `dataclasses` or `NamedTuple` :(
> > 
> > 
> > I wonder if there is a comment syntax (like `# pylint: disable=some-rule`) to exclude a class from MkDocs?
> > Otherwise, we should contribute to MkDocs, or rather its doc-string parser confused
> 
> Looking into the error from the CI run, it seems it is not MkDocs, but rather mkdocstrings.

Yeah sorry, that's what I mean every time I say mkdocs... this one seems weird, mkdocstrings is meant to support dataclasses and the error is to do with parsing the markdown `api_reference/validator/validator.md` :confused: 
CasperWA: > > > > Now we have the same problem as with the other PR, mkdocs can't handle `dataclasses` or `NamedTuple` :(
> > > 
> > > 
> > > I wonder if there is a comment syntax (like `# pylint: disable=some-rule`) to exclude a class from MkDocs?
> > > Otherwise, we should contribute to MkDocs, or rather its doc-string parser confused
> > 
> > 
> > Looking into the error from the CI run, it seems it is not MkDocs, but rather mkdocstrings.
> 
> Yeah sorry, that's what I mean every time I say mkdocs... this one seems weird, mkdocstrings is meant to support dataclasses and the error is to do with parsing the markdown `api_reference/validator/validator.md` confused

So what it's doing (I'm guessing) is following the reference and using [`pytkdocs`](https://github.com/pawamoy/pytkdocs) to extract the docstrings, then it's converting this to XML, which is where it fails, because the generated XML file is badly made.
ml-evs: > So what it's doing (I'm guessing) is following the reference (to `optimade/validator/validator.py`) and using [`pytkdocs`](https://github.com/pawamoy/pytkdocs) to extract the docstrings, then it's converting this to XML, which is where it fails, because the generated XML file is badly made.

So it "works" if I remove the explicit `dataclass.field` fields, though it still returns warnings for `__eq__`, `__init__` and `__repr__`:


Patch:
diff --git a/mkdocs.yml b/mkdocs.yml
index e5d0a559..4dc0288e 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -63,6 +63,7 @@ plugins:
               - "!__json_encoder__$"
               - "!__all__$"
               - "!__config__$"
+              - "!ValidatorResults$"
             members: true
             inherited_members: false
             docstring_style: google
diff --git a/optimade/validator/__init__.py b/optimade/validator/__init__.py
index 42834b56..5ca6961f 100644
--- a/optimade/validator/__init__.py
+++ b/optimade/validator/__init__.py
@@ -45,6 +45,12 @@ def validate():
         default=0,
         help="""Increase the verbosity of the output.""",
     )
+    parser.add_argument(
+        "-j",
+        "--json",
+        action="store_true",
+        help="""Only a JSON summary of the validator results will be printed to stdout.""",
+    )
     parser.add_argument(
         "-t",
         "--as-type",
@@ -97,6 +103,7 @@ def validate():
     validator = ImplementationValidator(
         base_url=args["base_url"],
         verbosity=args["verbosity"],
+        respond_json=args["json"],
         as_type=args["as_type"],
         index=args["index"],
         run_optional_tests=not args["skip_optional"],
diff --git a/optimade/validator/utils.py b/optimade/validator/utils.py
index 84102a68..6e890227 100644
--- a/optimade/validator/utils.py
+++ b/optimade/validator/utils.py
@@ -132,7 +132,6 @@ class Client:  # pragma: no cover
             if status_code != 429:
                 break
 
-            print("Hit rate limit, sleeping for 1 s...")
             time.sleep(1)
 
         else:
@@ -242,19 +241,20 @@ def test_case(test_fn: Callable[[Any], Tuple[Any, str]]):
             if not isinstance(result, Exception):
                 if not multistage:
                     if not optional:
-                        validator.success_count += 1
+                        validator.results.success_count += 1
                     else:
-                        validator.optional_success_count += 1
+                        validator.results.optional_success_count += 1
                     message = f"✔: {request} - {msg}"
                     if validator.verbosity > 0:
                         if optional:
                             print(message)
                         else:
                             print_success(message)
-                    elif optional:
-                        print(".", end="", flush=True)
-                    else:
-                        print_success(".", end="", flush=True)
+                    elif validator.verbosity == 0:
+                        if optional:
+                            print(".", end="", flush=True)
+                        else:
+                            print_success(".", end="", flush=True)
             else:
                 internal_error = False
                 request = request.replace("\n", "")
@@ -267,17 +267,21 @@ def test_case(test_fn: Callable[[Any], Tuple[Any, str]]):
 
                 if isinstance(result, InternalError):
                     internal_error = True
-                    validator.internal_failure_count += 1
+                    validator.results.internal_failure_count += 1
                     summary = f"!: {request} - {test_fn.__name__} - failed with internal error"
-                    validator.internal_failure_messages.append((summary, message))
+                    validator.results.internal_failure_messages.append(
+                        (summary, message)
+                    )
                 else:
                     summary = f"✖: {request} - {test_fn.__name__} - failed with error"
                     if not optional:
-                        validator.failure_count += 1
-                        validator.failure_messages.append((summary, message))
+                        validator.results.failure_count += 1
+                        validator.results.failure_messages.append((summary, message))
                     else:
-                        validator.optional_failure_count += 1
-                        validator.optional_failure_messages.append((summary, message))
+                        validator.results.optional_failure_count += 1
+                        validator.results.optional_failure_messages.append(
+                            (summary, message)
+                        )
 
                 if validator.verbosity > 0:
                     if internal_error:
@@ -292,7 +296,7 @@ def test_case(test_fn: Callable[[Any], Tuple[Any, str]]):
                         print_failure(summary)
                         for line in message:
                             print_warning(f"\t{line}")
-                else:
+                elif validator.verbosity == 0:
                     if internal_error:
                         print_notify("!", end="", flush=True)
                     elif optional:
diff --git a/optimade/validator/validator.py b/optimade/validator/validator.py
index 5fd18921..ed83be13 100644
--- a/optimade/validator/validator.py
+++ b/optimade/validator/validator.py
@@ -11,6 +11,7 @@ import sys
 import logging
 import random
 import urllib.parse
+import dataclasses
 from typing import Union, Tuple, Any, List, Dict, Optional
 
 try:
@@ -37,6 +38,24 @@ from optimade.validator.config import VALIDATOR_CONFIG as CONF
 
 VERSIONS_REGEXP = r"/v[0-9]+(\.[0-9]+){,2}"
 
+__all__ = ("ImplementationValidator",)
+
+
+@dataclasses.dataclass
+class ValidatorResults:
+    success_count: int = 0
+    failure_count: int = 0
+    internal_failure_count: int = 0
+    optional_success_count: int = 0
+    optional_failure_count: int = 0
+    failure_messages: List[Tuple[str, str]] = dataclasses.field(default_factory=list)
+    internal_failure_messages: List[Tuple[str, str]] = dataclasses.field(
+        default_factory=list
+    )
+    optional_failure_messages: List[Tuple[str, str]] = dataclasses.field(
+        default_factory=list
+    )
+
 
 class ImplementationValidator:
     """Class used to make a series of checks against a particular
@@ -63,6 +82,7 @@ class ImplementationValidator:
         client: Any = None,
         base_url: str = None,
         verbosity: int = 0,
+        respond_json: bool = False,
         page_limit: int = 5,
         max_retries: int = 5,
         run_optional_tests: bool = True,
@@ -82,6 +102,8 @@ class ImplementationValidator:
                 base of the OPTIMADE implementation.
             verbosity: The verbosity of the output and logging as an integer
                 (`0`: critical, `1`: warning, `2`: info, `3`: debug).
+            respond_json: If `True`, print only a JSON representation of the
+                results of validation to stdout.
             page_limit: The default page limit to apply to filters.
             max_retries: Argument is passed to the client for how many
                 attempts to make for a request before failing.
@@ -101,6 +123,7 @@ class ImplementationValidator:
         self.index = index
         self.run_optional_tests = run_optional_tests
         self.fail_fast = fail_fast
+        self.respond_json = respond_json
 
         if as_type is None:
             self.as_type_cls = None
@@ -147,14 +170,7 @@ class ImplementationValidator:
         self._test_id_by_type = {}
         self._entry_info_by_type = {}
 
-        self.success_count = 0
-        self.failure_count = 0
-        self.internal_failure_count = 0
-        self.optional_success_count = 0
-        self.optional_failure_count = 0
-        self.failure_messages = []
-        self.internal_failure_messages = []
-        self.optional_failure_messages = []
+        self.results = ValidatorResults()
 
     def _setup_log(self):
         """ Define stdout log based on given verbosity. """
@@ -164,7 +180,12 @@ class ImplementationValidator:
         stdout_handler.setFormatter(
             logging.Formatter("%(asctime)s - %(name)s | %(levelname)8s: %(message)s")
         )
-        self._log.addHandler(stdout_handler)
+
+        if not self.respond_json:
+            self._log.addHandler(stdout_handler)
+        else:
+            self.verbosity = -1
+
         if self.verbosity == 0:
             self._log.setLevel(logging.CRITICAL)
         elif self.verbosity == 1:
@@ -176,23 +197,27 @@ class ImplementationValidator:
 
     def print_summary(self):
         """ Print a summary of the results of validation. """
-        if self.failure_messages:
+        if self.respond_json:
+            print(json.dumps(dataclasses.asdict(self.results), indent=2))
+            return
+
+        if self.results.failure_messages:
             print("\n\nFAILURES")
             print("========\n")
-            for message in self.failure_messages:
+            for message in self.results.failure_messages:
                 print_failure(message[0])
                 for line in message[1]:
                     print_warning("\t" + line)
 
-        if self.optional_failure_messages:
+        if self.results.optional_failure_messages:
             print("\n\nOPTIONAL TEST FAILURES")
             print("======================\n")
-            for message in self.optional_failure_messages:
+            for message in self.results.optional_failure_messages:
                 print_notify(message[0])
                 for line in message[1]:
                     print_warning("\t" + line)
 
-        if self.internal_failure_messages:
+        if self.results.internal_failure_messages:
             print("\n\nINTERNAL FAILURES")
             print("=================\n")
             print(
@@ -201,13 +226,13 @@ class ImplementationValidator:
                 "https://github.com/Materials-Consortia/optimade-python-tools/issues/new.\n"
             )
 
-            for message in self.internal_failure_messages:
+            for message in self.results.internal_failure_messages:
                 print_warning(message[0])
                 for line in message[1]:
                     print_warning("\t" + line)
 
         if self.valid or (not self.valid and not self.fail_fast):
-            final_message = f"\n\nPassed {self.success_count} out of {self.success_count + self.failure_count + self.internal_failure_count} tests."
+            final_message = f"\n\nPassed {self.results.success_count} out of {self.results.success_count + self.results.failure_count + self.results.internal_failure_count} tests."
             if not self.valid:
                 print_failure(final_message)
             else:
@@ -215,8 +240,8 @@ class ImplementationValidator:
 
             if self.run_optional_tests and not self.fail_fast:
                 print(
-                    f"Additionally passed {self.optional_success_count} out of "
-                    f"{self.optional_success_count + self.optional_failure_count} optional tests."
+                    f"Additionally passed {self.results.optional_success_count} out of "
+                    f"{self.results.optional_success_count + self.results.optional_failure_count} optional tests."
                 )
 
     def validate_implementation(self):
@@ -238,11 +263,12 @@ class ImplementationValidator:
                 self.as_type_cls,
             )
             self._test_as_type()
-            self.valid = not bool(self.failure_count)
+            self.valid = not bool(self.results.failure_count)
             return
 
         # Test entire implementation
-        print(f"Testing entire implementation at {self.base_url}...")
+        if self.verbosity >= 0:
+            print(f"Testing entire implementation at {self.base_url}")
         info_endp = CONF.info_endpoint
         self._log.debug("Testing base info endpoint of %s", info_endp)
 
@@ -297,7 +323,9 @@ class ImplementationValidator:
         self._log.debug("Testing %s endpoint", CONF.links_endpoint)
         self._test_info_or_links_endpoint(CONF.links_endpoint)
 
-        self.valid = not (self.failure_count or self.internal_failure_count)
+        self.valid = not (
+            self.results.failure_count or self.results.internal_failure_count
+        )
 
         self.print_summary()
 
@@ -390,7 +418,7 @@ class ImplementationValidator:
                 f"Some 'MUST' properties were missing from info/{endp}: {missing}"
             )
 
-        return True, "Found all required properties in entry info for endpoint {endp}"
+        return True, f"Found all required properties in entry info for endpoint {endp}"
 
     @test_case
     def _get_archetypal_entry(self, endp: str) -> Tuple[Dict[str, Any], str]:


Test patch:
diff --git a/tests/server/test_server_validation.py b/tests/server/test_server_validation.py
index fbe35504..1548052c 100644
--- a/tests/server/test_server_validation.py
+++ b/tests/server/test_server_validation.py
@@ -1,4 +1,6 @@
 import os
+import json
+import dataclasses
 from traceback import print_exc
 
 import pytest
@@ -14,10 +16,32 @@ def test_with_validator(both_fake_remote_clients):
         index=both_fake_remote_clients.app == app,
         verbosity=5,
     )
-    try:
-        validator.validate_implementation()
-    except Exception:
-        print_exc()
+
+    validator.validate_implementation()
+    assert validator.valid
+
+
+def test_with_validator_json_response(both_fake_remote_clients, capsys):
+    """ Test that the validator writes compliant JSON when requested. """
+    from optimade.server.main_index import app
+
+    validator = ImplementationValidator(
+        client=both_fake_remote_clients,
+        index=both_fake_remote_clients.app == app,
+        respond_json=True,
+    )
+    validator.validate_implementation()
+
+    captured = capsys.readouterr()
+    json_response = json.loads(captured.out)
+    assert json_response["failure_count"] == 0
+    assert json_response["internal_failure_count"] == 0
+    assert json_response["optional_failure_count"] == 0
+    assert validator.results.failure_count == 0
+    assert validator.results.internal_failure_count == 0
+    assert validator.results.optional_failure_count == 0
+    assert dataclasses.asdict(validator.results) == json_response
+
     assert validator.valid
 
 
diff --git a/tests/validator/test_utils.py b/tests/validator/test_utils.py
index 12f0eba2..afce84f0 100644
--- a/tests/validator/test_utils.py
+++ b/tests/validator/test_utils.py
@@ -26,11 +26,11 @@ def test_normal_test_case():
     validator = ImplementationValidator(base_url="http://example.org", verbosity=0)
 
     output = dummy_test_case(validator, ([1, 2, 3], "message"), request="test_request")
-    assert validator.success_count == 1
-    assert validator.optional_success_count == 0
-    assert validator.failure_count == 0
-    assert validator.optional_failure_count == 0
-    assert validator.internal_failure_count == 0
+    assert validator.results.success_count == 1
+    assert validator.results.optional_success_count == 0
+    assert validator.results.failure_count == 0
+    assert validator.results.optional_failure_count == 0
+    assert validator.results.internal_failure_count == 0
     assert output[0] == [1, 2, 3]
     assert output[1] == "message"
 
@@ -42,11 +42,11 @@ def test_optional_test_case():
     output = dummy_test_case(
         validator, ("string response", "message"), request="test_request", optional=True
     )
-    assert validator.success_count == 0
-    assert validator.optional_success_count == 1
-    assert validator.failure_count == 0
-    assert validator.optional_failure_count == 0
-    assert validator.internal_failure_count == 0
+    assert validator.results.success_count == 0
+    assert validator.results.optional_success_count == 1
+    assert validator.results.failure_count == 0
+    assert validator.results.optional_failure_count == 0
+    assert validator.results.internal_failure_count == 0
     assert output[0] == "string response"
     assert output[1] == "message"
 
@@ -57,11 +57,11 @@ def test_ignored_test_case():
 
     # Test returns None, so should not increment success/failure
     output = dummy_test_case(validator, (None, "message"), request="test_request")
-    assert validator.success_count == 0
-    assert validator.optional_success_count == 0
-    assert validator.failure_count == 0
-    assert validator.optional_failure_count == 0
-    assert validator.internal_failure_count == 0
+    assert validator.results.success_count == 0
+    assert validator.results.optional_success_count == 0
+    assert validator.results.failure_count == 0
+    assert validator.results.optional_failure_count == 0
+    assert validator.results.internal_failure_count == 0
     assert output[0] is None
     assert output[1] == "message"
 
@@ -77,11 +77,11 @@ def test_skip_optional_test_case():
     output = dummy_test_case(
         validator, ({"test": "dict"}, "message"), request="test_request", optional=True
     )
-    assert validator.success_count == 0
-    assert validator.optional_success_count == 0
-    assert validator.failure_count == 0
-    assert validator.optional_failure_count == 0
-    assert validator.internal_failure_count == 0
+    assert validator.results.success_count == 0
+    assert validator.results.optional_success_count == 0
+    assert validator.results.failure_count == 0
+    assert validator.results.optional_failure_count == 0
+    assert validator.results.internal_failure_count == 0
     assert output[0] is None
     assert output[1] == "skipping optional"
 
@@ -89,11 +89,11 @@ def test_skip_optional_test_case():
     output = dummy_test_case(
         validator, ({"test": "dict"}, "message"), request="test_request", optional=False
     )
-    assert validator.success_count == 1
-    assert validator.optional_success_count == 0
-    assert validator.failure_count == 0
-    assert validator.optional_failure_count == 0
-    assert validator.internal_failure_count == 0
+    assert validator.results.success_count == 1
+    assert validator.results.optional_success_count == 0
+    assert validator.results.failure_count == 0
+    assert validator.results.optional_failure_count == 0
+    assert validator.results.internal_failure_count == 0
     assert output[0] == {"test": "dict"}
     assert output[1] == "message"
 
@@ -110,20 +110,20 @@ def test_expected_failure_test_case():
         request="test_request",
         raise_exception=ResponseError("Dummy error"),
     )
-    assert validator.success_count == 0
-    assert validator.optional_success_count == 0
-    assert validator.failure_count == 1
-    assert validator.optional_failure_count == 0
-    assert validator.internal_failure_count == 0
+    assert validator.results.success_count == 0
+    assert validator.results.optional_success_count == 0
+    assert validator.results.failure_count == 1
+    assert validator.results.optional_failure_count == 0
+    assert validator.results.internal_failure_count == 0
 
     assert output[0] is None
     assert output[1] == "ResponseError: Dummy error"
 
     assert (
-        validator.failure_messages[-1][0]
+        validator.results.failure_messages[-1][0]
         == "✖: http://example.org/test_request - dummy_test_case - failed with error"
     )
-    assert validator.failure_messages[-1][1] == ["ResponseError: Dummy error"]
+    assert validator.results.failure_messages[-1][1] == ["ResponseError: Dummy error"]
 
     output = dummy_test_case(
         validator,
@@ -132,20 +132,22 @@ def test_expected_failure_test_case():
         raise_exception=ResponseError("Dummy error"),
         optional=True,
     )
-    assert validator.success_count == 0
-    assert validator.optional_success_count == 0
-    assert validator.failure_count == 1
-    assert validator.optional_failure_count == 1
-    assert validator.internal_failure_count == 0
+    assert validator.results.success_count == 0
+    assert validator.results.optional_success_count == 0
+    assert validator.results.failure_count == 1
+    assert validator.results.optional_failure_count == 1
+    assert validator.results.internal_failure_count == 0
 
     assert output[0] is None
     assert output[1] == "ResponseError: Dummy error"
 
     assert (
-        validator.optional_failure_messages[-1][0]
+        validator.results.optional_failure_messages[-1][0]
         == "✖: http://example.org/test_request - dummy_test_case - failed with error"
     )
-    assert validator.optional_failure_messages[-1][1] == ["ResponseError: Dummy error"]
+    assert validator.results.optional_failure_messages[-1][1] == [
+        "ResponseError: Dummy error"
+    ]
 
     output = dummy_test_case(
         validator,
@@ -154,11 +156,11 @@ def test_expected_failure_test_case():
         raise_exception=json.JSONDecodeError("Dummy JSON error", "{}", 0),
         optional=True,
     )
-    assert validator.success_count == 0
-    assert validator.optional_success_count == 0
-    assert validator.failure_count == 1
-    assert validator.optional_failure_count == 2
-    assert validator.internal_failure_count == 0
+    assert validator.results.success_count == 0
+    assert validator.results.optional_success_count == 0
+    assert validator.results.failure_count == 1
+    assert validator.results.optional_failure_count == 2
+    assert validator.results.internal_failure_count == 0
 
     assert output[0] is None
     assert (
@@ -166,10 +168,10 @@ def test_expected_failure_test_case():
         == "Critical: unable to parse server response as JSON. JSONDecodeError: Dummy JSON error: line 1 column 1 (char 0)"
     )
     assert (
-        validator.optional_failure_messages[-1][0]
+        validator.results.optional_failure_messages[-1][0]
         == "✖: http://example.org/test_request - dummy_test_case - failed with error"
     )
-    assert validator.optional_failure_messages[-1][1] == [
+    assert validator.results.optional_failure_messages[-1][1] == [
         "Critical: unable to parse server response as JSON. JSONDecodeError: Dummy JSON error: line 1 column 1 (char 0)"
     ]
 
@@ -185,19 +187,19 @@ def test_unexpected_failure_test_case():
         request="test_request",
         raise_exception=FileNotFoundError("Unexpected error"),
     )
-    assert validator.success_count == 0
-    assert validator.optional_success_count == 0
-    assert validator.failure_count == 0
-    assert validator.optional_failure_count == 0
-    assert validator.internal_failure_count == 1
+    assert validator.results.success_count == 0
+    assert validator.results.optional_success_count == 0
+    assert validator.results.failure_count == 0
+    assert validator.results.optional_failure_count == 0
+    assert validator.results.internal_failure_count == 1
 
     assert output[0] is None
     assert output[1] == "FileNotFoundError: Unexpected error"
     assert (
-        validator.internal_failure_messages[-1][0]
+        validator.results.internal_failure_messages[-1][0]
         == "!: http://example.org/test_request - dummy_test_case - failed with internal error"
     )
-    assert validator.internal_failure_messages[-1][1] == [
+    assert validator.results.internal_failure_messages[-1][1] == [
         "FileNotFoundError: Unexpected error"
     ]
 
@@ -208,19 +210,19 @@ def test_unexpected_failure_test_case():
         raise_exception=FileNotFoundError("Unexpected error"),
         optional=True,
     )
-    assert validator.success_count == 0
-    assert validator.optional_success_count == 0
-    assert validator.failure_count == 0
-    assert validator.optional_failure_count == 0
-    assert validator.internal_failure_count == 2
+    assert validator.results.success_count == 0
+    assert validator.results.optional_success_count == 0
+    assert validator.results.failure_count == 0
+    assert validator.results.optional_failure_count == 0
+    assert validator.results.internal_failure_count == 2
 
     assert output[0] is None
     assert output[1] == "FileNotFoundError: Unexpected error"
     assert (
-        validator.internal_failure_messages[-1][0]
+        validator.results.internal_failure_messages[-1][0]
         == "!: http://example.org/test_request - dummy_test_case - failed with internal error"
     )
-    assert validator.internal_failure_messages[-1][1] == [
+    assert validator.results.internal_failure_messages[-1][1] == [
         "FileNotFoundError: Unexpected error"
     ]
 
@@ -236,11 +238,11 @@ def test_multistage_test_case():
         request="test_request",
         multistage=True,
     )
-    assert validator.success_count == 0
-    assert validator.optional_success_count == 0
-    assert validator.failure_count == 0
-    assert validator.optional_failure_count == 0
-    assert validator.internal_failure_count == 0
+    assert validator.results.success_count == 0
+    assert validator.results.optional_success_count == 0
+    assert validator.results.failure_count == 0
+    assert validator.results.optional_failure_count == 0
+    assert validator.results.internal_failure_count == 0
 
     assert output[0] == {"test": "dict"}
     assert output[1] == "message"
@@ -252,19 +254,21 @@ def test_multistage_test_case():
         raise_exception=ResponseError("Stage of test failed"),
         multistage=True,
     )
-    assert validator.success_count == 0
-    assert validator.optional_success_count == 0
-    assert validator.failure_count == 1
-    assert validator.optional_failure_count == 0
-    assert validator.internal_failure_count == 0
+    assert validator.results.success_count == 0
+    assert validator.results.optional_success_count == 0
+    assert validator.results.failure_count == 1
+    assert validator.results.optional_failure_count == 0
+    assert validator.results.internal_failure_count == 0
 
     assert output[0] is None
     assert output[1] == "ResponseError: Stage of test failed"
     assert (
-        validator.failure_messages[-1][0]
+        validator.results.failure_messages[-1][0]
         == "✖: http://example.org/test_request - dummy_test_case - failed with error"
     )
-    assert validator.failure_messages[-1][1] == ["ResponseError: Stage of test failed"]
+    assert validator.results.failure_messages[-1][1] == [
+        "ResponseError: Stage of test failed"
+    ]
 
 
 def test_fail_fast_test_case():
@@ -281,18 +285,18 @@ def test_fail_fast_test_case():
         raise_exception=ResponseError("Optional test failed"),
         optional=True,
     )
-    assert validator.success_count == 0
-    assert validator.optional_success_count == 0
-    assert validator.failure_count == 0
-    assert validator.optional_failure_count == 1
-    assert validator.internal_failure_count == 0
+    assert validator.results.success_count == 0
+    assert validator.results.optional_success_count == 0
+    assert validator.results.failure_count == 0
+    assert validator.results.optional_failure_count == 1
+    assert validator.results.internal_failure_count == 0
 
     assert output[0] is None
     assert output[1] == "ResponseError: Optional test failed"
-    assert validator.optional_failure_messages[-1][0] == (
+    assert validator.results.optional_failure_messages[-1][0] == (
         "✖: http://example.org/test_request - dummy_test_case - failed with error"
     )
-    assert validator.optional_failure_messages[-1][1] == [
+    assert validator.results.optional_failure_messages[-1][1] == [
         "ResponseError: Optional test failed"
     ]
 
@@ -305,15 +309,15 @@ def test_fail_fast_test_case():
             raise_exception=ResponseError("Non-optional test failed"),
             optional=False,
         )
-    assert validator.success_count == 0
-    assert validator.optional_success_count == 0
-    assert validator.failure_count == 1
-    assert validator.optional_failure_count == 1
-    assert validator.internal_failure_count == 0
-    assert validator.failure_messages[-1][0] == (
+    assert validator.results.success_count == 0
+    assert validator.results.optional_success_count == 0
+    assert validator.results.failure_count == 1
+    assert validator.results.optional_failure_count == 1
+    assert validator.results.internal_failure_count == 0
+    assert validator.results.failure_messages[-1][0] == (
         "✖: http://example.org/test_request - dummy_test_case - failed with error"
     )
-    assert validator.failure_messages[-1][1] == [
+    assert validator.results.failure_messages[-1][1] == [
         "ResponseError: Non-optional test failed"
     ]
 
@@ -325,15 +329,15 @@ def test_fail_fast_test_case():
             request="test_request",
             raise_exception=FileNotFoundError("Internal error"),
         )
-    assert validator.success_count == 0
-    assert validator.optional_success_count == 0
-    assert validator.failure_count == 1
-    assert validator.optional_failure_count == 1
-    assert validator.internal_failure_count == 1
-    assert validator.internal_failure_messages[-1][0] == (
+    assert validator.results.success_count == 0
+    assert validator.results.optional_success_count == 0
+    assert validator.results.failure_count == 1
+    assert validator.results.optional_failure_count == 1
+    assert validator.results.internal_failure_count == 1
+    assert validator.results.internal_failure_messages[-1][0] == (
         "!: http://example.org/test_request - dummy_test_case - failed with internal error"
     )
-    assert validator.internal_failure_messages[-1][1] == [
+    assert validator.results.internal_failure_messages[-1][1] == [
         "FileNotFoundError: Internal error"
     ]
 
@@ -353,8 +357,8 @@ def test_that_system_exit_is_fatal_in_test_case():
             optional=True,
         )
 
-    assert validator.success_count == 0
-    assert validator.optional_success_count == 0
-    assert validator.failure_count == 0
-    assert validator.optional_failure_count == 0
-    assert validator.internal_failure_count == 0
+    assert validator.results.success_count == 0
+    assert validator.results.optional_success_count == 0
+    assert validator.results.failure_count == 0
+    assert validator.results.optional_failure_count == 0
+    assert validator.results.internal_failure_count == 0