Repository: Materials-Consortia/optimade-python-tools

Issue:
'Chosen entry had no value for ...' when property is not requested
As of 995f32de7b842059f204969f36bd4aea67188d57 the validator throws the following (and similar) errors for test implementation for the COD:
```
✖: https://www.crystallography.net/cod-test/optimade/; testing queries for structures->cartesian_site_positions - _construct_queries_for_property - failed with error
	ResponseError: Chosen entry had no value for ''cartesian_site_positions''
```
I did some debugging (with #513 in action) to find out that chosen entries in fact have values for tested properties, but these properties do not seem to be requested in the initial request at first place. I would suggest requesting all optionally included properties via `response_fields` prior to running `_construct_queries_for_property`, as these tests will fail for implementations following the specs:

> Properties not marked as REQUIRED in any case, MUST be included only if explicitly requested in the query parameter `response_fields`. Otherwise they SHOULD NOT be included.

Feel free to close this issue if you think this is a duplicate. I gave existing issues a glance, but nothing caught my eye.

Discussion:
codecov[bot]: # [Codecov](https://codecov.io/gh/Materials-Consortia/optimade-python-tools/pull/515?src=pr&el=h1) Report
> Merging [#515](https://codecov.io/gh/Materials-Consortia/optimade-python-tools/pull/515?src=pr&el=desc) into [master](https://codecov.io/gh/Materials-Consortia/optimade-python-tools/commit/e7432a2cdbd38f78c7d2dbfacc115d461e4dd9c0?el=desc) will **increase** coverage by `0.14%`.
> The diff coverage is `89.04%`.

[![Impacted file tree graph](https://codecov.io/gh/Materials-Consortia/optimade-python-tools/pull/515/graphs/tree.svg?width=650&height=150&src=pr&token=UJAtmqkZZO)](https://codecov.io/gh/Materials-Consortia/optimade-python-tools/pull/515?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master     #515      +/-   ##
==========================================
+ Coverage   91.47%   91.62%   +0.14%     
==========================================
  Files          62       62              
  Lines        3110     3127      +17     
==========================================
+ Hits         2845     2865      +20     
+ Misses        265      262       -3     
```

| Flag | Coverage Δ | |
|---|---|---|
| #project | `91.62% <89.04%> (+0.14%)` | :arrow_up: |
| #validator | `63.57% <45.20%> (+0.03%)` | :arrow_up: |

Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.

| [Impacted Files](https://codecov.io/gh/Materials-Consortia/optimade-python-tools/pull/515?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [optimade/validator/validator.py](https://codecov.io/gh/Materials-Consortia/optimade-python-tools/pull/515/diff?src=pr&el=tree#diff-b3B0aW1hZGUvdmFsaWRhdG9yL3ZhbGlkYXRvci5weQ==) | `81.83% <81.81%> (-0.63%)` | :arrow_down: |
| [optimade/validator/utils.py](https://codecov.io/gh/Materials-Consortia/optimade-python-tools/pull/515/diff?src=pr&el=tree#diff-b3B0aW1hZGUvdmFsaWRhdG9yL3V0aWxzLnB5) | `94.83% <90.32%> (+4.54%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/Materials-Consortia/optimade-python-tools/pull/515?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/Materials-Consortia/optimade-python-tools/pull/515?src=pr&el=footer). Last update [e7432a2...b5a073e](https://codecov.io/gh/Materials-Consortia/optimade-python-tools/pull/515?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

ml-evs: I'll give @CasperWA until tomorrow to have a look if he wants to, if not I'll merge in the morning.
ml-evs: > I'm having a bit of trouble penetrating all of your changes here, but it seems all right.

Yeah, this one got a bit out of hand, but think it would have been more annoying as multiple PRs. I'll rebase this one into a few feature commits to maintain some kind of structure.

Patch:
diff --git a/optimade/server/routers/utils.py b/optimade/server/routers/utils.py
index 265ea537..45ecdefc 100644
--- a/optimade/server/routers/utils.py
+++ b/optimade/server/routers/utils.py
@@ -78,7 +78,15 @@ def handle_response_fields(
     new_results = []
     while results:
         entry = results.pop(0)
-        new_entry = entry.dict(exclude_unset=True)
+
+        # TODO: re-enable exclude_unset when proper handling of known/unknown fields
+        # has been implemented (relevant issue: https://github.com/Materials-Consortia/optimade-python-tools/issues/263)
+        # Have to handle top level fields explicitly here for now
+        new_entry = entry.dict(exclude_unset=False)
+        for field in ("relationships", "links", "meta", "type", "id"):
+            if field in new_entry and new_entry[field] is None:
+                del new_entry[field]
+
         for field in exclude_fields:
             if field in new_entry["attributes"]:
                 del new_entry["attributes"][field]
diff --git a/optimade/validator/config.py b/optimade/validator/config.py
index 4fe8384c..d9de66fd 100644
--- a/optimade/validator/config.py
+++ b/optimade/validator/config.py
@@ -18,6 +18,7 @@ from optimade.validator.utils import (
     ValidatorStructureResponseOne,
     ValidatorStructureResponseMany,
 )
+from optimade.server.mappers import BaseResourceMapper
 from optimade.server.schemas import (
     ENTRY_INFO_SCHEMAS,
     retrieve_queryable_properties,
@@ -153,6 +154,10 @@ class ValidatorConfig(BaseSettings):
         _NON_ENTRY_ENDPOINTS,
         description="The list specification-mandated endpoint names that do not contain entries",
     )
+    top_level_non_attribute_fields: Set[str] = Field(
+        BaseResourceMapper.TOP_LEVEL_NON_ATTRIBUTES_FIELDS,
+        description="Field names to treat as top-level",
+    )
 
 
 VALIDATOR_CONFIG = ValidatorConfig()
diff --git a/optimade/validator/utils.py b/optimade/validator/utils.py
index 08a69f99..5ffc948c 100644
--- a/optimade/validator/utils.py
+++ b/optimade/validator/utils.py
@@ -15,6 +15,7 @@ used by the validator. The two main features being:
 import time
 import sys
 import urllib.parse
+import dataclasses
 import traceback as tb
 from typing import List, Optional, Dict, Any, Callable, Tuple
 
@@ -67,6 +68,98 @@ def print_success(string, **kwargs):
     print(f"\033[92m\033[1m{string}\033[0m", **kwargs)
 
 
+@dataclasses.dataclass
+class ValidatorResults:
+    """ A dataclass to store and print the validation results. """
+
+    success_count: int = 0
+    failure_count: int = 0
+    internal_failure_count: int = 0
+    optional_success_count: int = 0
+    optional_failure_count: int = 0
+    failure_messages: List[Tuple[str, str]] = dataclasses.field(default_factory=list)
+    internal_failure_messages: List[Tuple[str, str]] = dataclasses.field(
+        default_factory=list
+    )
+    optional_failure_messages: List[Tuple[str, str]] = dataclasses.field(
+        default_factory=list
+    )
+    verbosity: int = 0
+
+    def add_success(self, summary: str, success_type: Optional[str] = None):
+        """Register a validation success to the results class.
+
+        Parameters:
+            summary: A summary of the success to be printed.
+            success_type: Either `None` or `"optional"` depending on the
+                type of the check.
+
+        """
+        success_types = (None, "optional")
+        if success_type not in success_types:
+            raise RuntimeError(
+                f"`success_type` must be one of {success_types}, not {success_type}."
+            )
+
+        if success_type is None:
+            self.success_count += 1
+        elif success_type == "optional":
+            self.optional_success_count += 1
+
+        message = f"✔: {summary}"
+        pretty_print = print if success_type == "optional" else print_success
+
+        if self.verbosity > 0:
+            pretty_print(message)
+        elif self.verbosity == 0:
+            pretty_print(".", end="", flush=True)
+
+    def add_failure(
+        self, summary: str, message: str, failure_type: Optional[str] = None
+    ):
+        """Register a validation failure to the results class with
+        corresponding summary, message and type.
+
+        Parameters:
+            summary: Short error message.
+            message: Full error message, potentially containing a traceback.
+            failure_type: Either `None`, `"internal"` or `"optional"`
+                depending on the type of check that was failed.
+
+        """
+        failure_types = (None, "internal", "optional")
+        if failure_type not in failure_types:
+            raise RuntimeError(
+                f"`failure_type` must be one of {failure_types}, not {failure_type}."
+            )
+
+        if failure_type is None:
+            self.failure_count += 1
+            self.failure_messages.append((summary, message))
+        elif failure_type == "internal":
+            self.internal_failure_count += 1
+            self.internal_failure_messages.append((summary, message))
+        elif failure_type == "optional":
+            self.optional_failure_count += 1
+            self.optional_failure_messages.append((summary, message))
+
+        pprint_types = {
+            "internal": (print_notify, print_warning),
+            "optional": (print, print),
+        }
+        pprint, warning_pprint = pprint_types.get(
+            "failure_type", (print_failure, print_warning)
+        )
+
+        symbol = "!" if failure_type == "internal" else "✖"
+        if self.verbosity == 0:
+            pprint(symbol, end="", flush=True)
+        elif self.verbosity > 0:
+            pprint(f"{symbol}: {summary}")
+            for line in message.split("\n"):
+                warning_pprint(f"\t{line}")
+
+
 class Client:  # pragma: no cover
     def __init__(self, base_url: str, max_retries=5):
         """Initialises the Client with the given `base_url` without testing
@@ -240,23 +333,9 @@ def test_case(test_fn: Callable[[Any], Tuple[Any, str]]):
 
             if not isinstance(result, Exception):
                 if not multistage:
-                    if not optional:
-                        validator.results.success_count += 1
-                    else:
-                        validator.results.optional_success_count += 1
-                    message = f"✔: {request} - {msg}"
-                    if validator.verbosity > 0:
-                        if optional:
-                            print(message)
-                        else:
-                            print_success(message)
-                    elif validator.verbosity == 0:
-                        if optional:
-                            print(".", end="", flush=True)
-                        else:
-                            print_success(".", end="", flush=True)
+                    success_type = "optional" if optional else None
+                    validator.results.add_success(f"{request} - {msg}", success_type)
             else:
-                internal_error = False
                 request = request.replace("\n", "")
                 message = msg.split("\n")
                 if validator.verbosity > 1:
@@ -265,44 +344,20 @@ def test_case(test_fn: Callable[[Any], Tuple[Any, str]]):
                     if not isinstance(result, ValidationError):
                         message += traceback.split("\n")
 
+                message = "\n".join(message)
+
                 if isinstance(result, InternalError):
-                    internal_error = True
-                    validator.results.internal_failure_count += 1
-                    summary = f"!: {request} - {test_fn.__name__} - failed with internal error"
-                    validator.results.internal_failure_messages.append(
-                        (summary, message)
+                    summary = (
+                        f"{request} - {test_fn.__name__} - failed with internal error"
                     )
+                    failure_type = "internal"
                 else:
-                    summary = f"✖: {request} - {test_fn.__name__} - failed with error"
-                    if not optional:
-                        validator.results.failure_count += 1
-                        validator.results.failure_messages.append((summary, message))
-                    else:
-                        validator.results.optional_failure_count += 1
-                        validator.results.optional_failure_messages.append(
-                            (summary, message)
-                        )
-
-                if validator.verbosity > 0:
-                    if internal_error:
-                        print_notify(summary)
-                        for line in message:
-                            print_warning(f"\t{line}")
-                    elif optional:
-                        print(summary)
-                        for line in message:
-                            print(f"\t{line}")
-                    else:
-                        print_failure(summary)
-                        for line in message:
-                            print_warning(f"\t{line}")
-                elif validator.verbosity == 0:
-                    if internal_error:
-                        print_notify("!", end="", flush=True)
-                    elif optional:
-                        print("✖", end="", flush=True)
-                    else:
-                        print_failure("✖", end="", flush=True)
+                    summary = f"{request} - {test_fn.__name__} - failed with error"
+                    failure_type = "optional" if optional else None
+
+                validator.results.add_failure(
+                    summary, message, failure_type=failure_type
+                )
 
                 # set failure result to None as this is expected by other functions
                 result = None
diff --git a/optimade/validator/validator.py b/optimade/validator/validator.py
index 0851db37..8763fc96 100644
--- a/optimade/validator/validator.py
+++ b/optimade/validator/validator.py
@@ -32,6 +32,7 @@ from optimade.validator.utils import (
     ResponseError,
     ValidatorEntryResponseOne,
     ValidatorEntryResponseMany,
+    ValidatorResults,
 )
 
 from optimade.validator.config import VALIDATOR_CONFIG as CONF
@@ -41,22 +42,6 @@ VERSIONS_REGEXP = r"/v[0-9]+(\.[0-9]+){,2}"
 __all__ = ("ImplementationValidator",)
 
 
-@dataclasses.dataclass
-class ValidatorResults:
-    success_count: int = 0
-    failure_count: int = 0
-    internal_failure_count: int = 0
-    optional_success_count: int = 0
-    optional_failure_count: int = 0
-    failure_messages: List[Tuple[str, str]] = dataclasses.field(default_factory=list)
-    internal_failure_messages: List[Tuple[str, str]] = dataclasses.field(
-        default_factory=list
-    )
-    optional_failure_messages: List[Tuple[str, str]] = dataclasses.field(
-        default_factory=list
-    )
-
-
 class ImplementationValidator:
     """Class used to make a series of checks against a particular
     OPTIMADE implementation over HTTP.
@@ -170,7 +155,7 @@ class ImplementationValidator:
         self._test_id_by_type = {}
         self._entry_info_by_type = {}
 
-        self.results = ValidatorResults()
+        self.results = ValidatorResults(verbosity=self.verbosity)
 
     def _setup_log(self):
         """ Define stdout log based on given verbosity. """
@@ -192,7 +177,7 @@ class ImplementationValidator:
             self._log.setLevel(logging.WARNING)
         elif self.verbosity == 2:
             self._log.setLevel(logging.INFO)
-        else:
+        elif self.verbosity > 0:
             self._log.setLevel(logging.DEBUG)
 
     def print_summary(self):
@@ -206,7 +191,7 @@ class ImplementationValidator:
             print("========\n")
             for message in self.results.failure_messages:
                 print_failure(message[0])
-                for line in message[1]:
+                for line in message[1].split("\n"):
                     print_warning("\t" + line)
 
         if self.results.optional_failure_messages:
@@ -214,7 +199,7 @@ class ImplementationValidator:
             print("======================\n")
             for message in self.results.optional_failure_messages:
                 print_notify(message[0])
-                for line in message[1]:
+                for line in message[1].split("\n"):
                     print_warning("\t" + line)
 
         if self.results.internal_failure_messages:
@@ -228,7 +213,7 @@ class ImplementationValidator:
 
             for message in self.results.internal_failure_messages:
                 print_warning(message[0])
-                for line in message[1]:
+                for line in message[1].split("\n"):
                     print_warning("\t" + line)
 
         if self.valid or (not self.valid and not self.fail_fast):
@@ -312,7 +297,7 @@ class ImplementationValidator:
         # in the next loop.
         for endp in self.available_json_endpoints:
             self._log.debug("Testing multiple entry endpoint of %s", endp)
-            self._test_multi_entry_endpoint(f"{endp}?page_limit={self.page_limit}")
+            self._test_multi_entry_endpoint(endp)
 
         # Test that the single IDs scraped earlier work with the single entry endpoint
         for endp in self.available_json_endpoints:
@@ -346,14 +331,21 @@ class ImplementationValidator:
         """
         entry_info = self._entry_info_by_type.get(endp)
         _impl_properties = self._check_entry_info(entry_info, endp)
+        prop_list = list(_impl_properties.keys())
 
-        chosen_entry, _ = self._get_archetypal_entry(endp)
+        self._check_response_fields(endp, prop_list)
+        chosen_entry, _ = self._get_archetypal_entry(endp, prop_list)
 
         if not entry_info:
             raise ResponseError(
                 f"Unable to generate filters for endpoint {endp}: entry info not found."
             )
 
+        if not chosen_entry:
+            raise ResponseError(
+                f"Unable to generate filters for endpoint {endp}: no valid entries found."
+            )
+
         for prop in _impl_properties:
             # check support level of property
             prop_type = _impl_properties[prop]["type"]
@@ -429,7 +421,9 @@ class ImplementationValidator:
         return True, f"Found all required properties in entry info for endpoint {endp}"
 
     @test_case
-    def _get_archetypal_entry(self, endp: str) -> Tuple[Dict[str, Any], str]:
+    def _get_archetypal_entry(
+        self, endp: str, properties: List[str]
+    ) -> Tuple[Dict[str, Any], str]:
         """Get a random entry from the first page of results for this
         endpoint.
 
@@ -444,9 +438,13 @@ class ImplementationValidator:
         response, _ = self._get_endpoint(endp)
         data_returned = response.json()["meta"]["data_returned"]
         if data_returned < 1:
-            raise ResponseError(f"No data returned from endpoint {endp}.")
+            return (
+                None,
+                "Endpoint {endp!r} has no entries, cannot get archetypal entry.",
+            )
+
         response, _ = self._get_endpoint(
-            f"{endp}?page_offset={random.randint(0, data_returned-1)}"
+            f"{endp}?page_offset={random.randint(0, data_returned-1)}&response_fields={','.join(properties)}"
         )
         archetypal_entry = response.json()["data"][0]
         return (
@@ -454,6 +452,43 @@ class ImplementationValidator:
             f"set archetypal entry for {endp} with ID {archetypal_entry['id']}.",
         )
 
+    @test_case
+    def _check_response_fields(self, endp: str, fields: List[str]) -> Tuple[bool, str]:
+        """Check that the response field query parameter is obeyed.
+
+        Parameters:
+            endp: The endpoint to query.
+            fields: The known fields for this endpoint to test.
+
+        Returns:
+            Bool indicating success and a summary message.
+
+        """
+
+        subset_fields = random.sample(fields, min(len(fields) - 1, 3))
+        test_query = f"{endp}?response_fields={','.join(subset_fields)}&page_limit=1"
+        response, _ = self._get_endpoint(test_query)
+
+        if response and len(response.json()["data"]) == 1:
+            doc = response.json()["data"][0]
+            expected_fields = set(subset_fields)
+            expected_fields -= CONF.top_level_non_attribute_fields
+            returned_fields = set(sorted(list(doc["attributes"].keys())))
+            returned_fields -= CONF.top_level_non_attribute_fields
+
+            if expected_fields != returned_fields:
+                raise RuntimeError(
+                    f"Response fields not obeyed by {endp!r}:\n{expected_fields}\n{returned_fields}"
+                )
+
+            return True, "Successfully limited response fields"
+
+        else:
+            return (
+                None,
+                f"Unable to test adherence to response fields as no entries were returned for endpoint {endp!r}.",
+            )
+
     @test_case
     def _construct_queries_for_property(
         self,
@@ -482,11 +517,16 @@ class ImplementationValidator:
 
         """
         # Explicitly handle top level keys that do not have types in info
+        if not chosen_entry:
+            raise ResponseError(
+                f"Chosen entry of endpoint '/{endp}' failed validation."
+            )
+
         if prop == "type":
             if chosen_entry["type"] == endp:
                 return True, f"Successfully validated {prop}"
             raise ResponseError(
-                f"Chosen entry of endpoint '/{endp}' had unexpected type '{chosen_entry['type']}'."
+                f"Chosen entry of endpoint '{endp}' had unexpected type {chosen_entry['type']!r}."
             )
 
         prop_type = (
@@ -511,7 +551,7 @@ class ImplementationValidator:
                     f"Found unknown field {prop!r} that did not start with provider prefix '_{self.provider_prefix}_'"
                 )
             return (
-                True,
+                None,
                 f"Found provider field {prop!r}, will not test queries as they are strictly optional.",
             )
 
@@ -594,17 +634,25 @@ class ImplementationValidator:
         if prop == "id":
             test_value = chosen_entry["id"]
         else:
-            test_value = chosen_entry["attributes"].get(prop, "missing")
+            test_value = chosen_entry["attributes"].get(prop, "_missing")
 
-        if test_value in ("missing", None):
+        if test_value in ("_missing", None):
             support = CONF.entry_schemas[endp].get(prop, {}).get("support")
             queryable = CONF.entry_schemas[endp].get(prop, {}).get("queryable")
+            submsg = "had no value" if test_value == "_missing" else "had `None` value"
             msg = (
-                f"Chosen entry had no value for {prop!r} with support level {support} and queryability {queryable}, "
-                "so cannot construct test queries. This field should potentially be removed from the `/info/{endp}` endpoint response."
+                f"Chosen entry {submsg} for {prop!r} with support level {support} and queryability {queryable}, "
+                f"so cannot construct test queries. This field should potentially be removed from the `/info/{endp}` endpoint response."
             )
-            if support == SupportLevel.OPTIONAL:
+            # None values are allowed for OPTIONAL and SHOULD, so we can just skip
+            if support in (
+                SupportLevel.OPTIONAL,
+                SupportLevel.SHOULD,
+            ):
+                self._log.info(msg)
                 return None, msg
+
+            # Otherwise, None values are not allowed for MUST's, and entire missing fields are not allowed
             raise ResponseError(msg)
 
         if prop_type == DataType.LIST:
@@ -631,11 +679,23 @@ class ImplementationValidator:
 
             query = f"{prop} {operator} {_test_value}"
             response, _ = self._get_endpoint(
-                f"{endp}?filter={query}", multistage=True, optional=query_optional
+                f"{endp}?filter={query}",
+                multistage=True,
+                optional=query_optional,
+                expected_status_code=(200, 501),
             )
             if not response:
                 return None, ""
 
+            if response.status_code == 501:
+                self._log.warning(
+                    f"Implementation returned {response.content} for {query}"
+                )
+                return (
+                    True,
+                    "Implementation safely reported that filter {query} was not implemented.",
+                )
+
             response = response.json()
             num_data_returned[operator] = response["meta"]["data_returned"]
 
@@ -739,7 +799,7 @@ class ImplementationValidator:
 
         return False
 
-    def _test_single_entry_endpoint(self, request_str: str) -> None:
+    def _test_single_entry_endpoint(self, endp: str) -> None:
         """Requests and deserializes a single entry endpoint with the
         appropriate model.
 
@@ -747,23 +807,32 @@ class ImplementationValidator:
             request_str: The single entry request to make, e.g. "structures/id_1".
 
         """
-        _type = request_str.split("?")[0]
-        response_cls_name = _type + "/"
+        response_cls_name = endp + "/"
         if response_cls_name in self._response_classes:
             response_cls = self._response_classes[response_cls_name]
         else:
             self._log.warning(
                 "Deserializing single entry response %s with generic response rather than defined endpoint.",
-                _type,
+                endp,
             )
             response_cls = ValidatorEntryResponseOne
-        if _type in self._test_id_by_type:
-            test_id = self._test_id_by_type[_type]
-            response, _ = self._get_endpoint(f"{_type}/{test_id}")
+
+        if endp in CONF.entry_schemas:
+            response_fields = (
+                set(CONF.entry_schemas[endp].keys())
+                - CONF.top_level_non_attribute_fields
+            )
+
+        if endp in self._test_id_by_type:
+            test_id = self._test_id_by_type[endp]
+            request_str = f"{endp}/{test_id}"
+            if response_fields:
+                request_str += f"?response_fields={','.join(response_fields)}"
+            response, _ = self._get_endpoint(request_str)
             if response:
                 self._deserialize_response(response, response_cls, request=request_str)
 
-    def _test_multi_entry_endpoint(self, request_str: str) -> None:
+    def _test_multi_entry_endpoint(self, endp: str) -> None:
         """Requests and deserializes a multi-entry endpoint with the
         appropriate model.
 
@@ -775,17 +844,28 @@ class ImplementationValidator:
                 "structures?filter=nsites<10"
 
         """
-        response, _ = self._get_endpoint(request_str)
-        _type = request_str.split("?")[0]
-        if _type in self._response_classes:
-            response_cls = self._response_classes[_type]
+        if endp in self._response_classes:
+            response_cls = self._response_classes[endp]
         else:
             self._log.warning(
                 "Deserializing multi entry response from %s with generic response rather than defined endpoint.",
-                _type,
+                endp,
             )
             response_cls = ValidatorEntryResponseMany
 
+        if endp in CONF.entry_schemas:
+            response_fields = (
+                set(CONF.entry_schemas[endp].keys())
+                - CONF.top_level_non_attribute_fields
+            )
+
+        request_str = f"{endp}?page_limit={self.page_limit}"
+
+        if response_fields:
+            request_str += f'&response_fields={",".join(response_fields)}'
+
+        response, _ = self._get_endpoint(request_str)
+
         self._test_page_limit(response)
 
         deserialized, _ = self._deserialize_response(
@@ -1111,7 +1191,7 @@ class ImplementationValidator:
 
     @test_case
     def _get_endpoint(
-        self, request_str: str, expected_status_code: int = 200
+        self, request_str: str, expected_status_code: Union[List[int], int] = 200
     ) -> Tuple[Optional[requests.Response], str]:
         """Gets the response from the endpoint specified by `request_str`.
         function is wrapped by the `test_case` decorator
@@ -1129,7 +1209,10 @@ class ImplementationValidator:
         request_str = request_str.replace("\n", "")
         response = self.client.get(request_str)
 
-        if response.status_code != expected_status_code:
+        if isinstance(expected_status_code, int):
+            expected_status_code = [expected_status_code]
+
+        if response.status_code not in expected_status_code:
             message = f"Request to '{request_str}' returned HTTP code {response.status_code} and not expected {expected_status_code}."
             message += "\nError(s):"
             for error in response.json().get("errors", []):


Test patch:
diff --git a/tests/validator/test_utils.py b/tests/validator/test_utils.py
index afce84f0..5b69199e 100644
--- a/tests/validator/test_utils.py
+++ b/tests/validator/test_utils.py
@@ -121,9 +121,9 @@ def test_expected_failure_test_case():
 
     assert (
         validator.results.failure_messages[-1][0]
-        == "✖: http://example.org/test_request - dummy_test_case - failed with error"
+        == "http://example.org/test_request - dummy_test_case - failed with error"
     )
-    assert validator.results.failure_messages[-1][1] == ["ResponseError: Dummy error"]
+    assert validator.results.failure_messages[-1][1] == "ResponseError: Dummy error"
 
     output = dummy_test_case(
         validator,
@@ -143,11 +143,12 @@ def test_expected_failure_test_case():
 
     assert (
         validator.results.optional_failure_messages[-1][0]
-        == "✖: http://example.org/test_request - dummy_test_case - failed with error"
+        == "http://example.org/test_request - dummy_test_case - failed with error"
+    )
+    assert (
+        validator.results.optional_failure_messages[-1][1]
+        == "ResponseError: Dummy error"
     )
-    assert validator.results.optional_failure_messages[-1][1] == [
-        "ResponseError: Dummy error"
-    ]
 
     output = dummy_test_case(
         validator,
@@ -169,11 +170,12 @@ def test_expected_failure_test_case():
     )
     assert (
         validator.results.optional_failure_messages[-1][0]
-        == "✖: http://example.org/test_request - dummy_test_case - failed with error"
+        == "http://example.org/test_request - dummy_test_case - failed with error"
+    )
+    assert (
+        validator.results.optional_failure_messages[-1][1]
+        == "Critical: unable to parse server response as JSON. JSONDecodeError: Dummy JSON error: line 1 column 1 (char 0)"
     )
-    assert validator.results.optional_failure_messages[-1][1] == [
-        "Critical: unable to parse server response as JSON. JSONDecodeError: Dummy JSON error: line 1 column 1 (char 0)"
-    ]
 
 
 def test_unexpected_failure_test_case():
@@ -197,11 +199,12 @@ def test_unexpected_failure_test_case():
     assert output[1] == "FileNotFoundError: Unexpected error"
     assert (
         validator.results.internal_failure_messages[-1][0]
-        == "!: http://example.org/test_request - dummy_test_case - failed with internal error"
+        == "http://example.org/test_request - dummy_test_case - failed with internal error"
+    )
+    assert (
+        validator.results.internal_failure_messages[-1][1]
+        == "FileNotFoundError: Unexpected error"
     )
-    assert validator.results.internal_failure_messages[-1][1] == [
-        "FileNotFoundError: Unexpected error"
-    ]
 
     output = dummy_test_case(
         validator,
@@ -220,11 +223,12 @@ def test_unexpected_failure_test_case():
     assert output[1] == "FileNotFoundError: Unexpected error"
     assert (
         validator.results.internal_failure_messages[-1][0]
-        == "!: http://example.org/test_request - dummy_test_case - failed with internal error"
+        == "http://example.org/test_request - dummy_test_case - failed with internal error"
+    )
+    assert (
+        validator.results.internal_failure_messages[-1][1]
+        == "FileNotFoundError: Unexpected error"
     )
-    assert validator.results.internal_failure_messages[-1][1] == [
-        "FileNotFoundError: Unexpected error"
-    ]
 
 
 def test_multistage_test_case():
@@ -264,11 +268,12 @@ def test_multistage_test_case():
     assert output[1] == "ResponseError: Stage of test failed"
     assert (
         validator.results.failure_messages[-1][0]
-        == "✖: http://example.org/test_request - dummy_test_case - failed with error"
+        == "http://example.org/test_request - dummy_test_case - failed with error"
+    )
+    assert (
+        validator.results.failure_messages[-1][1]
+        == "ResponseError: Stage of test failed"
     )
-    assert validator.results.failure_messages[-1][1] == [
-        "ResponseError: Stage of test failed"
-    ]
 
 
 def test_fail_fast_test_case():
@@ -294,11 +299,12 @@ def test_fail_fast_test_case():
     assert output[0] is None
     assert output[1] == "ResponseError: Optional test failed"
     assert validator.results.optional_failure_messages[-1][0] == (
-        "✖: http://example.org/test_request - dummy_test_case - failed with error"
+        "http://example.org/test_request - dummy_test_case - failed with error"
+    )
+    assert (
+        validator.results.optional_failure_messages[-1][1]
+        == "ResponseError: Optional test failed"
     )
-    assert validator.results.optional_failure_messages[-1][1] == [
-        "ResponseError: Optional test failed"
-    ]
 
     # Check that the same non-optional failures do trigger fail fast
     with pytest.raises(SystemExit):
@@ -315,11 +321,12 @@ def test_fail_fast_test_case():
     assert validator.results.optional_failure_count == 1
     assert validator.results.internal_failure_count == 0
     assert validator.results.failure_messages[-1][0] == (
-        "✖: http://example.org/test_request - dummy_test_case - failed with error"
+        "http://example.org/test_request - dummy_test_case - failed with error"
+    )
+    assert (
+        validator.results.failure_messages[-1][1]
+        == "ResponseError: Non-optional test failed"
     )
-    assert validator.results.failure_messages[-1][1] == [
-        "ResponseError: Non-optional test failed"
-    ]
 
     # Check that an internal error also triggers fast
     with pytest.raises(SystemExit):
@@ -335,11 +342,12 @@ def test_fail_fast_test_case():
     assert validator.results.optional_failure_count == 1
     assert validator.results.internal_failure_count == 1
     assert validator.results.internal_failure_messages[-1][0] == (
-        "!: http://example.org/test_request - dummy_test_case - failed with internal error"
+        "http://example.org/test_request - dummy_test_case - failed with internal error"
+    )
+    assert (
+        validator.results.internal_failure_messages[-1][1]
+        == "FileNotFoundError: Internal error"
     )
-    assert validator.results.internal_failure_messages[-1][1] == [
-        "FileNotFoundError: Internal error"
-    ]
 
 
 def test_that_system_exit_is_fatal_in_test_case():