Repository: Materials-Consortia/optimade-python-tools

Issue:
When using `--as-type` in validator, one does not get a summary (`--json` doesn't work)
It's an issue that using `--as-type` for the validator doesn't properly print the summary, which would ensure the `--json` option can be used.

Discussion:
codecov[bot]: # [Codecov](https://codecov.io/gh/Materials-Consortia/optimade-python-tools/pull/700?src=pr&el=h1) Report
> Merging [#700](https://codecov.io/gh/Materials-Consortia/optimade-python-tools/pull/700?src=pr&el=desc) (d4a6215) into [master](https://codecov.io/gh/Materials-Consortia/optimade-python-tools/commit/f9d3b8a8fdf5e438f2b7c746bcdefb661de9c86c?el=desc) (f9d3b8a) will **increase** coverage by `0.00%`.
> The diff coverage is `50.00%`.

[![Impacted file tree graph](https://codecov.io/gh/Materials-Consortia/optimade-python-tools/pull/700/graphs/tree.svg?width=650&height=150&src=pr&token=UJAtmqkZZO)](https://codecov.io/gh/Materials-Consortia/optimade-python-tools/pull/700?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master     #700   +/-   ##
=======================================
  Coverage   93.18%   93.18%           
=======================================
  Files          61       61           
  Lines        3361     3362    +1     
=======================================
+ Hits         3132     3133    +1     
  Misses        229      229           
```

| Flag | Coverage Δ | |
|---|---|---|
| project | `93.18% <50.00%> (+<0.01%)` | :arrow_up: |
| validator | `65.40% <50.00%> (-0.74%)` | :arrow_down: |

Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.

| [Impacted Files](https://codecov.io/gh/Materials-Consortia/optimade-python-tools/pull/700?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [optimade/validator/validator.py](https://codecov.io/gh/Materials-Consortia/optimade-python-tools/pull/700/diff?src=pr&el=tree#diff-b3B0aW1hZGUvdmFsaWRhdG9yL3ZhbGlkYXRvci5weQ==) | `82.31% <50.00%> (+0.03%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/Materials-Consortia/optimade-python-tools/pull/700?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/Materials-Consortia/optimade-python-tools/pull/700?src=pr&el=footer). Last update [f9d3b8a...d4a6215](https://codecov.io/gh/Materials-Consortia/optimade-python-tools/pull/700?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

CasperWA: > Thanks for this @CasperWA, could you add a check to the end of [tests.test_server_validation.test_as_type_with_validator](https://github.com/Materials-Consortia/optimade-python-tools/blob/d4a62151ca608abac683fb71592cb6200cbc7ff0/tests/server/test_server_validation.py#L67)?

I did something. I don't know if it's exactly what you had in mind?

Patch:
diff --git a/optimade/validator/validator.py b/optimade/validator/validator.py
index 959297e2..ff974674 100644
--- a/optimade/validator/validator.py
+++ b/optimade/validator/validator.py
@@ -252,6 +252,7 @@ class ImplementationValidator:
             )
             self._test_as_type()
             self.valid = not bool(self.results.failure_count)
+            self.print_summary()
             return
 
         # Test entire implementation
@@ -268,9 +269,9 @@ class ImplementationValidator:
                 f"Unable to deserialize response from introspective {info_endp!r} endpoint. "
                 "This is required for all further validation, so the validator will now exit."
             )
-            self.print_summary()
             # Set valid to False to ensure error code 1 is raised at CLI
             self.valid = False
+            self.print_summary()
             return
 
         # Grab the provider prefix from base info and use it when looking for provider fields


Test patch:
diff --git a/tests/server/test_server_validation.py b/tests/server/test_server_validation.py
index 1548052c..2a06d0b9 100644
--- a/tests/server/test_server_validation.py
+++ b/tests/server/test_server_validation.py
@@ -1,7 +1,6 @@
 import os
 import json
 import dataclasses
-from traceback import print_exc
 
 import pytest
 
@@ -64,7 +63,7 @@ def test_mongo_backend_package_used():
         )
 
 
-def test_as_type_with_validator(client):
+def test_as_type_with_validator(client, capsys):
     import unittest
 
     test_urls = {
@@ -80,14 +79,22 @@ def test_as_type_with_validator(client):
     ):
         for url, as_type in test_urls.items():
             validator = ImplementationValidator(
-                base_url=url, as_type=as_type, verbosity=5
+                base_url=url, as_type=as_type, respond_json=True
             )
-            try:
-                validator.validate_implementation()
-            except Exception:
-                print_exc()
+            validator.validate_implementation()
             assert validator.valid
 
+            captured = capsys.readouterr()
+            json_response = json.loads(captured.out)
+
+            assert json_response["failure_count"] == 0
+            assert json_response["internal_failure_count"] == 0
+            assert json_response["optional_failure_count"] == 0
+            assert validator.results.failure_count == 0
+            assert validator.results.internal_failure_count == 0
+            assert validator.results.optional_failure_count == 0
+            assert dataclasses.asdict(validator.results) == json_response
+
 
 def test_query_value_formatting(client):
     from optimade.models.optimade_json import DataType