Repository: Materials-Consortia/optimade-python-tools
Pull request #511 (2020-09-21 11:04:40)

Move entry schemas to separate submodule
This PR is a shortcut version of #277 whilst retaining most of its features. The `ENTRY_INFO_SCHEMA` constant dict has been moved to a new submodule and new extension fields ("support" and "queryable") are now made available so they can also be used by the validator more cleanly in #503, and eventually in the server code in #504. 

All that is missing to enable #277 is to wrap this in a class to provide a nicer interface for implementations to use their own models in `ENTRY_INFO_SCHEMAS` rather than the hardcoded `StructureResource` etc. I think this is a less important point than the other bits though.

codecov[bot]: # [Codecov](https://codecov.io/gh/Materials-Consortia/optimade-python-tools/pull/511?src=pr&el=h1) Report
> Merging [#511](https://codecov.io/gh/Materials-Consortia/optimade-python-tools/pull/511?src=pr&el=desc) into [master](https://codecov.io/gh/Materials-Consortia/optimade-python-tools/commit/41c26e5a7b8297aaffe9ae83112a17a0af01d3d4?el=desc) will **increase** coverage by `0.00%`.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/Materials-Consortia/optimade-python-tools/pull/511/graphs/tree.svg?width=650&height=150&src=pr&token=UJAtmqkZZO)](https://codecov.io/gh/Materials-Consortia/optimade-python-tools/pull/511?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master     #511   +/-   ##
=======================================
  Coverage   91.59%   91.59%           
=======================================
  Files          61       62    +1     
  Lines        3118     3119    +1     
=======================================
+ Hits         2856     2857    +1     
  Misses        262      262           
```

| Flag | Coverage Δ | |
|---|---|---|
| #project | `91.59% <100.00%> (+<0.01%)` | :arrow_up: |
| #validator | `63.73% <100.00%> (+0.01%)` | :arrow_up: |

Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.

| [Impacted Files](https://codecov.io/gh/Materials-Consortia/optimade-python-tools/pull/511?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [optimade/server/routers/utils.py](https://codecov.io/gh/Materials-Consortia/optimade-python-tools/pull/511/diff?src=pr&el=tree#diff-b3B0aW1hZGUvc2VydmVyL3JvdXRlcnMvdXRpbHMucHk=) | `97.72% <ø> (-0.29%)` | :arrow_down: |
| [optimade/server/routers/info.py](https://codecov.io/gh/Materials-Consortia/optimade-python-tools/pull/511/diff?src=pr&el=tree#diff-b3B0aW1hZGUvc2VydmVyL3JvdXRlcnMvaW5mby5weQ==) | `96.15% <100.00%> (ø)` | |
| [optimade/server/schemas.py](https://codecov.io/gh/Materials-Consortia/optimade-python-tools/pull/511/diff?src=pr&el=tree#diff-b3B0aW1hZGUvc2VydmVyL3NjaGVtYXMucHk=) | `100.00% <100.00%> (ø)` | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/Materials-Consortia/optimade-python-tools/pull/511?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/Materials-Consortia/optimade-python-tools/pull/511?src=pr&el=footer). Last update [41c26e5...eddaf86](https://codecov.io/gh/Materials-Consortia/optimade-python-tools/pull/511?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

ml-evs: > On another note, are you sure we don't need to take `allOf`, `anyOf`, etc. into account in the recursive function?

I'm sure _for now_... I think the problem we were having was also providing a schema for the links endpoint, which was running into the multiple allowed response types. We don't have that anywhere in our structure/reference models so this should be useful as is.



Patch:
diff --git a/docs/api_reference/server/schemas.md b/docs/api_reference/server/schemas.md
new file mode 100644
index 00000000..7bdb6e54
--- /dev/null
+++ b/docs/api_reference/server/schemas.md
@@ -0,0 +1,3 @@
+# schemas
+
+::: optimade.server.schemas
diff --git a/optimade/server/routers/info.py b/optimade/server/routers/info.py
index e138238f..f96aadfb 100644
--- a/optimade/server/routers/info.py
+++ b/optimade/server/routers/info.py
@@ -11,24 +11,17 @@ from optimade.models import (
     ErrorResponse,
     InfoResponse,
     EntryInfoResponse,
-    ReferenceResource,
-    StructureResource,
 )
+from optimade.server.schemas import ENTRY_INFO_SCHEMAS, retrieve_queryable_properties
 
 from optimade.server.routers.utils import (
     meta_values,
-    retrieve_queryable_properties,
     get_base_url,
 )
 
 
 router = APIRouter(redirect_slashes=True)
 
-ENTRY_INFO_SCHEMAS = {
-    "structures": StructureResource.schema,
-    "references": ReferenceResource.schema,
-}
-
 
 @router.get(
     "/info",
diff --git a/optimade/server/routers/utils.py b/optimade/server/routers/utils.py
index 06a7a70f..95c9649b 100644
--- a/optimade/server/routers/utils.py
+++ b/optimade/server/routers/utils.py
@@ -14,9 +14,6 @@ from optimade.models import (
     EntryResponseMany,
     EntryResponseOne,
     ToplevelLinks,
-    ReferenceResource,
-    StructureResource,
-    DataType,
 )
 
 from optimade.server.config import CONFIG
@@ -24,11 +21,6 @@ from optimade.server.entry_collections import EntryCollection
 from optimade.server.exceptions import BadRequest
 from optimade.server.query_params import EntryListingQueryParams, SingleEntryQueryParams
 
-ENTRY_INFO_SCHEMAS = {
-    "structures": StructureResource.schema,
-    "references": ReferenceResource.schema,
-}
-
 # we need to get rid of any release tags (e.g. -rc.2) and any build metadata (e.g. +py36)
 # from the api_version before allowing the URL
 BASE_URL_PREFIXES = {
@@ -271,34 +263,6 @@ def get_single_entry(
     )
 
 
-def retrieve_queryable_properties(schema: dict, queryable_properties: list) -> dict:
-    properties = {}
-    for name, value in schema["properties"].items():
-        if name in queryable_properties:
-            if "$ref" in value:
-                path = value["$ref"].split("/")[1:]
-                sub_schema = schema.copy()
-                while path:
-                    next_key = path.pop(0)
-                    sub_schema = sub_schema[next_key]
-                sub_queryable_properties = sub_schema["properties"].keys()
-                properties.update(
-                    retrieve_queryable_properties(sub_schema, sub_queryable_properties)
-                )
-            else:
-                properties[name] = {"description": value.get("description", "")}
-                if "unit" in value:
-                    properties[name]["unit"] = value["unit"]
-                # All properties are sortable with the MongoDB backend.
-                # While the result for sorting lists may not be as expected, they are still sorted.
-                properties[name]["sortable"] = True
-                # Try to get OpenAPI-specific "format" if possible, else get "type"; a mandatory OpenAPI key.
-                properties[name]["type"] = DataType.from_json_type(
-                    value.get("format", value["type"])
-                )
-    return properties
-
-
 def mongo_id_for_database(database_id: str, database_type: str) -> str:
     """Produce a MondoDB ObjectId for a database"""
     from bson.objectid import ObjectId
diff --git a/optimade/server/schemas.py b/optimade/server/schemas.py
new file mode 100644
index 00000000..d059a815
--- /dev/null
+++ b/optimade/server/schemas.py
@@ -0,0 +1,50 @@
+from optimade.models import DataType, StructureResource, ReferenceResource
+
+ENTRY_INFO_SCHEMAS = {
+    "structures": StructureResource.schema,
+    "references": ReferenceResource.schema,
+}
+
+
+def retrieve_queryable_properties(schema: dict, queryable_properties: list) -> dict:
+    """Recurisvely loops through the schema of a pydantic model and
+    resolves all references, returning a dictionary of all the
+    OPTIMADE-queryable properties of that model.
+
+    Parameters:
+        schema: The schema of the pydantic model.
+        queryable_properties: The list of properties to find in the schema.
+
+    Returns:
+        A flat dictionary with properties as keys, containing the field
+        description, unit, sortability, support level, queryability
+        and type, where provided.
+
+    """
+    properties = {}
+    for name, value in schema["properties"].items():
+        if name in queryable_properties:
+            if "$ref" in value:
+                path = value["$ref"].split("/")[1:]
+                sub_schema = schema.copy()
+                while path:
+                    next_key = path.pop(0)
+                    sub_schema = sub_schema[next_key]
+                sub_queryable_properties = sub_schema["properties"].keys()
+                properties.update(
+                    retrieve_queryable_properties(sub_schema, sub_queryable_properties)
+                )
+            else:
+                properties[name] = {"description": value.get("description", "")}
+                # Update schema with extension keys provided they are not None
+                for key in [_ for _ in ("unit", "queryable", "support") if _ in value]:
+                    properties[name][key] = value[key]
+                # All properties are sortable with the MongoDB backend.
+                # While the result for sorting lists may not be as expected, they are still sorted.
+                properties[name]["sortable"] = value.get("sortable", True)
+                # Try to get OpenAPI-specific "format" if possible, else get "type"; a mandatory OpenAPI key.
+                properties[name]["type"] = DataType.from_json_type(
+                    value.get("format", value["type"])
+                )
+
+    return properties


Test patch:
diff --git a/tests/server/test_schemas.py b/tests/server/test_schemas.py
new file mode 100644
index 00000000..dfb5c980
--- /dev/null
+++ b/tests/server/test_schemas.py
@@ -0,0 +1,32 @@
+from optimade.server.schemas import ENTRY_INFO_SCHEMAS, retrieve_queryable_properties
+
+
+def test_schemas():
+    """Test that the default `ENTRY_INFO_SCHEMAS` contain
+    all the required information about the OPTIMADE properties
+    after dereferencing.
+
+    """
+    for entry in ("Structures", "References"):
+        schema = ENTRY_INFO_SCHEMAS[entry.lower()]()
+
+        top_level_props = ("id", "type", "attributes")
+        properties = retrieve_queryable_properties(schema, top_level_props)
+
+        fields = list(
+            schema["definitions"][f"{entry[:-1]}ResourceAttributes"][
+                "properties"
+            ].keys()
+        )
+        fields += ["id", "type"]
+
+        # Check all fields are present
+        assert all(field in properties for field in fields)
+
+        # Check that there are no references to definitions remaining
+        assert "$ref" not in properties
+        assert not any("$ref" in properties[field] for field in properties)
+
+        # Check that all expected keys are present for OPTIMADE fields
+        for key in ("type", "sortable", "queryable", "description"):
+            assert all(key in properties[field] for field in properties)