Repository: Materials-Consortia/optimade-python-tools

Issue:
Missing optional fields are not returned as null when requested with response_fields
According to the specification, if an optional field is asked for using the `response_fields` query parameter, it must be returned as `null` if missing from the document. We currently don't do this (see e.g. `immutable_id` in the test server).

This is also related to the problem of returning errors when unknown fields are requested (PR: #504 Issue: #263)

Discussion:
codecov[bot]: # [Codecov](https://codecov.io/gh/Materials-Consortia/optimade-python-tools/pull/797?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Materials-Consortia) Report
> Merging [#797](https://codecov.io/gh/Materials-Consortia/optimade-python-tools/pull/797?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Materials-Consortia) (6d13f3e) into [master](https://codecov.io/gh/Materials-Consortia/optimade-python-tools/commit/37e3c9739faf17889eba1f79e5e97aa8cba97b60?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Materials-Consortia) (37e3c97) will **decrease** coverage by `0.24%`.
> The diff coverage is `90.55%`.

[![Impacted file tree graph](https://codecov.io/gh/Materials-Consortia/optimade-python-tools/pull/797/graphs/tree.svg?width=650&height=150&src=pr&token=UJAtmqkZZO&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Materials-Consortia)](https://codecov.io/gh/Materials-Consortia/optimade-python-tools/pull/797?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Materials-Consortia)

```diff
@@            Coverage Diff             @@
##           master     #797      +/-   ##
==========================================
- Coverage   92.71%   92.46%   -0.25%     
==========================================
  Files          68       68              
  Lines        3666     3691      +25     
==========================================
+ Hits         3399     3413      +14     
- Misses        267      278      +11     
```

| Flag | Coverage Δ | |
|---|---|---|
| project | `92.46% <90.55%> (-0.25%)` | :arrow_down: |
| validator | `92.46% <90.55%> (-0.25%)` | :arrow_down: |

Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Materials-Consortia#carryforward-flags-in-the-pull-request-comment) to find out more.

| [Impacted Files](https://codecov.io/gh/Materials-Consortia/optimade-python-tools/pull/797?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Materials-Consortia) | Coverage Δ | |
|---|---|---|
| [optimade/filtertransformers/base\_transformer.py](https://codecov.io/gh/Materials-Consortia/optimade-python-tools/pull/797/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Materials-Consortia#diff-b3B0aW1hZGUvZmlsdGVydHJhbnNmb3JtZXJzL2Jhc2VfdHJhbnNmb3JtZXIucHk=) | `91.45% <84.74%> (-7.29%)` | :arrow_down: |
| [optimade/filtertransformers/elasticsearch.py](https://codecov.io/gh/Materials-Consortia/optimade-python-tools/pull/797/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Materials-Consortia#diff-b3B0aW1hZGUvZmlsdGVydHJhbnNmb3JtZXJzL2VsYXN0aWNzZWFyY2gucHk=) | `82.95% <90.00%> (-0.83%)` | :arrow_down: |
| [optimade/filtertransformers/mongo.py](https://codecov.io/gh/Materials-Consortia/optimade-python-tools/pull/797/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Materials-Consortia#diff-b3B0aW1hZGUvZmlsdGVydHJhbnNmb3JtZXJzL21vbmdvLnB5) | `96.95% <94.73%> (-0.62%)` | :arrow_down: |
| [optimade/filtertransformers/\_\_init\_\_.py](https://codecov.io/gh/Materials-Consortia/optimade-python-tools/pull/797/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Materials-Consortia#diff-b3B0aW1hZGUvZmlsdGVydHJhbnNmb3JtZXJzL19faW5pdF9fLnB5) | `100.00% <100.00%> (ø)` | |
| [optimade/server/entry\_collections/elasticsearch.py](https://codecov.io/gh/Materials-Consortia/optimade-python-tools/pull/797/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Materials-Consortia#diff-b3B0aW1hZGUvc2VydmVyL2VudHJ5X2NvbGxlY3Rpb25zL2VsYXN0aWNzZWFyY2gucHk=) | `97.59% <100.00%> (-0.33%)` | :arrow_down: |
| [optimade/server/mappers/entries.py](https://codecov.io/gh/Materials-Consortia/optimade-python-tools/pull/797/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Materials-Consortia#diff-b3B0aW1hZGUvc2VydmVyL21hcHBlcnMvZW50cmllcy5weQ==) | `98.80% <100.00%> (+0.28%)` | :arrow_up: |
| [optimade/server/mappers/references.py](https://codecov.io/gh/Materials-Consortia/optimade-python-tools/pull/797/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Materials-Consortia#diff-b3B0aW1hZGUvc2VydmVyL21hcHBlcnMvcmVmZXJlbmNlcy5weQ==) | `100.00% <100.00%> (ø)` | |
| [optimade/server/mappers/structures.py](https://codecov.io/gh/Materials-Consortia/optimade-python-tools/pull/797/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Materials-Consortia#diff-b3B0aW1hZGUvc2VydmVyL21hcHBlcnMvc3RydWN0dXJlcy5weQ==) | `100.00% <100.00%> (ø)` | |
| [optimade/server/schemas.py](https://codecov.io/gh/Materials-Consortia/optimade-python-tools/pull/797/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Materials-Consortia#diff-b3B0aW1hZGUvc2VydmVyL3NjaGVtYXMucHk=) | `100.00% <100.00%> (ø)` | |
| [optimade/server/exception\_handlers.py](https://codecov.io/gh/Materials-Consortia/optimade-python-tools/pull/797/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Materials-Consortia#diff-b3B0aW1hZGUvc2VydmVyL2V4Y2VwdGlvbl9oYW5kbGVycy5weQ==) | `83.09% <0.00%> (-1.41%)` | :arrow_down: |
| ... and [1 more](https://codecov.io/gh/Materials-Consortia/optimade-python-tools/pull/797/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Materials-Consortia) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/Materials-Consortia/optimade-python-tools/pull/797?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Materials-Consortia).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Materials-Consortia)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/Materials-Consortia/optimade-python-tools/pull/797?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Materials-Consortia). Last update [37e3c97...6d13f3e](https://codecov.io/gh/Materials-Consortia/optimade-python-tools/pull/797?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Materials-Consortia). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Materials-Consortia).

ml-evs: >     * [x]  Handle unknown `response_fields`      
>       * Return any requested, missing field as `None` if present in `response_fields`
>       * Raise an error if an unknown, unprefixed field is requested in `response_fields` (e.g. `fractional_coordinates`)
>       * Emit a warning if a provider-specific field is requested that does not exist (e.g. `_exmpl_test`)
>       * Passthrough other-provider-specific fields as if they were missing from the entry (i.e. replace them with `null`)

@CasperWA: this might be an over-interpretation of the spec. Looks like other implementations just return `null` for any requested response field, but above I have described (and implemented) something that more closely follows the recommendations on filters.

ml-evs: I think that knocks everything off @CasperWA... thanks for the review as always. Will rebase all code review changes into a big commit at the end, once we're happy...
ml-evs: Hi @JPBergsma, I have rebased your final suggestions into one commit eb69f62. Thanks again for the thorough review, it would be great to merge this with enough time to prepare tutorials for the workshop!

Patch:
diff --git a/.github/workflows/deps_lint.yml b/.github/workflows/deps_lint.yml
index 67ae8818..342f3417 100644
--- a/.github/workflows/deps_lint.yml
+++ b/.github/workflows/deps_lint.yml
@@ -276,7 +276,7 @@ jobs:
     - name: Build
       run: |
         invoke create-api-reference-docs
-        mkdocs build --strict --verbose
+        mkdocs build --strict
 
   test_build:
     runs-on: ubuntu-latest
diff --git a/optimade/filtertransformers/__init__.py b/optimade/filtertransformers/__init__.py
index f671e475..2e9a99f7 100644
--- a/optimade/filtertransformers/__init__.py
+++ b/optimade/filtertransformers/__init__.py
@@ -4,6 +4,9 @@ given backend.
 
 """
 
-from optimade.filtertransformers.base_transformer import BaseTransformer
+from optimade.filtertransformers.base_transformer import BaseTransformer, Quantity
 
-__all__ = ("BaseTransformer",)
+__all__ = (
+    "BaseTransformer",
+    "Quantity",
+)
diff --git a/optimade/filtertransformers/base_transformer.py b/optimade/filtertransformers/base_transformer.py
index ee02edb0..f9d26087 100644
--- a/optimade/filtertransformers/base_transformer.py
+++ b/optimade/filtertransformers/base_transformer.py
@@ -1,18 +1,86 @@
+"""This submodule implements the
+[`BaseTransformer`][optimade.filtertransformers.base_transformer.BaseTransformer]
+and [`Quantity`][optimade.filtertransformers.base_transformer.Quantity] classes
+for turning filters parsed by lark into backend-specific queries.
+
+"""
+
 import abc
-from lark import Transformer, v_args
-from typing import Dict
+from typing import Dict, Any, Type, Optional
+import warnings
+
+from lark import Transformer, v_args, Tree
+
 from optimade.server.mappers import BaseResourceMapper
+from optimade.server.exceptions import BadRequest
+from optimade.server.warnings import UnknownProviderProperty
+
+
+__all__ = (
+    "BaseTransformer",
+    "Quantity",
+)
+
+
+class Quantity:
+    """Class to provide information about available quantities to the transformer.
+
+    The transformer can use [`Quantity`][optimade.filtertransformers.base_transformer.Quantity]'s to
 
-__all__ = ("BaseTransformer",)
+    * do some semantic checks,
+    * map quantities to the underlying backend field name.
+
+    Attributes:
+        name: The name of the quantity as used in the filter expressions.
+        backend_field: The name of the field for this quantity in the backend database, will be
+            `name` by default.
+        length_quantity: Another (typically integer) [`Quantity`][optimade.filtertransformers.base_transformer.Quantity]
+            that can be queried as the length of this quantity, e.g. `elements` and `nelements`. Backends
+            can then decide whether to use this for all "LENGTH" queries.
+
+    """
+
+    name: str
+    backend_field: Optional[str]
+    length_quantity: Optional["Quantity"]
+
+    def __init__(
+        self,
+        name: str,
+        backend_field: str = None,
+        length_quantity: "Quantity" = None,
+    ):
+        """Initialise the `quantity` from it's name and aliases.
+
+        Parameters:
+            name: The name of the quantity as used in the filter expressions.
+            backend_field: The name of the field for this quantity in the backend database, will be
+                `name` by default.
+            length_quantity: Another (typically integer) [`Quantity`][optimade.filtertransformers.base_transformer.Quantity]
+                that can be queried as the length of this quantity, e.g. `elements` and `nelements`. Backends
+                can then decide whether to use this for all "LENGTH" queries.
+
+        """
+
+        self.name = name
+        self.backend_field = backend_field if backend_field is not None else name
+        self.length_quantity = length_quantity
 
 
 class BaseTransformer(abc.ABC, Transformer):
     """Generic filter transformer that handles various
     parts of the grammar in a backend non-specific way.
 
+    Attributes:
+        operator_map: A map from comparison operators
+            to their backend-specific versions.
+        mapper: A resource mapper object that defines the
+            expected fields and acts as a container for
+            various field-related configuration.
+
     """
 
-    # map from standard comparison operators to the backend-specific version
+    mapper: Optional[BaseResourceMapper] = None
     operator_map: Dict[str, str] = {
         "<": None,
         "<=": None,
@@ -22,29 +90,101 @@ class BaseTransformer(abc.ABC, Transformer):
         "=": None,
     }
 
-    # map from back-end specific operators to their inverse
-    # e.g. {"$lt": "$gt"} for MongoDB.
-    reversed_operator_map: Dict[str, str] = {}
+    # map from operators to their syntactic (as opposed to logical) inverse to handle
+    # equivalence between cases like "A > 3" and "3 < A".
+    _reversed_operator_map = {
+        ">": "<",
+        ">=": "<=",
+        "<": ">",
+        "<=": ">=",
+        "=": "=",
+        "!=": "!=",
+    }
+
+    _quantity_type: Type[Quantity] = Quantity
+    _quantities = None
 
-    def __init__(self, mapper: BaseResourceMapper = None):
+    def __init__(
+        self, mapper: BaseResourceMapper = None
+    ):  # pylint: disable=super-init-not-called
         """Initialise the transformer object, optionally loading in a
         resource mapper for use when post-processing.
 
         """
         self.mapper = mapper
 
-    def postprocess(self, query):
+    @property
+    def backend_mapping(self) -> Dict[str, Quantity]:
+        """A mapping between backend field names (aliases) and the corresponding
+        [`Quantity`][optimade.filtertransformers.base_transformer.Quantity] object.
+        """
+        return {
+            quantity.backend_field: quantity for _, quantity in self.quantities.items()
+        }
+
+    @property
+    def quantities(self) -> Dict[str, Quantity]:
+        """A mapping from the OPTIMADE field name to the corresponding
+        [`Quantity`][optimade.filtertransformers.base_transformer.Quantity] objects.
+        """
+        if self._quantities is None:
+            self._quantities = self._build_quantities()
+
+        return self._quantities
+
+    @quantities.setter
+    def quantities(self, quantities: Dict[str, Quantity]) -> None:
+        self._quantities = quantities
+
+    def _build_quantities(self) -> Dict[str, Quantity]:
+        """Creates a dictionary of field names mapped to
+        [`Quantity`][optimade.filtertransformers.base_transformer.Quantity] objects from the
+        fields registered by the mapper.
+
+        """
+
+        quantities = {}
+
+        if self.mapper is not None:
+            for field in self.mapper.ALL_ATTRIBUTES:
+                alias = self.mapper.get_backend_field(field)
+                # Allow length aliases to be defined relative to either backend fields or OPTIMADE fields,
+                # with preference for those defined from OPTIMADE fields
+                length_alias = self.mapper.length_alias_for(
+                    field
+                ) or self.mapper.length_alias_for(alias)
+
+                if field not in quantities:
+                    quantities[field] = self._quantity_type(
+                        name=field, backend_field=alias
+                    )
+
+                if length_alias:
+                    if length_alias not in quantities:
+                        quantities[length_alias] = self._quantity_type(
+                            name=length_alias,
+                            backend_field=self.mapper.get_backend_field(length_alias),
+                        )
+                    quantities[field].length_quantity = quantities[length_alias]
+
+        return quantities
+
+    def postprocess(self, query) -> Any:
         """Post-process the query according to the rules defined for
-        the backend.
+        the backend, returning the backend-specific query.
 
         """
         return query
 
-    def transform(self, tree):
-        """Transform the query using the Lark transformer then post-process."""
+    def transform(self, tree: Tree) -> Any:
+        """Transform the query using the Lark `Transformer` then run the
+        backend-specific post-processing methods.
+
+        """
         return self.postprocess(super().transform(tree))
 
     def __default__(self, data, children, meta):
+        """The default rule to call when no definition is found for a particular construct."""
         raise NotImplementedError(
             f"Calling __default__, i.e., unknown grammar concept. data: {data}, children: {children}, meta: {meta}"
         )
@@ -82,9 +222,56 @@ class BaseTransformer(abc.ABC, Transformer):
         """
         raise NotImplementedError("Comparing strings is not yet implemented.")
 
-    def property(self, arg):
-        """property: IDENTIFIER ( "." IDENTIFIER )*"""
-        return ".".join(arg)
+    def property(self, args: list) -> Any:
+        """property: IDENTIFIER ( "." IDENTIFIER )*
+
+        If this transformer has an associated mapper, the property
+        will be compared to possible relationship entry types and
+        for any supported provider prefixes. If there is a match,
+        this rule will return a string and not a dereferenced
+        [`Quantity`][optimade.filtertransformers.base_transformer.Quantity].
+
+        Raises:
+            BadRequest: If the property does not match any
+                of the above rules.
+
+        """
+        quantity_name = str(args[0])
+
+        # If the quantity name matches an entry type (indicating a relationship filter)
+        # then simply return the quantity name; the inherited property
+        # must then handle any further nested identifiers
+        if self.mapper:
+            if quantity_name in self.mapper.RELATIONSHIP_ENTRY_TYPES:
+                return quantity_name
+
+        if self.quantities and quantity_name not in self.quantities:
+            # If the quantity is provider-specific, but does not match this provider,
+            # then return the quantity name such that it can be treated as unknown.
+            # If the prefix does not match another known provider, also emit a warning
+            # If the prefix does match a known provider, do not return a warning.
+            # Following [Handling unknown property names](https://github.com/Materials-Consortia/OPTIMADE/blob/master/optimade.rst#handling-unknown-property-names)
+            if self.mapper and quantity_name.startswith("_"):
+                prefix = quantity_name.split("_")[1]
+                if prefix not in self.mapper.SUPPORTED_PREFIXES:
+                    if prefix not in self.mapper.KNOWN_PROVIDER_PREFIXES:
+                        warnings.warn(
+                            UnknownProviderProperty(
+                                f"Field {quantity_name!r} has an unrecognised prefix: this property has been treated as UNKNOWN."
+                            )
+                        )
+
+                    return quantity_name
+
+            raise BadRequest(
+                detail=f"'{quantity_name}' is not a known or searchable quantity"
+            )
+
+        quantity = self.quantities.get(quantity_name, None)
+        if quantity is None:
+            quantity = self._quantity_type(name=str(quantity_name))
+
+        return quantity
 
     @v_args(inline=True)
     def string(self, string):
@@ -111,69 +298,62 @@ class BaseTransformer(abc.ABC, Transformer):
         # Note: Return as is.
         return value
 
-    @abc.abstractmethod
     def value_list(self, arg):
         """value_list: [ OPERATOR ] value ( "," [ OPERATOR ] value )*"""
 
-    @abc.abstractmethod
     def value_zip(self, arg):
         """value_zip: [ OPERATOR ] value ":" [ OPERATOR ] value (":" [ OPERATOR ] value)*"""
+        pass
 
-    @abc.abstractmethod
     def value_zip_list(self, arg):
         """value_zip_list: value_zip ( "," value_zip )*"""
 
-    @abc.abstractmethod
     def expression(self, arg):
         """expression: expression_clause ( OR expression_clause )"""
 
-    @abc.abstractmethod
     def expression_clause(self, arg):
         """expression_clause: expression_phrase ( AND expression_phrase )*"""
 
-    @abc.abstractmethod
     def expression_phrase(self, arg):
         """expression_phrase: [ NOT ] ( comparison | "(" expression ")" )"""
 
-    @abc.abstractmethod
     def property_first_comparison(self, arg):
-        """property_first_comparison: property ( value_op_rhs | known_op_rhs | fuzzy_string_op_rhs | set_op_rhs |
-        set_zip_op_rhs | length_op_rhs )
+        """property_first_comparison:
+        property ( value_op_rhs
+                 | known_op_rhs
+                 | fuzzy_string_op_rhs
+                 | set_op_rhs
+                 | set_zip_op_rhs
+                 | length_op_rhs )
 
         """
 
-    @abc.abstractmethod
     def constant_first_comparison(self, arg):
         """constant_first_comparison: constant OPERATOR ( non_string_value | not_implemented_string )"""
 
     @v_args(inline=True)
-    @abc.abstractmethod
     def value_op_rhs(self, operator, value):
         """value_op_rhs: OPERATOR value"""
 
-    @abc.abstractmethod
     def known_op_rhs(self, arg):
         """known_op_rhs: IS ( KNOWN | UNKNOWN )"""
 
-    @abc.abstractmethod
     def fuzzy_string_op_rhs(self, arg):
         """fuzzy_string_op_rhs: CONTAINS value | STARTS [ WITH ] value | ENDS [ WITH ] value"""
 
-    @abc.abstractmethod
     def set_op_rhs(self, arg):
         """set_op_rhs: HAS ( [ OPERATOR ] value | ALL value_list | ANY value_list | ONLY value_list )"""
 
-    @abc.abstractmethod
     def length_op_rhs(self, arg):
         """length_op_rhs: LENGTH [ OPERATOR ] value"""
 
-    @abc.abstractmethod
     def set_zip_op_rhs(self, arg):
-        """set_zip_op_rhs: property_zip_addon HAS ( value_zip | ONLY value_zip_list | ALL value_zip_list |
-        ANY value_zip_list )
+        """set_zip_op_rhs: property_zip_addon HAS ( value_zip
+        | ONLY value_zip_list
+        | ALL value_zip_list
+        | ANY value_zip_list )
 
         """
 
-    @abc.abstractmethod
     def property_zip_addon(self, arg):
         """property_zip_addon: ":" property (":" property)*"""
diff --git a/optimade/filtertransformers/elasticsearch.py b/optimade/filtertransformers/elasticsearch.py
index 0314683c..bc9b7955 100644
--- a/optimade/filtertransformers/elasticsearch.py
+++ b/optimade/filtertransformers/elasticsearch.py
@@ -1,35 +1,23 @@
-from typing import List
-
+from typing import Dict, Union, Type, Optional
 from lark import v_args
 from elasticsearch_dsl import Q, Text, Keyword, Integer, Field
-from optimade.filtertransformers import BaseTransformer
-from optimade.server.exceptions import BadRequest
-
-__all__ = ("ElasticTransformer",)
 
+from optimade.filtertransformers import BaseTransformer, Quantity
+from optimade.server.mappers import BaseResourceMapper
 
-_cmp_operators = {">": "gt", ">=": "gte", "<": "lt", "<=": "lte"}
-_rev_cmp_operators = {">": "<", ">=": "<=", "<": ">", "<=": ">=", "=": "=", "!=": "!="}
-_has_operators = {"ALL": "must", "ANY": "should"}
-_length_quantities = {
-    "elements": "nelements",
-    "elements_ratios": "nelements",
-    "dimension_types": "dimension_types",
-}
-
+__all__ = ("ElasticTransformer",)
 
-class Quantity:
-    """Class to provide information about available quantities to the transformer.
 
-    The elasticsearch transformer will use `Quantity`s to (a) do some semantic checks,
-    (b) map quantities to the underlying elastic index.
+class ElasticsearchQuantity(Quantity):
+    """Elasticsearch-specific extension of the underlying
+    [`Quantity`][optimade.filtertransformers.base_transformer.Quantity] class.
 
     Attributes:
         name: The name of the quantity as used in the filter expressions.
-        es_field: The name of the field for this quanity in elastic search, will be
+        backend_field: The name of the field for this quantity in Elasticsearch, will be
             ``name`` by default.
-        elastic_mapping_type: A decendent of an elasticsearch_dsl Field that denotes which
-            mapping was used in the elastic search index.
+        elastic_mapping_type: A decendent of an `elasticsearch_dsl.Field` that denotes which
+            mapping type was used in the Elasticsearch index.
         length_quantity: Elasticsearch does not support length of arrays, but we can
             map fields with array to other fields with ints about the array length. The
             LENGTH operator will only be supported for quantities with this attribute.
@@ -45,62 +33,145 @@ class Quantity:
             work for quantities that share the same nested object quantity.
     """
 
+    name: str
+    backend_field: Optional[str]
+    length_quantity: Optional["ElasticsearchQuantity"]
+    elastic_mapping_type: Optional[Field]
+    has_only_quantity: Optional["ElasticsearchQuantity"]
+    nested_quantity: Optional["ElasticsearchQuantity"]
+
     def __init__(
         self,
-        name,
-        es_field: str = None,
+        name: str,
+        backend_field: str = None,
+        length_quantity: "ElasticsearchQuantity" = None,
         elastic_mapping_type: Field = None,
-        length_quantity: "Quantity" = None,
-        has_only_quantity: "Quantity" = None,
-        nested_quantity: "Quantity" = None,
+        has_only_quantity: "ElasticsearchQuantity" = None,
+        nested_quantity: "ElasticsearchQuantity" = None,
     ):
+        """Initialise the quantity from its name, aliases and mapping type.
+
+        Parameters:
+            name: The name of the quantity as used in the filter expressions.
+            backend_field: The name of the field for this quantity in Elasticsearch, will be
+                ``name`` by default.
+            elastic_mapping_type: A decendent of an `elasticsearch_dsl.Field` that denotes which
+                mapping type was used in the Elasticsearch index.
+            length_quantity: Elasticsearch does not support length of arrays, but we can
+                map fields with array to other fields with ints about the array length. The
+                LENGTH operator will only be supported for quantities with this attribute.
+            has_only_quantity: Elasticsearch does not support exclusive search on arrays, like
+                a list of chemical elements. But, we can order all elements by atomic number
+                and use a keyword field with all elements to perform this search. This only
+                works for elements (i.e. labels in ``CHEMICAL_SYMBOLS``) and quantities
+                with this attribute.
+            nested_quantity: To support optimade's 'zipped tuple' feature (e.g.
+                'elements:elements_ratios HAS "H":>0.33), we use elasticsearch nested objects
+                and nested queries. This quantity will provide the field for the nested
+                object that contains the quantity (and others). The zipped tuples will only
+                work for quantities that share the same nested object quantity.
+        """
+
+        super().__init__(name, backend_field, length_quantity)
 
-        self.name = name
-        self.es_field = es_field if es_field is not None else name
         self.elastic_mapping_type = (
             Keyword if elastic_mapping_type is None else elastic_mapping_type
         )
-        self.length_quantity = length_quantity
         self.has_only_quantity = has_only_quantity
         self.nested_quantity = nested_quantity
 
-    def __repr__(self):
-        return self.name
-
 
 class ElasticTransformer(BaseTransformer):
-    """Transformer that transforms ``v0.10.1`` grammar parse trees into queries.
+    """Transformer that transforms ``v0.10.1``/`v1.0` grammar parse
+    trees into Elasticsearch queries.
+
+    Uses elasticsearch_dsl and will produce an `elasticsearch_dsl.Q` instance.
 
-    Uses elasticsearch_dsl and will produce a `Q` instance.
     """
 
-    def __init__(self, quantities: List[Quantity]):
-        """
-        Arguments:
-            quantities: A list of :class:`Quantity`s that describe how optimade (and other)
-                quantities are mapped to the elasticsearch index.
+    operator_map = {
+        "<": "lt",
+        "<=": "lte",
+        ">": "gt",
+        ">=": "gte",
+    }
+
+    _quantity_type: Type[ElasticsearchQuantity] = ElasticsearchQuantity
+
+    def __init__(
+        self, mapper: BaseResourceMapper = None, quantities: Dict[str, Quantity] = None
+    ):
+        if quantities is not None:
+            self.quantities = quantities
+
+        super().__init__(mapper=mapper)
+
+    def _field(self, quantity: Union[str, Quantity], nested: Quantity = None) -> str:
+        """Used to unwrap from `property` to the string backend field name.
+
+        If passed a `Quantity` (or a derived `ElasticsearchQuantity`), this method
+        returns the backend field name, modulo some handling of nested fields.
+
+        If passed a string quantity name:
+        - Check that the name does not match a relationship type,
+          raising a `NotImplementedError` if it does.
+        - If the string is prefixed by an underscore, assume this is a
+          provider-specific field from another provider and simply return it.
+          The original `property` rule would have already filtered out provider
+          fields for this backend appropriately as `Quantity` objects.
+
+        Returns:
+            The field name to use for database queries.
+
         """
-        self.index_mapping = {quantity.name: quantity for quantity in quantities}
-        super().__init__()
 
-    def _field(self, quantity, nested=None):
+        if isinstance(quantity, str):
+            if quantity in self.mapper.RELATIONSHIP_ENTRY_TYPES:
+                raise NotImplementedError(
+                    f"Unable to filter on relationships with type {quantity!r}"
+                )
+
+            # In this case, the property rule has already filtered out fields
+            # that do not match this provider, so this indicates an "other provider"
+            # field that should be passed over
+            if quantity.startswith("_"):
+                return quantity
+
         if nested is not None:
-            return "%s.%s" % (nested.es_field, quantity.name)
+            return "%s.%s" % (nested.backend_field, quantity.name)
 
-        return quantity.es_field
+        return quantity.backend_field
 
-    def _query_op(self, quantity, op, value, nested=None):
-        """
-        Return a range, match, or term query for the given quantity, comparison
-        operator, and value
+    def _query_op(
+        self,
+        quantity: Union[ElasticsearchQuantity, str],
+        op: str,
+        value: Union[str, float, int],
+        nested: ElasticsearchQuantity = None,
+    ) -> Q:
+        """Return a range, match, or term query for the given quantity, comparison
+        operator, and value.
+
+        Returns:
+            An elasticsearch_dsl query.
+
+        Raises:
+            BadRequest: If the query is not well-defined or is not supported.
         """
         field = self._field(quantity, nested=nested)
-        if op in _cmp_operators:
-            return Q("range", **{field: {_cmp_operators[op]: value}})
+        if op in self.operator_map:
+            return Q("range", **{field: {self.operator_map[op]: value}})
 
-        if quantity.elastic_mapping_type == Text:
+        # If quantity is an "other provider" field then use Keyword as the default
+        # mapping type. These queries should not match on anything as the field
+        # is not present in the index.
+        elastic_mapping_type = Keyword
+        if isinstance(quantity, ElasticsearchQuantity):
+            elastic_mapping_type = quantity.elastic_mapping_type
+
+        if elastic_mapping_type == Text:
             query_type = "match"
-        elif quantity.elastic_mapping_type in [Keyword, Integer]:
+        elif elastic_mapping_type in [Keyword, Integer]:
             query_type = "term"
         else:
             raise NotImplementedError("Quantity has unsupported ES field type")
@@ -116,9 +187,8 @@ class ElasticTransformer(BaseTransformer):
             return ~Q(query_type, **{field: value}) & Q("exists", field=field)
 
     def _has_query_op(self, quantities, op, predicate_zip_list):
-        """
-        Returns a bool query that combines the operator queries ():func:`_query_op`)
-        for each predicate and zipped quantity pericates combinations.
+        """Returns a bool query that combines the operator calls `_query_op`
+        for each predicate and zipped quantity predicate combination.
         """
         if op == "HAS":
             kind = "must"  # in case of HAS we do a must over the "list" of the one given element
@@ -258,7 +328,7 @@ class ElasticTransformer(BaseTransformer):
         if not isinstance(quantity, Quantity):
             raise TypeError("Only quantities can be compared to constant values.")
 
-        return self._query_op(quantity, _rev_cmp_operators[op], value)
+        return self._query_op(quantity, self._reversed_operator_map[op], value)
 
     @v_args(inline=True)
     def value_op_rhs(self, op, value):
@@ -274,9 +344,15 @@ class ElasticTransformer(BaseTransformer):
             op = "="
 
         def query(quantity):
+
+            # This is only the case if quantity is an "other" provider's field,
+            # in which case, we should treat it as unknown and try to do a null query
+            if isinstance(quantity, str):
+                return self._query_op(quantity, op, value)
+
             if quantity.length_quantity is None:
                 raise NotImplementedError(
-                    "LENGTH is not supported for %s" % quantity.name
+                    f"LENGTH is not supported for {quantity.name!r}"
                 )
             quantity = quantity.length_quantity
             return self._query_op(quantity, op, value)
@@ -368,19 +444,6 @@ class ElasticTransformer(BaseTransformer):
 
         return lambda quantity: Q("wildcard", **{self._field(quantity): wildcard})
 
-    def property(self, args):
-        # property: IDENTIFIER ( "." IDENTIFIER )*
-        quantity_name = args[0]
-
-        if quantity_name not in self.index_mapping:
-            raise BadRequest(detail=f"{quantity_name} is not a searchable quantity")
-
-        quantity = self.index_mapping.get(quantity_name, None)
-        if quantity is None:
-            quantity = Quantity(name=quantity_name)
-
-        return quantity
-
     @v_args(inline=True)
     def string(self, string):
         # string: ESCAPED_STRING
diff --git a/optimade/filtertransformers/mongo.py b/optimade/filtertransformers/mongo.py
index b86addca..e70c22d1 100755
--- a/optimade/filtertransformers/mongo.py
+++ b/optimade/filtertransformers/mongo.py
@@ -1,17 +1,27 @@
+"""This submodule implements the
+[`MongoTransformer`][optimade.filtertransformers.mongo.MongoTransformer],
+which takes the parsed filter and converts it to a valid pymongo/BSON query.
+"""
+
+
 import copy
 import warnings
+from typing import Dict, Any
 from lark import v_args, Token
-from optimade.filtertransformers.base_transformer import BaseTransformer
+from optimade.filtertransformers.base_transformer import BaseTransformer, Quantity
+
+
 from optimade.server.warnings import TimestampNotRFCCompliant
 
 __all__ = ("MongoTransformer",)
 
 
 class MongoTransformer(BaseTransformer):
-    """Transformer for MongoDB backend. Parses lark tree into
-    a dictionary to be passed to pymongo/mongomock. Uses
-    post-processing functions to handle aliasing and some
-    specific edge-cases for MongoDB.
+    """A filter transformer for the MongoDB backend.
+
+    Parses a lark tree into a dictionary representation to be
+    used by pymongo or mongomock. Uses post-processing functions
+    to handle some specific edge-cases for MongoDB.
 
     """
 
@@ -23,28 +33,14 @@ class MongoTransformer(BaseTransformer):
         "!=": "$ne",
         "=": "$eq",
     }
-    reversed_operator_map = {
-        "$lt": "$gt",
-        "$lte": "$gte",
-        "$gt": "$lt",
-        "$gte": "$lte",
-        "$ne": "$ne",
-        "$eq": "$eq",
-    }
-
-    def postprocess(self, query):
-        """Used to post-process the final parsed query."""
-        if self.mapper:
-            # important to apply length alias before normal aliases
-            query = self._apply_length_aliases(query)
-            query = self._apply_aliases(query)
 
+    def postprocess(self, query: Dict[str, Any]):
+        """Used to post-process the nested dictionary of the parsed query."""
         query = self._apply_relationship_filtering(query)
         query = self._apply_length_operators(query)
         query = self._apply_unknown_or_null_filter(query)
         query = self._apply_mongo_id_filter(query)
         query = self._apply_mongo_date_filter(query)
-
         return query
 
     def value_list(self, arg):
@@ -81,20 +77,52 @@ class MongoTransformer(BaseTransformer):
         # expression_phrase: [ NOT ] ( comparison | "(" expression ")" )
         return self._recursive_expression_phrase(arg)
 
-    def property_first_comparison(self, arg):
+    @v_args(inline=True)
+    def property_first_comparison(self, quantity, query):
         # property_first_comparison: property ( value_op_rhs | known_op_rhs | fuzzy_string_op_rhs | set_op_rhs |
         # set_zip_op_rhs | length_op_rhs )
 
         # Awkwardly, MongoDB will match null fields in $ne filters,
         # so we need to add a check for null equality in evey $ne query.
-        if "$ne" in arg[1]:
-            return {"$and": [{arg[0]: arg[1]}, {arg[0]: {"$ne": None}}]}
-
-        return {arg[0]: arg[1]}
+        if "$ne" in query:
+            return {"$and": [{quantity: query}, {quantity: {"$ne": None}}]}
+
+        # Check if a $size query is being made (indicating a length_op_rhs filter); if so, check for
+        # a defined length alias to replace the $size call with the corresponding filter on the
+        # length quantity then carefully merge the two queries.
+        #
+        # e.g. `("elements", {"$size": 2, "$all": ["Ag", "Au"]})` should become
+        # `{"elements": {"$all": ["Ag", "Au"]}, "nelements": 2}` if the `elements` -> `nelements`
+        # length alias is defined.
+        if "$size" in query:
+            if (
+                getattr(self.backend_mapping.get(quantity), "length_quantity", None)
+                is not None
+            ):
+                size_query = {
+                    self.backend_mapping[
+                        quantity
+                    ].length_quantity.backend_field: query.pop("$size")
+                }
+
+                final_query = {}
+                if query:
+                    final_query = {quantity: query}
+                for q in size_query:
+                    if q in final_query:
+                        final_query[q].update(size_query[q])
+                    else:
+                        final_query[q] = size_query[q]
+
+                return final_query
+
+        return {quantity: query}
 
     def constant_first_comparison(self, arg):
         # constant_first_comparison: constant OPERATOR ( non_string_value | not_implemented_string )
-        return {arg[2]: {self.reversed_operator_map[self.operator_map[arg[1]]]: arg[0]}}
+        return {
+            arg[2]: {self.operator_map[self._reversed_operator_map[arg[1]]]: arg[0]}
+        }
 
     @v_args(inline=True)
     def value_op_rhs(self, operator, value):
@@ -147,12 +175,20 @@ class MongoTransformer(BaseTransformer):
             f"set_op_rhs not implemented for use with OPERATOR. Given: {arg}"
         )
 
+    def property(self, args):
+        # property: IDENTIFIER ( "." IDENTIFIER )*
+        quantity = super().property(args)
+        if isinstance(quantity, Quantity):
+            quantity = quantity.backend_field
+
+        return ".".join([quantity] + args[1:])
+
     def length_op_rhs(self, arg):
         # length_op_rhs: LENGTH [ OPERATOR ] value
         if len(arg) == 2 or (len(arg) == 3 and arg[1] == "="):
             return {"$size": arg[-1]}
 
-        elif arg[1] in self.operator_map and arg[1] != "!=":
+        if arg[1] in self.operator_map and arg[1] != "!=":
             # create an invalid query that needs to be post-processed
             # e.g. {'$size': {'$gt': 2}}, which is not allowed by Mongo.
             return {"$size": {self.operator_map[arg[1]]: arg[-1]}}
@@ -197,64 +233,15 @@ class MongoTransformer(BaseTransformer):
         # simple case of negating one expression, from NOT (expr) to ~expr.
         return {prop: {"$not": expr} for prop, expr in arg[1].items()}
 
-    def _apply_length_aliases(self, filter_: dict) -> dict:
-        """Recursively search query for any `$size` operations, and check
-        if the property can be replaced with its corresponding length
-        alias.
-
-        """
-
-        def check_for_size(prop, expr):
-            return (
-                isinstance(expr, dict)
-                and "$size" in expr
-                and self.mapper.length_alias_for(prop)
-            )
-
-        def replace_with_length_alias(subdict, prop, expr):
-            subdict[self.mapper.length_alias_for(prop)] = expr["$size"]
-            subdict[prop].pop("$size")
-            if not subdict[prop]:
-                subdict.pop(prop)
-            return subdict
-
-        return recursive_postprocessing(
-            filter_, check_for_size, replace_with_length_alias
-        )
-
-    def _apply_aliases(self, filter_: dict) -> dict:
-        """Check whether any fields in the filter have aliases so
-        that they can be renamed for the Mongo query.
-
-        """
-        # if there are no defined aliases, just skip
-        if not self.mapper.all_aliases():
-            return filter_
-
-        def check_for_alias(prop, expr):
-            return self.mapper.get_backend_field(prop) != prop
-
-        def apply_alias(subdict, prop, expr):
-            if isinstance(subdict, dict):
-                subdict[self.mapper.get_backend_field(prop)] = self._apply_aliases(
-                    subdict.pop(prop)
-                )
-            elif isinstance(subdict, str):
-                subdict = self.mapper.get_backend_field(subdict)
-
-            return subdict
-
-        return recursive_postprocessing(filter_, check_for_alias, apply_alias)
-
     def _apply_length_operators(self, filter_: dict) -> dict:
-        """Check for any invalid pymongo queries that involve
-        applying an operator to the length of a field, and transform
+        """Check for any invalid pymongo queries that involve applying a
+        comparison operator to the length of a field, and transform
         them into a test for existence of the relevant entry, e.g.
         "list LENGTH > 3" becomes "does the 4th list entry exist?".
 
         """
 
-        def check_for_length_op_filter(prop, expr):
+        def check_for_length_op_filter(_, expr):
             return (
                 isinstance(expr, dict)
                 and "$size" in expr
@@ -300,7 +287,7 @@ class MongoTransformer(BaseTransformer):
 
         """
 
-        def check_for_entry_type(prop, expr):
+        def check_for_entry_type(prop, _):
             return str(prop).count(".") == 1 and str(prop).split(".")[0] in (
                 "structures",
                 "references",
@@ -320,7 +307,11 @@ class MongoTransformer(BaseTransformer):
                     subdict["$and"] = []
                 subdict["$and"].extend(
                     [
-                        {f"relationships.{_prop}.data": {"$size": expr.pop("$size")}},
+                        {
+                            f"relationships.{_prop}.data": {
+                                "$size": expr.pop("$size"),
+                            }
+                        },
                         {f"relationships.{_prop}.data.{_field}": expr},
                     ]
                 )
diff --git a/optimade/server/entry_collections/elasticsearch.py b/optimade/server/entry_collections/elasticsearch.py
index 05b02f86..e302d46e 100644
--- a/optimade/server/entry_collections/elasticsearch.py
+++ b/optimade/server/entry_collections/elasticsearch.py
@@ -2,8 +2,7 @@ import json
 from pathlib import Path
 from typing import Tuple, List, Optional, Dict, Any, Iterable, Union
 
-from optimade.filterparser import LarkParser
-from optimade.filtertransformers.elasticsearch import ElasticTransformer, Quantity
+from optimade.filtertransformers.elasticsearch import ElasticTransformer
 from optimade.server.config import CONFIG
 from optimade.server.logger import LOGGER
 from optimade.models import EntryResource
@@ -38,35 +37,13 @@ class ElasticCollection(EntryCollection):
             client: A preconfigured Elasticsearch client.
 
         """
-        self.client = client if client else CLIENT
-
-        self.resource_cls = resource_cls
-        self.resource_mapper = resource_mapper
-        self.provider_prefix = CONFIG.provider.prefix
-        self.provider_fields = CONFIG.provider_fields.get(resource_mapper.ENDPOINT, [])
-        self.parser = LarkParser()
-
-        quantities = {}
-        for field in self.all_fields:
-            alias = self.resource_mapper.get_backend_field(field)
-            length_alias = self.resource_mapper.length_alias_for(field)
-
-            quantities[field] = Quantity(name=field, es_field=alias)
-            if length_alias is not None:
-                quantities[length_alias] = Quantity(name=length_alias)
-                quantities[field].length_quantity = quantities[length_alias]
-
-        if "elements" in quantities:
-            quantities["elements"].has_only_quantity = Quantity(name="elements_only")
-            quantities["elements"].nested_quantity = quantities["elements_ratios"]
-
-        if "elements_ratios" in quantities:
-            quantities["elements_ratios"].nested_quantity = quantities[
-                "elements_ratios"
-            ]
-
-        self.transformer = ElasticTransformer(quantities=quantities.values())
+        super().__init__(
+            resource_cls=resource_cls,
+            resource_mapper=resource_mapper,
+            transformer=ElasticTransformer(mapper=resource_mapper),
+        )
 
+        self.client = client if client else CLIENT
         self.name = name
 
         # If we are creating a new collection from scratch, also create the index,
diff --git a/optimade/server/entry_collections/entry_collections.py b/optimade/server/entry_collections/entry_collections.py
index b5061c80..88a92d8d 100644
--- a/optimade/server/entry_collections/entry_collections.py
+++ b/optimade/server/entry_collections/entry_collections.py
@@ -1,5 +1,5 @@
 from abc import abstractmethod, ABC
-from typing import Tuple, List, Union, Dict, Any
+from typing import Tuple, List, Union, Dict, Any, Set
 import warnings
 import re
 
@@ -12,7 +12,7 @@ from optimade.server.config import CONFIG, SupportedBackend
 from optimade.server.exceptions import BadRequest, Forbidden
 from optimade.server.mappers import BaseResourceMapper
 from optimade.server.query_params import EntryListingQueryParams, SingleEntryQueryParams
-from optimade.server.warnings import FieldValueNotRecognized
+from optimade.server.warnings import FieldValueNotRecognized, UnknownProviderProperty
 
 
 def create_collection(
@@ -144,21 +144,43 @@ class EntryCollection(ABC):
                 )
 
         exclude_fields = self.all_fields - response_fields
+        include_fields = (
+            response_fields - self.resource_mapper.TOP_LEVEL_NON_ATTRIBUTES_FIELDS
+        )
+
+        bad_optimade_fields = set()
+        bad_provider_fields = set()
+        for field in include_fields:
+            if field not in self.resource_mapper.ALL_ATTRIBUTES:
+                if field.startswith("_"):
+                    if any(
+                        field.startswith(f"_{prefix}_")
+                        for prefix in self.resource_mapper.SUPPORTED_PREFIXES
+                    ):
+                        bad_provider_fields.add(field)
+                else:
+                    bad_optimade_fields.add(field)
+
+        if bad_provider_fields:
+            warnings.warn(
+                message=f"Unrecognised field(s) for this provider requested in `response_fields`: {bad_provider_fields}.",
+                category=UnknownProviderProperty,
+            )
+
+        if bad_optimade_fields:
+            raise BadRequest(
+                detail=f"Unrecognised OPTIMADE field(s) in requested `response_fields`: {bad_optimade_fields}."
+            )
 
         if results:
-            if isinstance(results, dict):
-                results = self.resource_cls(**self.resource_mapper.map_back(results))
-            else:
-                results = [
-                    self.resource_cls(**self.resource_mapper.map_back(doc))
-                    for doc in results
-                ]
+            results = self.resource_mapper.deserialize(results)
 
         return (
             results,
             data_returned,
             more_data_available,
             exclude_fields,
+            include_fields,
         )
 
     @abstractmethod
@@ -197,7 +219,7 @@ class EntryCollection(ABC):
 
         return fields
 
-    def get_attribute_fields(self) -> set:
+    def get_attribute_fields(self) -> Set[str]:
         """Get the set of attribute fields
 
         Return only the _first-level_ attribute fields from the schema of the resource class,
diff --git a/optimade/server/mappers/entries.py b/optimade/server/mappers/entries.py
index f157c621..00101b87 100644
--- a/optimade/server/mappers/entries.py
+++ b/optimade/server/mappers/entries.py
@@ -1,9 +1,30 @@
-from typing import Tuple, Optional
+from typing import Tuple, Optional, Type, Set, Dict, Any, Union, List, Iterable
 import warnings
 
+from optimade.models.entries import EntryResource
+
 __all__ = ("BaseResourceMapper",)
 
 
+class classproperty(property):
+    """A simple extension of the property decorator that binds to types
+    rather than instances.
+
+    Modelled on this [StackOverflow answer](https://stackoverflow.com/a/5192374)
+    with some tweaks to allow mkdocstrings to do its thing.
+
+    """
+
+    def __init__(self, func):
+        self.__name__ = func.__name__
+        self.__module__ = func.__module__
+        self.__doc__ = func.__doc__
+        self.__wrapped__ = func
+
+    def __get__(self, _, owner):
+        return self.__wrapped__(owner)
+
+
 class BaseResourceMapper:
     """
     Generic Resource Mapper that defines and performs the mapping
@@ -11,30 +32,48 @@ class BaseResourceMapper:
     the specification.
 
     Attributes:
-        ENDPOINT (str): defines the endpoint for which to apply this
-            mapper.
-        ALIASES (Tuple[Tuple[str, str]]): a tuple of aliases between
+        ALIASES: a tuple of aliases between
             OPTIMADE field names and the field names in the database ,
             e.g. `(("elements", "custom_elements_field"))`.
-        LENGTH_ALIASES (Tuple[Tuple[str, str]]): a tuple of aliases between
+        LENGTH_ALIASES: a tuple of aliases between
             a field name and another field that defines its length, to be used
             when querying, e.g. `(("elements", "nelements"))`.
             e.g. `(("elements", "custom_elements_field"))`.
-        PROVIDER_FIELDS (Tuple[str]): a tuple of extra field names that this
+        ENTRY_RESOURCE_CLASS: The entry type that this mapper corresponds to.
+        PROVIDER_FIELDS: a tuple of extra field names that this
             mapper should support when querying with the database prefix.
-        REQUIRED_FIELDS (set[str]): the set of fieldnames to return
-            when mapping to the OPTIMADE format.
-        TOP_LEVEL_NON_ATTRIBUTES_FIELDS (set[str]): the set of top-level
+        TOP_LEVEL_NON_ATTRIBUTES_FIELDS: the set of top-level
             field names common to all endpoints.
+        SUPPORTED_PREFIXES: The set of prefixes registered by this mapper.
+        ALL_ATTRIBUTES: The set of attributes defined across the entry
+            resource class and the server configuration.
+        ENTRY_RESOURCE_ATTRIBUTES: A dictionary of attributes and their definitions
+            defined by the schema of the entry resource class.
+        ENDPOINT: The expected endpoint name for this resource, as defined by
+            the `type` in the schema of the entry resource class.
 
     """
 
-    ENDPOINT: str = ""
+    try:
+        from optimade.server.data import (
+            providers as PROVIDERS,
+        )  # pylint: disable=no-name-in-module
+    except (ImportError, ModuleNotFoundError):
+        PROVIDERS = {}
+
+    KNOWN_PROVIDER_PREFIXES: Set[str] = set(
+        prov["id"] for prov in PROVIDERS.get("data", [])
+    )
     ALIASES: Tuple[Tuple[str, str]] = ()
     LENGTH_ALIASES: Tuple[Tuple[str, str]] = ()
     PROVIDER_FIELDS: Tuple[str] = ()
-    REQUIRED_FIELDS: set = set()
-    TOP_LEVEL_NON_ATTRIBUTES_FIELDS: set = {"id", "type", "relationships", "links"}
+    ENTRY_RESOURCE_CLASS: Type[EntryResource] = EntryResource
+    RELATIONSHIP_ENTRY_TYPES: Set[str] = {"references", "structures"}
+    TOP_LEVEL_NON_ATTRIBUTES_FIELDS: Set[str] = {"id", "type", "relationships", "links"}
+    SUPPORTED_PREFIXES: Set[str]
+    ALL_ATTRIBUTES: Set[str]
+    ENTRY_RESOURCE_ATTRIBUTES: Dict[str, Any]
+    ENDPOINT: str
 
     @classmethod
     def all_aliases(cls) -> Tuple[Tuple[str, str]]:
@@ -62,6 +101,55 @@ class BaseResourceMapper:
             + cls.ALIASES
         )
 
+    @classproperty
+    def SUPPORTED_PREFIXES(cls) -> Set[str]:
+        """A set of prefixes handled by this entry type.
+
+        !!! note
+            This implementation only includes the provider prefix,
+            but in the future this property may be extended to include other
+            namespaces (for serving fields from, e.g., other providers or
+            domain-specific terms).
+
+        """
+        from optimade.server.config import CONFIG
+
+        return {CONFIG.provider.prefix}
+
+    @classproperty
+    def ALL_ATTRIBUTES(cls) -> Set[str]:
+        """Returns all attributes served by this entry."""
+        from optimade.server.config import CONFIG
+
+        return (
+            set(cls.ENTRY_RESOURCE_ATTRIBUTES)
+            .union(
+                cls.get_optimade_field(field)
+                for field in CONFIG.provider_fields.get(cls.ENDPOINT, ())
+            )
+            .union(set(cls.get_optimade_field(field) for field in cls.PROVIDER_FIELDS))
+        )
+
+    @classproperty
+    def ENTRY_RESOURCE_ATTRIBUTES(cls) -> Dict[str, Any]:
+        """Returns the dictionary of attributes defined by the underlying entry resource class."""
+        from optimade.server.schemas import retrieve_queryable_properties
+
+        return retrieve_queryable_properties(cls.ENTRY_RESOURCE_CLASS.schema())
+
+    @classproperty
+    def ENDPOINT(cls) -> str:
+        """Returns the expected endpoint for this mapper, corresponding
+        to the `type` property of the resource class.
+
+        """
+        return (
+            cls.ENTRY_RESOURCE_CLASS.schema()
+            .get("properties", {})
+            .get("type", {})
+            .get("const", "")
+        )
+
     @classmethod
     def all_length_aliases(cls) -> Tuple[Tuple[str, str]]:
         """Returns all of the associated length aliases for this class,
@@ -203,9 +291,7 @@ class BaseResourceMapper:
             REQUIRED response fields.
 
         """
-        res = cls.TOP_LEVEL_NON_ATTRIBUTES_FIELDS.copy()
-        res.update(cls.REQUIRED_FIELDS)
-        return res
+        return cls.TOP_LEVEL_NON_ATTRIBUTES_FIELDS
 
     @classmethod
     def map_back(cls, doc: dict) -> dict:
@@ -250,3 +336,12 @@ class BaseResourceMapper:
         newdoc["attributes"] = attributes
 
         return newdoc
+
+    @classmethod
+    def deserialize(
+        cls, results: Union[dict, Iterable[dict]]
+    ) -> Union[List[EntryResource], EntryResource]:
+        if isinstance(results, dict):
+            return cls.ENTRY_RESOURCE_CLASS(**cls.map_back(results))
+
+        return [cls.ENTRY_RESOURCE_CLASS(**cls.map_back(doc)) for doc in results]
diff --git a/optimade/server/mappers/links.py b/optimade/server/mappers/links.py
index 7290e213..cba0e28b 100644
--- a/optimade/server/mappers/links.py
+++ b/optimade/server/mappers/links.py
@@ -1,4 +1,5 @@
 from optimade.server.mappers.entries import BaseResourceMapper
+from optimade.models.links import LinksResource
 
 __all__ = ("LinksMapper",)
 
@@ -7,6 +8,8 @@ class LinksMapper(BaseResourceMapper):
 
     ENDPOINT = "links"
 
+    ENTRY_RESOURCE_CLASS = LinksResource
+
     @classmethod
     def map_back(cls, doc: dict) -> dict:
         """Map properties from MongoDB to OPTIMADE
diff --git a/optimade/server/mappers/references.py b/optimade/server/mappers/references.py
index 01e9f964..e756296e 100644
--- a/optimade/server/mappers/references.py
+++ b/optimade/server/mappers/references.py
@@ -1,8 +1,9 @@
 from optimade.server.mappers.entries import BaseResourceMapper
+from optimade.models.references import ReferenceResource
 
 __all__ = ("ReferenceMapper",)
 
 
 class ReferenceMapper(BaseResourceMapper):
 
-    ENDPOINT = "references"
+    ENTRY_RESOURCE_CLASS = ReferenceResource
diff --git a/optimade/server/mappers/structures.py b/optimade/server/mappers/structures.py
index c6332b95..a63bb171 100644
--- a/optimade/server/mappers/structures.py
+++ b/optimade/server/mappers/structures.py
@@ -1,15 +1,15 @@
 from optimade.server.mappers.entries import BaseResourceMapper
+from optimade.models.structures import StructureResource
 
 __all__ = ("StructureMapper",)
 
 
 class StructureMapper(BaseResourceMapper):
 
-    ENDPOINT = "structures"
-
     LENGTH_ALIASES = (
         ("elements", "nelements"),
         ("element_ratios", "nelements"),
         ("cartesian_site_positions", "nsites"),
         ("species_at_sites", "nsites"),
     )
+    ENTRY_RESOURCE_CLASS = StructureResource
diff --git a/optimade/server/routers/utils.py b/optimade/server/routers/utils.py
index 45ecdefc..bedb6597 100644
--- a/optimade/server/routers/utils.py
+++ b/optimade/server/routers/utils.py
@@ -2,7 +2,7 @@
 import re
 import urllib
 from datetime import datetime
-from typing import Union, List, Dict
+from typing import Union, List, Dict, Set
 
 from fastapi import HTTPException, Request
 from starlette.datastructures import URL as StarletteURL
@@ -63,34 +63,44 @@ def meta_values(
 
 
 def handle_response_fields(
-    results: Union[List[EntryResource], EntryResource], exclude_fields: set
-) -> dict:
-    """Handle query parameter ``response_fields``
+    results: Union[List[EntryResource], EntryResource],
+    exclude_fields: Set[str],
+    include_fields: Set[str],
+) -> List[dict]:
+    """Handle query parameter `response_fields`.
 
-    It is assumed that all fields are under ``attributes``.
+    It is assumed that all fields are under `attributes`.
     This is due to all other top-level fields are REQUIRED in the response.
 
-    :param exclude_fields: Fields under ``attributes`` to be excluded from the response.
+    Parameters:
+        exclude_fields: Fields under `attributes` to be excluded from the response.
+        include_fields: Fields under `attributes` that were requested that should be
+            set to null if missing in the entry.
+
+    Returns:
+        List of resulting resources as dictionaries after pruning according to
+        the `response_fields` OPTIMADE URL query parameter.
+
     """
     if not isinstance(results, list):
         results = [results]
 
     new_results = []
     while results:
-        entry = results.pop(0)
-
-        # TODO: re-enable exclude_unset when proper handling of known/unknown fields
-        # has been implemented (relevant issue: https://github.com/Materials-Consortia/optimade-python-tools/issues/263)
-        # Have to handle top level fields explicitly here for now
-        new_entry = entry.dict(exclude_unset=False)
-        for field in ("relationships", "links", "meta", "type", "id"):
-            if field in new_entry and new_entry[field] is None:
-                del new_entry[field]
+        new_entry = results.pop(0).dict(exclude_unset=True)
 
+        # Remove fields excluded by their omission in `response_fields`
         for field in exclude_fields:
             if field in new_entry["attributes"]:
                 del new_entry["attributes"][field]
+
+        # Include missing fields that were requested in `response_fields`
+        for field in include_fields:
+            if field not in new_entry["attributes"]:
+                new_entry["attributes"][field] = None
+
         new_results.append(new_entry)
+
     return new_results
 
 
@@ -164,7 +174,7 @@ def get_included_relationships(
         )
 
         # still need to handle pagination
-        ref_results, _, _, _ = ENTRY_COLLECTIONS[entry_type].find(params)
+        ref_results, _, _, _, _ = ENTRY_COLLECTIONS[entry_type].find(params)
         included[entry_type] = ref_results
 
     # flatten dict by endpoint to list
@@ -201,7 +211,13 @@ def get_entries(
     """Generalized /{entry} endpoint getter"""
     from optimade.server.routers import ENTRY_COLLECTIONS
 
-    results, data_returned, more_data_available, fields = collection.find(params)
+    (
+        results,
+        data_returned,
+        more_data_available,
+        fields,
+        include_fields,
+    ) = collection.find(params)
 
     include = []
     if getattr(params, "include", False):
@@ -219,8 +235,8 @@ def get_entries(
     else:
         links = ToplevelLinks(next=None)
 
-    if fields:
-        results = handle_response_fields(results, fields)
+    if fields or include_fields:
+        results = handle_response_fields(results, fields, include_fields)
 
     return response(
         links=links,
@@ -245,7 +261,13 @@ def get_single_entry(
     from optimade.server.routers import ENTRY_COLLECTIONS
 
     params.filter = f'id="{entry_id}"'
-    results, data_returned, more_data_available, fields = collection.find(params)
+    (
+        results,
+        data_returned,
+        more_data_available,
+        fields,
+        include_fields,
+    ) = collection.find(params)
 
     include = []
     if getattr(params, "include", False):
@@ -260,8 +282,8 @@ def get_single_entry(
 
     links = ToplevelLinks(next=None)
 
-    if fields and results is not None:
-        results = handle_response_fields(results, fields)[0]
+    if fields or include_fields and results is not None:
+        results = handle_response_fields(results, fields, include_fields)[0]
 
     return response(
         links=links,
diff --git a/optimade/server/schemas.py b/optimade/server/schemas.py
index d059a815..47dd9dbd 100644
--- a/optimade/server/schemas.py
+++ b/optimade/server/schemas.py
@@ -6,8 +6,10 @@ ENTRY_INFO_SCHEMAS = {
 }
 
 
-def retrieve_queryable_properties(schema: dict, queryable_properties: list) -> dict:
-    """Recurisvely loops through the schema of a pydantic model and
+def retrieve_queryable_properties(
+    schema: dict, queryable_properties: list = None
+) -> dict:
+    """Recursively loops through the schema of a pydantic model and
     resolves all references, returning a dictionary of all the
     OPTIMADE-queryable properties of that model.
 
@@ -23,7 +25,7 @@ def retrieve_queryable_properties(schema: dict, queryable_properties: list) -> d
     """
     properties = {}
     for name, value in schema["properties"].items():
-        if name in queryable_properties:
+        if not queryable_properties or name in queryable_properties:
             if "$ref" in value:
                 path = value["$ref"].split("/")[1:]
                 sub_schema = schema.copy()
@@ -44,7 +46,7 @@ def retrieve_queryable_properties(schema: dict, queryable_properties: list) -> d
                 properties[name]["sortable"] = value.get("sortable", True)
                 # Try to get OpenAPI-specific "format" if possible, else get "type"; a mandatory OpenAPI key.
                 properties[name]["type"] = DataType.from_json_type(
-                    value.get("format", value["type"])
+                    value.get("format", value.get("type"))
                 )
 
     return properties
diff --git a/optimade/server/warnings.py b/optimade/server/warnings.py
index 4fb60658..1db05c3b 100644
--- a/optimade/server/warnings.py
+++ b/optimade/server/warnings.py
@@ -49,3 +49,10 @@ class TimestampNotRFCCompliant(OptimadeWarning):
     RFC 3339 compliant. This may cause undefined behaviour in the query results.
 
     """
+
+
+class UnknownProviderProperty(OptimadeWarning):
+    """A provider-specific property has been requested via `response_fields` or as in a `filter` that is not
+    recognised by this implementation.
+
+    """
diff --git a/optimade/validator/validator.py b/optimade/validator/validator.py
index 866061a6..8009bbec 100644
--- a/optimade/validator/validator.py
+++ b/optimade/validator/validator.py
@@ -387,8 +387,47 @@ class ImplementationValidator:
                 optional=optional,
             )
 
+        self._test_unknown_provider_property(endp)
+        self._test_completely_unknown_property(endp)
+
         return True, f"successfully recursed through endpoint {endp}."
 
+    @test_case
+    def _test_completely_unknown_property(self, endp):
+        request = f"{endp}?filter=crazyfield = 2"
+        response, _ = self._get_endpoint(
+            request,
+            expected_status_code=400,
+        )
+
+        return True, "unknown field returned 400 Bad Request, as expected"
+
+    @test_case
+    def _test_unknown_provider_property(self, endp):
+
+        dummy_provider_field = "_crazyprovider_field"
+
+        request = f"{endp}?filter={dummy_provider_field}=2"
+        response, _ = self._get_endpoint(
+            request,
+            multistage=True,
+            request=request,
+        )
+
+        if response is not None:
+            deserialized, _ = self._deserialize_response(
+                response, CONF.response_classes[endp], request=request, multistage=True
+            )
+
+            return (
+                True,
+                "Unknown provider field was ignored when filtering, as expected",
+            )
+
+        raise ResponseError(
+            "Failed to handle field from unknown provider; should return without affecting filter results"
+        )
+
     def _check_entry_info(self, entry_info: Dict[str, Any], endp: str) -> List[str]:
         """Checks that `entry_info` contains all the required properties,
         and returns the property list for the endpoint.


Test patch:
diff --git a/tests/filtertransformers/test_base.py b/tests/filtertransformers/test_base.py
new file mode 100644
index 00000000..9cc251d5
--- /dev/null
+++ b/tests/filtertransformers/test_base.py
@@ -0,0 +1,27 @@
+from optimade.filtertransformers import BaseTransformer
+
+
+def test_quantity_builder(mapper):
+    class DummyTransformer(BaseTransformer):
+        pass
+
+    class AwkwardMapper(mapper("StructureMapper")):
+
+        ALIASES = (("elements", "my_elements"), ("nelements", "nelem"))
+        LENGTH_ALIASES = (
+            ("chemsys", "nelements"),
+            ("cartesian_site_positions", "nsites"),
+            ("elements", "nelements"),
+        )
+        PROVIDER_FIELDS = ("chemsys",)
+
+    m = AwkwardMapper
+    t = DummyTransformer(mapper=m)
+
+    assert "_exmpl_chemsys" in t.quantities
+    assert t.quantities["_exmpl_chemsys"].name == "_exmpl_chemsys"
+    assert t.quantities["_exmpl_chemsys"].backend_field == "chemsys"
+    assert t.quantities["_exmpl_chemsys"].length_quantity.name == "nelements"
+    assert t.quantities["_exmpl_chemsys"].length_quantity.backend_field == "nelem"
+
+    assert t.quantities["elements"].length_quantity.backend_field == "nelem"
diff --git a/tests/filtertransformers/test_elasticsearch.py b/tests/filtertransformers/test_elasticsearch.py
index 0e5ac1e0..bf72b729 100644
--- a/tests/filtertransformers/test_elasticsearch.py
+++ b/tests/filtertransformers/test_elasticsearch.py
@@ -6,7 +6,9 @@ elasticsearch_dsl = pytest.importorskip(
 )
 
 from optimade.filterparser import LarkParser
-from optimade.filtertransformers.elasticsearch import ElasticTransformer, Quantity
+from optimade.filtertransformers.elasticsearch import (
+    ElasticTransformer,
+)
 
 
 @pytest.fixture
@@ -16,28 +18,9 @@ def parser():
 
 @pytest.fixture
 def transformer():
-    nelements = Quantity(name="nelements")
-    elements_only = Quantity(name="elements_only")
-    elements_ratios = Quantity(name="elements_ratios")
-    elements_ratios.nested_quantity = elements_ratios
-    elements = Quantity(
-        name="elements",
-        length_quantity=nelements,
-        has_only_quantity=elements_only,
-        nested_quantity=elements_ratios,
-    )
-    dimension_types = Quantity(name="dimension_types")
-    dimension_types.length_quantity = dimension_types
-
-    quantities = [
-        nelements,
-        elements_only,
-        elements_ratios,
-        elements,
-        dimension_types,
-        Quantity(name="chemical_formula_reduced"),
-    ]
-    return ElasticTransformer(quantities=quantities)
+    from optimade.server.mappers import StructureMapper
+
+    return ElasticTransformer(mapper=StructureMapper())
 
 
 test_queries = [
@@ -71,14 +54,15 @@ test_queries = [
     ('chemical_formula_reduced ENDS "C"', 1),
     ("elements LENGTH = 2", 3),
     ("elements LENGTH = 3", 1),
-    ("dimension_types LENGTH = 0", 3),
-    ("dimension_typesLENGTH = 1", 1),
-    ("nelements = 2 AND dimension_types LENGTH = 1", 1),
-    ("nelements = 3 AND dimension_types LENGTH = 1", 0),
-    ("nelements = 3 OR dimension_types LENGTH = 1", 2),
-    ("nelements > 1 OR dimension_types LENGTH = 1 AND nelements = 2", 4),
-    ("(nelements > 1 OR dimension_types LENGTH = 1) AND nelements = 2", 3),
-    ("NOT dimension_types LENGTH = 1", 3),
+    ("elements LENGTH > 1", 3),
+    ("elementsLENGTH = 1", 1),
+    ("nelements = 2 AND elements LENGTH = 2", 1),
+    ("nelements = 3 AND elements LENGTH = 1", 0),
+    ("nelements = 3 OR elements LENGTH = 1", 2),
+    ("nelements > 1 OR elements LENGTH = 1 AND nelements = 2", 4),
+    ("(nelements > 1 OR elements LENGTH = 1) AND nelements = 2", 3),
+    ("NOT elements LENGTH = 1", 3),
+    ("_exmpl2_field = 2", 1),
 ]
 
 
@@ -87,3 +71,26 @@ def test_parse_n_transform(query, parser, transformer):
     tree = parser.parse(query[0])
     result = transformer.transform(tree)
     assert result is not None
+
+
+def test_bad_queries(parser, transformer):
+    filter_ = "unknown_field = 0"
+    with pytest.raises(
+        Exception, match="'unknown_field' is not a known or searchable quantity"
+    ) as exc_info:
+        transformer.transform(parser.parse(filter_))
+    assert exc_info.type.__name__ == "VisitError"
+
+    filter_ = "dimension_types LENGTH = 0"
+    with pytest.raises(
+        Exception, match="LENGTH is not supported for 'dimension_types'"
+    ) as exc_info:
+        transformer.transform(parser.parse(filter_))
+    assert exc_info.type.__name__ == "VisitError"
+
+    filter_ = "_exmpl_field = 1"
+    with pytest.raises(
+        Exception, match="'_exmpl_field' is not a known or searchable quantity"
+    ) as exc_info:
+        transformer.transform(parser.parse(filter_))
+    assert exc_info.type.__name__ == "VisitError"
diff --git a/tests/filtertransformers/test_mongo.py b/tests/filtertransformers/test_mongo.py
index d797e618..a42fad5c 100644
--- a/tests/filtertransformers/test_mongo.py
+++ b/tests/filtertransformers/test_mongo.py
@@ -35,12 +35,14 @@ class TestMongoTransformer:
         with pytest.raises(BadRequest):
             self.transform("BadLuck IS KNOWN")  # contains upper-case letters
 
+    def test_provider_property_name(self):
         # database-provider-specific prefixes
         assert self.transform("_exmpl_formula_sum = 1") == {
             "_exmpl_formula_sum": {"$eq": 1}
         }
         assert self.transform("_exmpl_band_gap = 1") == {"_exmpl_band_gap": {"$eq": 1}}
 
+    def test_nested_property_names(self):
         # Nested property names
         assert self.transform("identifier1.identifierd2 = 42") == {
             "identifier1.identifierd2": {"$eq": 42}
@@ -260,39 +262,53 @@ class TestMongoTransformer:
         with pytest.raises(VisitError):
             self.transform('"some string" > "some other string"')
 
-    def test_filtering_on_relationships(self):
+    def test_filtering_on_relationships(self, mapper):
         """Test the nested properties with special names
         like "structures", "references" etc. are applied
         to the relationships field.
 
         """
+        from optimade.filtertransformers.mongo import MongoTransformer
+
+        t = MongoTransformer(mapper=mapper("StructureMapper"))
+        p = LarkParser(version=self.version, variant=self.variant)
 
-        assert self.transform('references.id HAS "dummy/2019"') == {
+        assert t.transform(p.parse('references.id HAS "dummy/2019"')) == {
             "relationships.references.data.id": {"$in": ["dummy/2019"]}
         }
 
-        assert self.transform('structures.id HAS ANY "dummy/2019", "dijkstra1968"') == {
+        assert t.transform(
+            p.parse('structures.id HAS ANY "dummy/2019", "dijkstra1968"')
+        ) == {
             "relationships.structures.data.id": {"$in": ["dummy/2019", "dijkstra1968"]}
         }
 
-        assert self.transform('structures.id HAS ALL "dummy/2019", "dijkstra1968"') == {
+        assert t.transform(
+            p.parse('structures.id HAS ALL "dummy/2019", "dijkstra1968"')
+        ) == {
             "relationships.structures.data.id": {"$all": ["dummy/2019", "dijkstra1968"]}
         }
 
-        assert self.transform('structures.id HAS ONLY "dummy/2019"') == {
+        assert t.transform(p.parse('structures.id HAS ONLY "dummy/2019"')) == {
             "$and": [
                 {"relationships.structures.data": {"$size": 1}},
                 {"relationships.structures.data.id": {"$all": ["dummy/2019"]}},
             ]
         }
 
-        assert self.transform(
-            'structures.id HAS ONLY "dummy/2019" AND structures.id HAS "dummy/2019"'
+        assert t.transform(
+            p.parse(
+                'structures.id HAS ONLY "dummy/2019" AND structures.id HAS "dummy/2019"'
+            )
         ) == {
             "$and": [
                 {
                     "$and": [
-                        {"relationships.structures.data": {"$size": 1}},
+                        {
+                            "relationships.structures.data": {
+                                "$size": 1,
+                            }
+                        },
                         {"relationships.structures.data.id": {"$all": ["dummy/2019"]}},
                     ]
                 },
@@ -300,6 +316,19 @@ class TestMongoTransformer:
             ],
         }
 
+    def test_other_provider_fields(self, mapper):
+        """Test that fields from other providers generate
+        queries that treat the value of the field as `null`.
+
+        """
+        from optimade.filtertransformers.mongo import MongoTransformer
+
+        t = MongoTransformer(mapper=mapper("StructureMapper"))
+        p = LarkParser(version=self.version, variant=self.variant)
+        assert t.transform(p.parse("_other_provider_field > 1")) == {
+            "_other_provider_field": {"$gt": 1}
+        }
+
     def test_not_implemented(self):
         """Test that list properties that are currently not implemented
         give a sensible response.
@@ -346,7 +375,7 @@ class TestMongoTransformer:
     def test_list_length_aliases(self, mapper):
         from optimade.filtertransformers.mongo import MongoTransformer
 
-        transformer = MongoTransformer(mapper=mapper("StructureMapper")())
+        transformer = MongoTransformer(mapper=mapper("StructureMapper"))
         parser = LarkParser(version=self.version, variant=self.variant)
 
         assert transformer.transform(parser.parse("elements LENGTH 3")) == {
@@ -411,7 +440,7 @@ class TestMongoTransformer:
         class MyMapper(mapper("StructureMapper")):
             ALIASES = (("last_modified", "ctime"),)
 
-        transformer = MongoTransformer(mapper=MyMapper())
+        transformer = MongoTransformer(mapper=MyMapper)
         parser = LarkParser(version=self.version, variant=self.variant)
 
         assert transformer.transform(
@@ -447,7 +476,7 @@ class TestMongoTransformer:
         class MyMapper(mapper("StructureMapper")):
             ALIASES = (("immutable_id", "_id"),)
 
-        transformer = MongoTransformer(mapper=MyMapper())
+        transformer = MongoTransformer(mapper=MyMapper)
         parser = LarkParser(version=self.version, variant=self.variant)
 
         assert transformer.transform(
@@ -474,7 +503,11 @@ class TestMongoTransformer:
         from optimade.filtertransformers.mongo import MongoTransformer
 
         class MyMapper(mapper("StructureMapper")):
-            ALIASES = (("elements", "my_elements"), ("nelements", "nelem"))
+            ALIASES = (
+                ("elements", "my_elements"),
+                ("nelements", "nelem"),
+                ("elements_ratios", "ratios"),
+            )
             LENGTH_ALIASES = (
                 ("chemsys", "nelements"),
                 ("cartesian_site_positions", "nsites"),
@@ -482,7 +515,8 @@ class TestMongoTransformer:
             )
             PROVIDER_FIELDS = ("chemsys",)
 
-        transformer = MongoTransformer(mapper=MyMapper())
+        m = MyMapper
+        transformer = MongoTransformer(mapper=m)
         parser = LarkParser(version=self.version, variant=self.variant)
 
         assert transformer.transform(
@@ -492,11 +526,16 @@ class TestMongoTransformer:
             parser.parse("cartesian_site_positions LENGTH < 3")
         ) == {"nsites": {"$lt": 3}}
         assert transformer.transform(
-            parser.parse("cartesian_site_positions LENGTH 3")
+            parser.parse("cartesian_site_positions LENGTH = 3")
         ) == {"nsites": 3}
         assert transformer.transform(
             parser.parse("cartesian_site_positions LENGTH 3")
         ) == {"nsites": 3}
+
+        assert transformer.transform(parser.parse("elements_ratios LENGTH 3")) == {
+            "ratios": {"$size": 3}
+        }
+
         assert transformer.transform(
             parser.parse("cartesian_site_positions LENGTH >= 10")
         ) == {"nsites": {"$gte": 10}}
@@ -515,48 +554,56 @@ class TestMongoTransformer:
             "my_elements": {"$in": ["Ag"]}
         }
 
-        assert transformer.transform(parser.parse("chemsys LENGTH 3")) == {"nelem": 3}
+        assert transformer.transform(parser.parse("_exmpl_chemsys LENGTH 3")) == {
+            "nelem": 3
+        }
 
     def test_aliases(self, mapper):
         """Test that valid aliases are allowed, but do not affect r-values."""
         from optimade.filtertransformers.mongo import MongoTransformer
 
-        class MyStructureMapper(mapper("BaseResourceMapper")):
+        class MyStructureMapper(mapper("StructureMapper")):
             ALIASES = (
                 ("elements", "my_elements"),
                 ("A", "D"),
+                ("property_a", "D"),
                 ("B", "E"),
                 ("C", "F"),
                 ("_exmpl_nested_field", "nested_field"),
             )
 
-        mapper = MyStructureMapper()
+            PROVIDER_FIELDS = ("D", "E", "F", "nested_field")
+
+        mapper = MyStructureMapper
         t = MongoTransformer(mapper=mapper)
+        p = LarkParser(version=self.version, variant=self.variant)
 
         assert mapper.get_backend_field("elements") == "my_elements"
-
-        test_filter = {"elements": {"$in": ["A", "B", "C"]}}
-        assert t.postprocess(test_filter) == {"my_elements": {"$in": ["A", "B", "C"]}}
-        test_filter = {"$and": [{"elements": {"$in": ["A", "B", "C"]}}]}
-        assert t.postprocess(test_filter) == {
-            "$and": [{"my_elements": {"$in": ["A", "B", "C"]}}]
+        test_filter = 'elements HAS "A"'
+        assert t.transform(p.parse(test_filter)) == {"my_elements": {"$in": ["A"]}}
+        test_filter = 'elements HAS ANY "A","B","C" AND elements HAS "D"'
+        assert t.transform(p.parse(test_filter)) == {
+            "$and": [
+                {"my_elements": {"$in": ["A", "B", "C"]}},
+                {"my_elements": {"$in": ["D"]}},
+            ]
         }
-        test_filter = {"elements": "A"}
-        assert t.postprocess(test_filter) == {"my_elements": "A"}
-        test_filter = ["A", "B", "C"]
-        assert t.postprocess(test_filter) == ["A", "B", "C"]
-
-        test_filter = ["A", "elements", "C"]
-        assert t.postprocess(test_filter) == ["A", "elements", "C"]
+        test_filter = 'elements = "A"'
+        assert t.transform(p.parse(test_filter)) == {"my_elements": {"$eq": "A"}}
+        test_filter = 'property_a HAS "B"'
+        assert t.transform(p.parse(test_filter)) == {"D": {"$in": ["B"]}}
 
-        test_filter = {"_exmpl_nested_field.sub_property": {"$gt": 1234.5}}
-        assert t.postprocess(test_filter) == {
+        test_filter = "_exmpl_nested_field.sub_property > 1234.5"
+        assert t.transform(p.parse(test_filter)) == {
             "nested_field.sub_property": {"$gt": 1234.5}
         }
 
-        test_filter = {"_exmpl_nested_field.sub_property.x": {"$exists": False}}
-        assert t.postprocess(test_filter) == {
-            "nested_field.sub_property.x": {"$exists": False}
+        test_filter = "_exmpl_nested_field.sub_property.x IS UNKNOWN"
+        assert t.transform(p.parse(test_filter)) == {
+            "$or": [
+                {"nested_field.sub_property.x": {"$exists": False}},
+                {"nested_field.sub_property.x": {"$eq": None}},
+            ]
         }
 
     def test_list_properties(self):
@@ -580,7 +627,12 @@ class TestMongoTransformer:
             "$and": [
                 {"elements": {"$in": ["H"]}},
                 {"elements": {"$all": ["H", "He", "Ga", "Ta"]}},
-                {"elements": {"$all": ["H", "He", "Ga", "Ta"], "$size": 4}},
+                {
+                    "elements": {
+                        "$all": ["H", "He", "Ga", "Ta"],
+                        "$size": 4,
+                    }
+                },
                 {"elements": {"$in": ["H", "He", "Ga", "Ta"]}},
             ]
         }
diff --git a/tests/server/conftest.py b/tests/server/conftest.py
index 292c030d..5353f298 100644
--- a/tests/server/conftest.py
+++ b/tests/server/conftest.py
@@ -1,4 +1,4 @@
-from typing import Union
+from typing import Union, Dict
 
 import pytest
 
@@ -109,6 +109,7 @@ def check_response(get_good_response):
         page_limit: int = CONFIG.page_limit,
         expected_return: int = None,
         expected_as_is: bool = False,
+        expected_warnings: List[Dict[str, str]] = None,
         server: Union[str, OptimadeTestClient] = "regular",
     ):
         response = get_good_response(request, server)
@@ -128,6 +129,15 @@ def check_response(get_good_response):
         else:
             assert expected_ids == response_ids
 
+        if expected_warnings:
+            assert "warnings" in response["meta"]
+            assert len(expected_warnings) == len(response["meta"]["warnings"])
+            for ind, warn in enumerate(expected_warnings):
+                for key in warn:
+                    assert response["meta"]["warnings"][ind][key] == warn[key]
+        else:
+            assert "warnings" not in response["meta"]
+
     return inner
 
 
diff --git a/tests/server/query_params/conftest.py b/tests/server/query_params/conftest.py
index 3b81d379..bae88288 100644
--- a/tests/server/query_params/conftest.py
+++ b/tests/server/query_params/conftest.py
@@ -74,10 +74,10 @@ def check_required_fields_response(get_good_response):
         expected_fields |= (
             get_mapper[endpoint].get_required_fields() - known_unused_fields
         )
-        expected_fields.add("attributes")
         request = f"/{endpoint}?response_fields={','.join(expected_fields)}"
 
         response = get_good_response(request, server)
+        expected_fields.add("attributes")
 
         response_fields = set()
         for entry in response["data"]:
diff --git a/tests/server/query_params/test_filter.py b/tests/server/query_params/test_filter.py
index 6a7771cc..2ad9d096 100644
--- a/tests/server/query_params/test_filter.py
+++ b/tests/server/query_params/test_filter.py
@@ -149,7 +149,7 @@ def test_list_length_comparisons_aliased(check_response, check_error_response):
             request,
             expected_status=501,
             expected_title="NotImplementedError",
-            expected_detail="LENGTH is not supported for nsites",
+            expected_detail="LENGTH is not supported for 'nsites'",
         )
     else:
         check_response(request, expected_ids)
@@ -192,6 +192,10 @@ def test_list_has_only(check_response):
     expected_ids = ["mpf_1"]
     check_response(request, expected_ids)
 
+    request = '/structures?filter=elements HAS ONLY "Ac" AND nelements IS KNOWN'
+    expected_ids = ["mpf_1"]
+    check_response(request, expected_ids)
+
 
 def test_list_correlated(check_error_response):
     request = '/structures?filter=elements:elements_ratios HAS "Ag":"0.2"'
@@ -206,39 +210,57 @@ def test_list_correlated(check_error_response):
 
 
 def test_timestamp_query(check_response):
-    from optimade.server.warnings import TimestampNotRFCCompliant
 
     request = '/structures?filter=last_modified="2019-06-08T05:13:37.331Z"&page_limit=5'
     expected_ids = ["mpf_1", "mpf_2", "mpf_3"]
-    if CONFIG.database_backend == SupportedBackend.ELASTIC:
-        check_response(request, expected_ids, expected_as_is=True)
-    else:
-        with pytest.warns(TimestampNotRFCCompliant):
-            check_response(request, expected_ids, expected_as_is=True)
+    expected_warnings = None
+    if CONFIG.database_backend in (
+        SupportedBackend.MONGOMOCK,
+        SupportedBackend.MONGODB,
+    ):
+        expected_warnings = [{"title": "TimestampNotRFCCompliant"}]
+    check_response(
+        request, expected_ids, expected_as_is=True, expected_warnings=expected_warnings
+    )
 
     request = '/structures?filter=last_modified<"2019-06-08T05:13:37.331Z"&page_limit=5'
     expected_ids = ["mpf_3819"]
-    if CONFIG.database_backend == SupportedBackend.ELASTIC:
-        check_response(request, expected_ids, expected_as_is=True)
-    else:
-        with pytest.warns(TimestampNotRFCCompliant):
-            check_response(request, expected_ids, expected_as_is=True)
+    expected_warnings = None
+    if CONFIG.database_backend in (
+        SupportedBackend.MONGOMOCK,
+        SupportedBackend.MONGODB,
+    ):
+        expected_warnings = [{"title": "TimestampNotRFCCompliant"}]
+    check_response(
+        request, expected_ids, expected_as_is=True, expected_warnings=expected_warnings
+    )
 
     request = '/structures?filter=last_modified="2018-06-08T05:13:37.945Z"&page_limit=5'
     expected_ids = ["mpf_3819"]
-    if CONFIG.database_backend == SupportedBackend.ELASTIC:
-        check_response(request, expected_ids, expected_as_is=True)
-    else:
-        with pytest.warns(TimestampNotRFCCompliant):
-            check_response(request, expected_ids, expected_as_is=True)
+    expected_warnings = None
+    if CONFIG.database_backend in (
+        SupportedBackend.MONGOMOCK,
+        SupportedBackend.MONGODB,
+    ):
+        expected_warnings = [{"title": "TimestampNotRFCCompliant"}]
+    check_response(
+        request, expected_ids, expected_as_is=True, expected_warnings=expected_warnings
+    )
 
     request = '/structures?filter=last_modified>"2018-06-08T05:13:37.945Z" AND last_modified<="2019-06-08T05:13:37.331Z"&page_limit=5'
     expected_ids = ["mpf_1", "mpf_2", "mpf_3"]
-    if CONFIG.database_backend == SupportedBackend.ELASTIC:
-        check_response(request, expected_ids, expected_as_is=True)
-    else:
-        with pytest.warns(TimestampNotRFCCompliant):
-            check_response(request, expected_ids, expected_as_is=True)
+    expected_warnings = None
+    if CONFIG.database_backend in (
+        SupportedBackend.MONGOMOCK,
+        SupportedBackend.MONGODB,
+    ):
+        expected_warnings = [
+            {"title": "TimestampNotRFCCompliant"},
+            {"title": "TimestampNotRFCCompliant"},
+        ]
+    check_response(
+        request, expected_ids, expected_as_is=True, expected_warnings=expected_warnings
+    )
 
 
 def test_is_known(check_response):
@@ -378,9 +400,9 @@ def test_filter_on_relationships(check_response, check_error_response):
     if CONFIG.database_backend == SupportedBackend.ELASTIC:
         check_error_response(
             request,
-            expected_status=400,
-            expected_title="Bad Request",
-            expected_detail="references is not a searchable quantity",
+            expected_status=501,
+            expected_title="NotImplementedError",
+            expected_detail="Unable to filter on relationships with type 'references'",
         )
         pytest.xfail("Elasticsearch backend does not support relationship filtering.")
     check_response(request, expected_ids)
@@ -401,3 +423,81 @@ def test_filter_on_relationships(check_response, check_error_response):
         expected_title="NotImplementedError",
         expected_detail=error_detail,
     )
+
+
+@pytest.mark.xfail(
+    CONFIG.database_backend == SupportedBackend.MONGOMOCK,
+    reason=(
+        "mongomock<=3.22.1 has a bug that causes {'$size': 1} queries on missing fields to match all documents: "
+        "(https://github.com/mongomock/mongomock/issues/710). This check can be removed once mongomock 3.22.2 has been released."
+    ),
+)
+def test_filter_on_unknown_fields(check_response, check_error_response):
+
+    request = "/structures?filter=unknown_field = 1"
+    error_detail = "'unknown_field' is not a known or searchable quantity"
+    check_error_response(
+        request,
+        expected_status=400,
+        expected_title="Bad Request",
+        expected_detail=error_detail,
+    )
+
+    request = "/structures?filter=_exmpl_unknown_field = 1"
+    error_detail = "'_exmpl_unknown_field' is not a known or searchable quantity"
+    check_error_response(
+        request,
+        expected_status=400,
+        expected_title="Bad Request",
+        expected_detail=error_detail,
+    )
+
+    request = "/structures?filter=_exmpl_unknown_field LENGTH 1"
+    error_detail = "'_exmpl_unknown_field' is not a known or searchable quantity"
+    check_error_response(
+        request,
+        expected_status=400,
+        expected_title="Bad Request",
+        expected_detail=error_detail,
+    )
+
+    request = "/structures?filter=_exmpl1_unknown_field = 1"
+    expected_ids = []
+    expected_warnings = [
+        {
+            "title": "UnknownProviderProperty",
+            "detail": "Field '_exmpl1_unknown_field' has an unrecognised prefix: this property has been treated as UNKNOWN.",
+        }
+    ]
+    check_response(
+        request, expected_ids=expected_ids, expected_warnings=expected_warnings
+    )
+
+    request = "/structures?filter=_exmpl1_unknown_field LENGTH 1"
+    expected_ids = []
+    expected_warnings = [
+        {
+            "title": "UnknownProviderProperty",
+            "detail": "Field '_exmpl1_unknown_field' has an unrecognised prefix: this property has been treated as UNKNOWN.",
+        }
+    ]
+    check_response(
+        request, expected_ids=expected_ids, expected_warnings=expected_warnings
+    )
+
+    request = '/structures?filter=_exmpl1_unknown_field HAS "Si"'
+    expected_ids = []
+    expected_warnings = [
+        {
+            "title": "UnknownProviderProperty",
+            "detail": "Field '_exmpl1_unknown_field' has an unrecognised prefix: this property has been treated as UNKNOWN.",
+        }
+    ]
+    check_response(
+        request, expected_ids=expected_ids, expected_warnings=expected_warnings
+    )
+
+    # Should not warn as the "_optimade" prefix is registered
+    request = "/structures?filter=_optimade_random_field = 1"
+    expected_ids = []
+    check_response(request, expected_ids=expected_ids)
diff --git a/tests/server/query_params/test_response_fields.py b/tests/server/query_params/test_response_fields.py
index 58c3e3d8..e55394a7 100644
--- a/tests/server/query_params/test_response_fields.py
+++ b/tests/server/query_params/test_response_fields.py
@@ -25,3 +25,43 @@ def test_required_fields_structures(check_required_fields_response):
     non_used_top_level_fields = {"links"}
     expected_fields = {"elements", "nelements"}
     check_required_fields_response(endpoint, non_used_top_level_fields, expected_fields)
+
+
+def test_unknown_field_structures(
+    check_required_fields_response, check_error_response, check_response, structures
+):
+    """Check that unknown fields are returned as `null` in entries."""
+    endpoint = "structures"
+    non_used_top_level_fields = {"links"}
+
+    expected_fields = {"_optimade_field"}
+    check_required_fields_response(endpoint, non_used_top_level_fields, expected_fields)
+
+    expected_fields = {"optimade_field"}
+    check_error_response(
+        request=f"/{endpoint}?response_fields={','.join(expected_fields)}",
+        expected_status=400,
+        expected_title="Bad Request",
+        expected_detail="Unrecognised OPTIMADE field(s) in requested `response_fields`: {'optimade_field'}.",
+    )
+
+    expected_fields = {"_exmpl_optimade_field"}
+    expected_ids = [doc["task_id"] for doc in structures]
+    expected_warnings = [
+        {
+            "title": "UnknownProviderProperty",
+            "detail": "Unrecognised field(s) for this provider requested in `response_fields`: {'_exmpl_optimade_field'}.",
+        }
+    ]
+    check_response(
+        request=f"/{endpoint}?response_fields={','.join(expected_fields)}",
+        expected_ids=expected_ids,
+        expected_warnings=expected_warnings,
+    )
+
+    expected_fields = {"_exmpl123_optimade_field"}
+    expected_ids = [doc["task_id"] for doc in structures]
+    check_response(
+        request=f"/{endpoint}?response_fields={','.join(expected_fields)}",
+        expected_ids=expected_ids,
+    )
diff --git a/tests/server/routers/test_structures.py b/tests/server/routers/test_structures.py
index b6258f84..05b1a9f4 100644
--- a/tests/server/routers/test_structures.py
+++ b/tests/server/routers/test_structures.py
@@ -183,3 +183,27 @@ class TestStructuresWithNullFieldsMatchUnknownFilter(RegularEndpointTests):
     def test_structures_endpoint_data(self):
         """Check that all structures are returned."""
         assert len(self.json_response["data"]) == 17
+
+
+class TestStructuresWithUnknownResponseFields(RegularEndpointTests):
+    """Tests that structures with e.g., `'assemblies':null` do get
+    returned for queries testing for "UNKNOWN" fields.
+
+    """
+
+    request_str = "/structures?filter=assemblies IS UNKNOWN&response_fields=assemblies,_other_provider_field,chemical_formula_anonymous"
+    response_cls = StructureResponseMany
+
+    def test_structures_endpoint_data(self):
+        """Check that all structures are returned."""
+        assert len(self.json_response["data"]) == 17
+        keys = ("_other_provider_field", "assemblies", "chemical_formula_anonymous")
+        for key in keys:
+            assert all(key in doc["attributes"] for doc in self.json_response["data"])
+        assert all(
+            doc["attributes"]["_other_provider_field"] is None
+            for doc in self.json_response["data"]
+        )
+        assert all(
+            len(doc["attributes"]) == len(keys) for doc in self.json_response["data"]
+        )