Repository: Materials-Consortia/optimade-python-tools
Pull request #1509 (2023-02-09 19:55:56)

Add ASE ingester and generalize other ingestion utilities
This PR adds `from_ase` as a new ingester type (following #1296 for pymatgen) and generalises the ingester functionality into a `.ingest_from` method of the base adapter, e.g.,

```python

from optimade.adapters import Structure

atoms = ase.Atoms(...)
structure = pymatgen.core.Structure(...)

Structure.ingest_from(atoms)  # implicit type detection
Structure.ingest_from(structure)  # implicit type detection
Structure.ingest_from(atoms, "ase")  # use key into ingester dict to specify
Structure.ingest_from(pymatgen, "pymatgen")  # use key into ingester dict to specify

```

Also adds several utilities for e.g., normalizing formulae and other fields. These are now used in the adapters and spread across various utils modules (to avoid circular imports).

codecov[bot]: # [Codecov](https://codecov.io/gh/Materials-Consortia/optimade-python-tools/pull/1509?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Materials-Consortia) Report
> Merging [#1509](https://codecov.io/gh/Materials-Consortia/optimade-python-tools/pull/1509?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Materials-Consortia) (57f5675) into [master](https://codecov.io/gh/Materials-Consortia/optimade-python-tools/commit/7c8f9a63ff169fd1e1c90676e304ca9c99e5b939?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Materials-Consortia) (7c8f9a6) will **increase** coverage by `0.01%`.
> The diff coverage is `92.30%`.

```diff
@@            Coverage Diff             @@
##           master    #1509      +/-   ##
==========================================
+ Coverage   90.95%   90.97%   +0.01%     
==========================================
  Files          74       74              
  Lines        4411     4464      +53     
==========================================
+ Hits         4012     4061      +49     
- Misses        399      403       +4     
```

| Flag | Coverage Δ | |
|---|---|---|
| project | `90.97% <92.30%> (+0.01%)` | :arrow_up: |
| validator | `90.32% <92.30%> (+0.02%)` | :arrow_up: |

Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Materials-Consortia#carryforward-flags-in-the-pull-request-comment) to find out more.

| [Impacted Files](https://codecov.io/gh/Materials-Consortia/optimade-python-tools/pull/1509?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Materials-Consortia) | Coverage Δ | |
|---|---|---|
| [optimade/adapters/base.py](https://codecov.io/gh/Materials-Consortia/optimade-python-tools/pull/1509?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Materials-Consortia#diff-b3B0aW1hZGUvYWRhcHRlcnMvYmFzZS5weQ==) | `96.96% <83.33%> (-3.04%)` | :arrow_down: |
| [optimade/models/utils.py](https://codecov.io/gh/Materials-Consortia/optimade-python-tools/pull/1509?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Materials-Consortia#diff-b3B0aW1hZGUvbW9kZWxzL3V0aWxzLnB5) | `92.50% <91.66%> (-0.29%)` | :arrow_down: |
| [optimade/adapters/structures/ase.py](https://codecov.io/gh/Materials-Consortia/optimade-python-tools/pull/1509?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Materials-Consortia#diff-b3B0aW1hZGUvYWRhcHRlcnMvc3RydWN0dXJlcy9hc2UucHk=) | `96.77% <92.59%> (-3.23%)` | :arrow_down: |
| [optimade/adapters/structures/adapter.py](https://codecov.io/gh/Materials-Consortia/optimade-python-tools/pull/1509?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Materials-Consortia#diff-b3B0aW1hZGUvYWRhcHRlcnMvc3RydWN0dXJlcy9hZGFwdGVyLnB5) | `100.00% <100.00%> (ø)` | |
| [optimade/adapters/structures/pymatgen.py](https://codecov.io/gh/Materials-Consortia/optimade-python-tools/pull/1509?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Materials-Consortia#diff-b3B0aW1hZGUvYWRhcHRlcnMvc3RydWN0dXJlcy9weW1hdGdlbi5weQ==) | `98.50% <100.00%> (-0.16%)` | :arrow_down: |
| [optimade/adapters/structures/utils.py](https://codecov.io/gh/Materials-Consortia/optimade-python-tools/pull/1509?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Materials-Consortia#diff-b3B0aW1hZGUvYWRhcHRlcnMvc3RydWN0dXJlcy91dGlscy5weQ==) | `80.74% <100.00%> (+0.74%)` | :arrow_up: |
| [optimade/models/structures.py](https://codecov.io/gh/Materials-Consortia/optimade-python-tools/pull/1509?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Materials-Consortia#diff-b3B0aW1hZGUvbW9kZWxzL3N0cnVjdHVyZXMucHk=) | `96.42% <100.00%> (+0.67%)` | :arrow_up: |

Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Materials-Consortia). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Materials-Consortia)

ml-evs: Hi @JPBergsma, will merge this tomorrow, let me know if you want to have a second look first and I'll hold off.

Patch:
diff --git a/README.md b/README.md
index 3dea2fe7..7183842d 100644
--- a/README.md
+++ b/README.md
@@ -4,10 +4,7 @@
 <img width="100px" align="center" src="https://matsci.org/uploads/default/original/2X/b/bd2f59b3bf14fb046b74538750699d7da4c19ac1.svg">
 </div>
 
-<h1 align="center">
-OPTIMADE Python tools
-</h1>
-
+# <div align="center">OPTIMADE Python tools</div>
 
 <div align="center">
 
@@ -50,6 +47,7 @@ This is to enable interoperability among databases that serve crystal structures
 This repository contains a library of tools for implementing and consuming [OPTIMADE APIs](https://www.optimade.org) using Python:
 
 1. [pydantic](https://github.com/pydantic/pydantic) data models for all [OPTIMADE entry types](https://www.optimade.org/optimade-python-tools/latest/all_models/) and endpoint responses, and a [Lark](https://github.com/lark-parser/lark) [EBNF grammar](https://en.wikipedia.org/wiki/Extended_Backus%E2%80%93Naur_form) implementation for the OPTIMADE filter language.
+1. Adapters to map OPTIMADE data to and from many commonly used atomistic Python frameworks (e.g., [pymatgen](https://pymatgen.org/), [ASE](https://wiki.fysik.dtu.dk/ase/)) and crystallographic file types (e.g., [CIF](https://www.iucr.org/resources/cif)), using the `optimade.adapters` module.
 1. A configurable reference server implementation that can make use of either MongoDB or Elasticsearch database backends out-of-the-box, and is readily extensible to other backends. Try it out on the [demo site](https://optimade.fly.dev)! The OpenAPI schemas of the server are used to construct the [OPTIMADE schemas](https://schemas.optimade.org/) site.
 1. An [OPTIMADE client](https://www.optimade.org/optimade-python-tools/latest/getting_started/client/) (`optimade-get`) that can query multiple [OPTIMADE providers](https://optimade.org/providers-dashboard) concurrently with a given filter, at the command-line or from Python code.
 1. A fuzzy API validator tool, which may be called from the shell (`optimade-validator`) or used as a GitHub Action from [optimade-validator-action](https://github.com/Materials-Consortia/optimade-validator-action); this validator is used to construct the [providers dashboard](https://optimade.org/providers-dashboard).
diff --git a/optimade/adapters/base.py b/optimade/adapters/base.py
index d0849087..f1d4bcb6 100644
--- a/optimade/adapters/base.py
+++ b/optimade/adapters/base.py
@@ -34,6 +34,9 @@ class EntryAdapter:
     Attributes:
         ENTRY_RESOURCE: Entry resource to store entry as.
         _type_converters: Dictionary of valid conversion types for entry.
+        _type_ingesters: Dictionary of valid ingestion types mapped to ingestion functions.
+        _type_ingesters_by_type: Dictionary mapping the keys of `_type_ingesters` to data
+            types that can be ingested.
         as_<_type_converters>: Convert entry to a type listed in `_type_converters`.
         from_<_type_converters>: Convert an external type to the corresponding OPTIMADE model.
 
@@ -42,6 +45,7 @@ class EntryAdapter:
     ENTRY_RESOURCE: Type[EntryResource] = EntryResource
     _type_converters: Dict[str, Callable] = {}
     _type_ingesters: Dict[str, Callable] = {}
+    _type_ingesters_by_type: Dict[str, Type] = {}
 
     def __init__(self, entry: dict) -> None:
         """
@@ -116,6 +120,48 @@ class EntryAdapter:
 
         return self._converted[format]
 
+    @classmethod
+    def ingest_from(cls, data: Any, format: Optional[str] = None) -> Any:
+        """Convert desired format to OPTIMADE format.
+
+        Parameters:
+            data (Any): The data to convert.
+            format (str): Type or format to which the entry should be converted.
+
+        Raises:
+            AttributeError: If `format` can not be found in `_type_ingesters`.
+
+        Returns:
+            The ingested Structure.
+
+        """
+
+        if format is None:
+            for key, instance_type in cls._type_ingesters_by_type.items():
+                if isinstance(data, instance_type):
+                    format = key
+                    break
+
+            else:
+                raise AttributeError(
+                    f"Non entry type to data of type {type(data)} from.\n"
+                    f"Valid entry types: {tuple(cls._type_ingesters.keys())}"
+                )
+
+        if format not in cls._type_ingesters:
+            raise AttributeError(
+                f"Non-valid entry type to ingest from: {format}\n"
+                f"Valid entry types: {tuple(cls._type_ingesters.keys())}"
+            )
+
+        return cls(
+            {
+                "attributes": cls._type_ingesters[format](data).dict(),
+                "id": "",
+                "type": "structures",
+            }
+        )
+
     @staticmethod
     def _get_model_attributes(
         starting_instances: Union[Tuple[BaseModel, ...], List[BaseModel]], name: str
diff --git a/optimade/adapters/structures/adapter.py b/optimade/adapters/structures/adapter.py
index 5e9d8780..4ae62249 100644
--- a/optimade/adapters/structures/adapter.py
+++ b/optimade/adapters/structures/adapter.py
@@ -4,10 +4,12 @@ from optimade.adapters.base import EntryAdapter
 from optimade.models import StructureResource
 
 from .aiida import get_aiida_structure_data
-from .ase import get_ase_atoms
+from .ase import Atoms as ASEAtoms
+from .ase import from_ase_atoms, get_ase_atoms
 from .cif import get_cif
 from .jarvis import get_jarvis_atoms
 from .proteindatabank import get_pdb, get_pdbx_mmcif
+from .pymatgen import Structure as PymatgenStructure
 from .pymatgen import from_pymatgen, get_pymatgen
 
 
@@ -55,4 +57,10 @@ class Structure(EntryAdapter):
 
     _type_ingesters: Dict[str, Callable] = {
         "pymatgen": from_pymatgen,
+        "ase": from_ase_atoms,
+    }
+
+    _type_ingesters_by_type: Dict[str, Type] = {
+        "pymatgen": PymatgenStructure,
+        "ase": ASEAtoms,
     }
diff --git a/optimade/adapters/structures/ase.py b/optimade/adapters/structures/ase.py
index 93ea84bf..020487ed 100644
--- a/optimade/adapters/structures/ase.py
+++ b/optimade/adapters/structures/ase.py
@@ -10,10 +10,17 @@ For more information on the ASE code see [their documentation](https://wiki.fysi
 from typing import Dict
 
 from optimade.adapters.exceptions import ConversionError
-from optimade.adapters.structures.utils import species_from_species_at_sites
+from optimade.adapters.structures.utils import (
+    elements_ratios_from_species_at_sites,
+    species_from_species_at_sites,
+)
 from optimade.models import Species as OptimadeStructureSpecies
 from optimade.models import StructureFeatures
 from optimade.models import StructureResource as OptimadeStructure
+from optimade.models.structures import StructureResourceAttributes
+from optimade.models.utils import anonymize_formula, reduce_formula
+
+EXTRA_FIELD_PREFIX = "ase"
 
 try:
     from ase import Atom, Atoms
@@ -26,7 +33,7 @@ except (ImportError, ModuleNotFoundError):
     ASE_NOT_FOUND = "ASE not found, cannot convert structure to an ASE Atoms"
 
 
-__all__ = ("get_ase_atoms",)
+__all__ = ("get_ase_atoms", "from_ase_atoms")
 
 
 def get_ase_atoms(optimade_structure: OptimadeStructure) -> Atoms:
@@ -82,6 +89,69 @@ def get_ase_atoms(optimade_structure: OptimadeStructure) -> Atoms:
 
         atoms.append(Atom(symbol=species_name, position=site, mass=mass))
 
+    info = {}
+    for key in attributes.dict().keys():
+        if key.startswith("_"):
+            ase_key = key
+            if key.startswith(f"_{EXTRA_FIELD_PREFIX}_"):
+                ase_key = "".join(key.split(f"_{EXTRA_FIELD_PREFIX}_")[1:])
+            info[ase_key] = getattr(attributes, key)
+
     return Atoms(
-        symbols=atoms, cell=attributes.lattice_vectors, pbc=attributes.dimension_types
+        symbols=atoms,
+        cell=attributes.lattice_vectors,
+        pbc=attributes.dimension_types,
+        info=info if info else None,
+    )
+
+
+def from_ase_atoms(atoms: Atoms) -> StructureResourceAttributes:
+    """Convert an ASE `Atoms` object into an OPTIMADE `StructureResourceAttributes` model.
+
+    Parameters:
+        atoms: The ASE `Atoms` object to convert.
+
+    Returns:
+        An OPTIMADE `StructureResourceAttributes` model, which can be converted to a raw Python
+            dictionary with `.dict()` or to JSON with `.json()`.
+
+    """
+    if not isinstance(atoms, Atoms):
+        raise RuntimeError(
+            f"Cannot convert type {type(atoms)} into an OPTIMADE `StructureResourceAttributes` model."
+        )
+
+    attributes = {}
+    attributes["cartesian_site_positions"] = atoms.positions.tolist()
+    attributes["lattice_vectors"] = atoms.cell.tolist()
+    attributes["species_at_sites"] = atoms.get_chemical_symbols()
+    attributes["elements_ratios"] = elements_ratios_from_species_at_sites(
+        attributes["species_at_sites"]
+    )
+    attributes["species"] = species_from_species_at_sites(
+        attributes["species_at_sites"]
     )
+    attributes["dimension_types"] = [int(_) for _ in atoms.pbc.tolist()]
+    attributes["nperiodic_dimensions"] = sum(attributes["dimension_types"])
+    attributes["nelements"] = len(attributes["species"])
+    attributes["elements"] = sorted([_.name for _ in attributes["species"]])
+    attributes["nsites"] = len(attributes["species_at_sites"])
+
+    attributes["chemical_formula_descriptive"] = atoms.get_chemical_formula()
+    attributes["chemical_formula_reduced"] = reduce_formula(
+        atoms.get_chemical_formula()
+    )
+    attributes["chemical_formula_anonymous"] = anonymize_formula(
+        attributes["chemical_formula_reduced"],
+    )
+    attributes["last_modified"] = None
+    attributes["immutable_id"] = None
+    attributes["structure_features"] = []
+
+    for key in atoms.info:
+        optimade_key = key.lower()
+        if not key.startswith(f"_{EXTRA_FIELD_PREFIX}"):
+            optimade_key = f"_{EXTRA_FIELD_PREFIX}_{optimade_key}"
+        attributes[optimade_key] = atoms.info[key]
+
+    return StructureResourceAttributes(**attributes)
diff --git a/optimade/adapters/structures/pymatgen.py b/optimade/adapters/structures/pymatgen.py
index 274c40f5..8d46bf7f 100644
--- a/optimade/adapters/structures/pymatgen.py
+++ b/optimade/adapters/structures/pymatgen.py
@@ -16,9 +16,10 @@ from optimade.adapters.structures.utils import (
 from optimade.models import Species as OptimadeStructureSpecies
 from optimade.models import StructureResource as OptimadeStructure
 from optimade.models import StructureResourceAttributes
+from optimade.models.utils import anonymize_formula, reduce_formula
 
 try:
-    from pymatgen.core import Composition, Lattice, Molecule, Structure
+    from pymatgen.core import Lattice, Molecule, Structure
 
 except (ImportError, ModuleNotFoundError):
     from warnings import warn
@@ -168,14 +169,14 @@ def from_pymatgen(pmg_structure: Structure) -> StructureResourceAttributes:
     attributes["dimension_types"] = [int(_) for _ in pmg_structure.lattice.pbc]
     attributes["nperiodic_dimensions"] = sum(attributes["dimension_types"])
     attributes["nelements"] = len(pmg_structure.composition.elements)
-    attributes["chemical_formula_anonymous"] = _pymatgen_anonymized_formula_to_optimade(
-        pmg_structure.composition
+    attributes["chemical_formula_anonymous"] = anonymize_formula(
+        pmg_structure.composition.formula
     )
     attributes["elements"] = sorted(
         [_.symbol for _ in pmg_structure.composition.elements]
     )
-    attributes["chemical_formula_reduced"] = _pymatgen_reduced_formula_to_optimade(
-        pmg_structure.composition
+    attributes["chemical_formula_reduced"] = reduce_formula(
+        pmg_structure.composition.formula
     )
     attributes["chemical_formula_descriptive"] = pmg_structure.composition.formula
     attributes["elements_ratios"] = [
@@ -188,33 +189,3 @@ def from_pymatgen(pmg_structure: Structure) -> StructureResourceAttributes:
     attributes["structure_features"] = []
 
     return StructureResourceAttributes(**attributes)
-
-
-def _pymatgen_anonymized_formula_to_optimade(composition: Composition) -> str:
-    """Construct an OPTIMADE `chemical_formula_anonymous` from a pymatgen `Composition`."""
-    import re
-
-    from optimade.models.utils import anonymous_element_generator
-
-    return "".join(
-        [
-            "".join(x)
-            for x in zip(
-                anonymous_element_generator(),
-                reversed(re.split("[A-Z]", composition.anonymized_formula)[1:]),
-            )
-        ]
-    )
-
-
-def _pymatgen_reduced_formula_to_optimade(composition: Composition) -> str:
-    """Construct an OPTIMADE `chemical_formula_reduced` from a pymatgen `Composition`."""
-    import numpy
-
-    numbers = [int(_) for _ in composition.to_reduced_dict.values()]
-    gcd = numpy.gcd.reduce(numbers)
-    return "".join(
-        _
-        + f"{int(composition.to_reduced_dict[_]) // gcd if composition.to_reduced_dict[_] // gcd > 1 else ''}"
-        for _ in sorted([_.symbol for _ in composition.elements])
-    )
diff --git a/optimade/adapters/structures/utils.py b/optimade/adapters/structures/utils.py
index 0c2bf9f3..2b36570b 100644
--- a/optimade/adapters/structures/utils.py
+++ b/optimade/adapters/structures/utils.py
@@ -355,3 +355,14 @@ def species_from_species_at_sites(
         OptimadeStructureSpecies(name=_, concentration=[1.0], chemical_symbols=[_])
         for _ in set(species_at_sites)
     ]
+
+
+def elements_ratios_from_species_at_sites(species_at_sites: List[str]) -> List[float]:
+    """Compute the OPTIMADE `elements_ratios` field from `species_at_sites` in the case where `species_at_sites` refers
+    to sites wholly occupied by the given elements, e.g., not arbitrary species labels or with partial/mixed occupancy.
+
+    """
+    elements = set(species_at_sites)
+    counts = {e: species_at_sites.count(e) for e in elements}
+    num_sites = len(species_at_sites)
+    return [counts[e] / num_sites for e in sorted(elements)]
diff --git a/optimade/models/structures.py b/optimade/models/structures.py
index c9424254..aa89afe6 100644
--- a/optimade/models/structures.py
+++ b/optimade/models/structures.py
@@ -1,10 +1,7 @@
 # pylint: disable=no-self-argument,line-too-long,no-name-in-module
-import math
 import re
-import sys
 import warnings
 from enum import Enum, IntEnum
-from functools import reduce
 from typing import List, Optional, Union
 
 from pydantic import BaseModel, conlist, root_validator, validator
@@ -18,6 +15,7 @@ from optimade.models.utils import (
     OptimadeField,
     StrictField,
     SupportLevel,
+    reduce_formula,
 )
 from optimade.warnings import MissingExpectedField
 
@@ -895,18 +893,10 @@ The properties of the species are found in the property `species`.
         if value is None:
             return value
 
-        numbers = [n.strip() or 1 for n in re.split(r"[A-Z][a-z]*", value)]
-        # Need to remove leading 1 from split and convert to ints
-        numbers = [int(n) for n in numbers[1:]]
-
-        if sys.version_info[1] >= 9:
-            gcd = math.gcd(*numbers)
-        else:
-            gcd = reduce(math.gcd, numbers)
-
-        if gcd != 1:
+        reduced_formula = reduce_formula(value)
+        if reduced_formula != value:
             raise ValueError(
-                f"{field.name} {value!r} is not properly reduced: greatest common divisor was {gcd}, expected 1."
+                f"{field.name} {value!r} is not properly reduced: expected {reduced_formula!r}."
             )
 
         return value
diff --git a/optimade/models/utils.py b/optimade/models/utils.py
index 0474cdf9..8967b6d5 100644
--- a/optimade/models/utils.py
+++ b/optimade/models/utils.py
@@ -1,9 +1,11 @@
 import inspect
 import itertools
+import math
 import re
 import warnings
 from enum import Enum
-from typing import TYPE_CHECKING, Optional
+from functools import reduce
+from typing import TYPE_CHECKING, List, Optional
 
 from pydantic import Field
 from pydantic.fields import FieldInfo
@@ -228,6 +230,63 @@ def anonymous_element_generator():
             yield "".join(s)
 
 
+def _reduce_or_anonymize_formula(
+    formula: str, alphabetize: bool = True, anonymize: bool = False
+) -> str:
+    """Takes an input formula, reduces it and either alphabetizes or anonymizes it."""
+    import re
+    import sys
+
+    numbers: List[int] = [
+        int(n.strip() or 1) for n in re.split(r"[A-Z][a-z]*", formula)[1:]
+    ]
+    # Need to remove leading 1 from split and convert to ints
+
+    species = re.findall("[A-Z][a-z]*", formula)
+
+    if sys.version_info[1] >= 9:
+        gcd = math.gcd(*numbers)
+    else:
+        gcd = reduce(math.gcd, numbers)
+
+    if not len(species) == len(numbers):
+        raise ValueError(f"Something is wrong with the input formula: {formula}")
+
+    numbers = [n // gcd for n in numbers]
+
+    if anonymize:
+        numbers = sorted(numbers, reverse=True)
+        species = [s for _, s in zip(numbers, anonymous_element_generator())]
+
+    elif alphabetize:
+        species, numbers = zip(*sorted(zip(species, numbers)))
+
+    return "".join(f"{s}{n if n != 1 else ''}" for n, s in zip(numbers, species))
+
+
+def anonymize_formula(formula: str) -> str:
+    """Takes a string representation of a chemical formula of the form `[A-Z][a-z]*[0-9]*` (potentially with whitespace) and
+    returns the OPTIMADE `chemical_formula_anonymous` representation, i.e., a reduced chemical formula comprising of element symbols
+    drawn from A, B, C... ordered from largest proportion to smallest.
+
+    Returns:
+        The anonymous chemical formula in the OPTIMADE representation.
+
+    """
+    return _reduce_or_anonymize_formula(formula, alphabetize=False, anonymize=True)
+
+
+def reduce_formula(formula: str) -> str:
+    """Takes a string representation of a chemical formula of the form `[A-Z][a-z]*[0-9]*` (potentially with whitespace) and
+    reduces it by the GCD of the proportion integers present in the formula, stripping any leftover "1" values.
+
+    Returns:
+        The reduced chemical formula in the OPTIMADE representation.
+
+    """
+    return _reduce_or_anonymize_formula(formula, alphabetize=True, anonymize=False)
+
+
 ANONYMOUS_ELEMENTS = tuple(itertools.islice(anonymous_element_generator(), 150))
 """ Returns the first 150 values of the anonymous element generator. """
 


Test patch:
diff --git a/tests/adapters/structures/test_ase.py b/tests/adapters/structures/test_ase.py
index a2062b2c..8e4103c8 100644
--- a/tests/adapters/structures/test_ase.py
+++ b/tests/adapters/structures/test_ase.py
@@ -45,3 +45,23 @@ def test_special_species(SPECIAL_SPECIES_STRUCTURES):
 def test_null_species(null_species_structure):
     """Make sure null species are handled"""
     assert isinstance(get_ase_atoms(null_species_structure), Atoms)
+
+
+def test_extra_info_keys(RAW_STRUCTURES):
+    """Test that provider fields/ASE metadata is preserved during conversion."""
+    structure = RAW_STRUCTURES[0]
+    structure["attributes"]["_ase_key"] = "some value"
+    structure["attributes"]["_ase_another_key"] = [1, 2, 3]
+    structure["attributes"]["_key_without_ase_prefix"] = [4, 5, 6]
+
+    atoms = Structure(structure).as_ase
+    assert atoms.info["key"] == "some value"
+    assert atoms.info["another_key"] == [1, 2, 3]
+    assert atoms.info["_key_without_ase_prefix"] == [4, 5, 6]
+
+    roundtrip_structure = Structure.ingest_from(atoms).attributes.dict()
+    assert roundtrip_structure["_ase_key"] == "some value"
+    assert roundtrip_structure["_ase_another_key"] == [1, 2, 3]
+
+    # This key should have the _ase prefix re-added
+    assert roundtrip_structure["_ase__key_without_ase_prefix"] == [4, 5, 6]
diff --git a/tests/adapters/structures/test_pymatgen.py b/tests/adapters/structures/test_pymatgen.py
index b27de65c..f51c6649 100644
--- a/tests/adapters/structures/test_pymatgen.py
+++ b/tests/adapters/structures/test_pymatgen.py
@@ -18,7 +18,6 @@ from optimade.adapters import Structure
 from optimade.adapters.structures.pymatgen import (
     _get_molecule,
     _get_structure,
-    from_pymatgen,
     get_pymatgen,
 )
 
@@ -56,29 +55,3 @@ def test_special_species(SPECIAL_SPECIES_STRUCTURES):
 def test_null_species(null_species_structure):
     """Make sure null species are handled"""
     assert isinstance(get_pymatgen(null_species_structure), PymatgenStructure)
-
-
-def test_successful_ingestion(RAW_STRUCTURES):
-    import numpy as np
-
-    lossy_keys = (
-        "chemical_formula_descriptive",
-        "chemical_formula_hill",
-        "last_modified",
-        "assemblies",
-        "attached",
-        "immutable_id",
-        "species",
-        "fractional_site_positions",
-    )
-    array_keys = ("cartesian_site_positions", "lattice_vectors")
-    for structure in RAW_STRUCTURES:
-        converted = from_pymatgen(get_pymatgen(Structure(structure))).dict()
-        for k in converted:
-            if k not in lossy_keys:
-                if k in array_keys:
-                    np.testing.assert_almost_equal(
-                        converted[k], structure["attributes"][k]
-                    )
-                else:
-                    assert converted[k] == structure["attributes"][k]
diff --git a/tests/adapters/structures/test_structures.py b/tests/adapters/structures/test_structures.py
index fd82033c..0a96f57b 100644
--- a/tests/adapters/structures/test_structures.py
+++ b/tests/adapters/structures/test_structures.py
@@ -1,4 +1,5 @@
 """Test Structure adapter"""
+
 import pytest
 
 from optimade.adapters import Structure
@@ -146,3 +147,88 @@ def test_common_converters(raw_structure, RAW_STRUCTURES):
         raw_structure_property_set = set(raw_structure.keys())
         resource_property_set = set(Structure(raw_structure).as_dict.keys())
         assert raw_structure_property_set.issubset(resource_property_set)
+
+
+def compare_lossy_conversion(structure_attributes, reconverted_structure_attributes):
+    """Compare two structures, allowing for some loss of information and mapping of prefixed keys."""
+
+    try:
+        import numpy as np
+    except ImportError:
+        pytest.node.warn(
+            pytest.PytestWarning(
+                "numpy not found, some cases of conversion tests will be skipped"
+            )
+        )
+        np = None
+
+    lossy_keys = (
+        "chemical_formula_descriptive",
+        "chemical_formula_hill",
+        "last_modified",
+        "assemblies",
+        "attached",
+        "immutable_id",
+        "species",
+        "fractional_site_positions",
+    )
+    array_keys = ("cartesian_site_positions", "lattice_vectors")
+
+    for k in reconverted_structure_attributes:
+        if k not in lossy_keys:
+            if k in array_keys and np is not None:
+                np.testing.assert_almost_equal(
+                    reconverted_structure_attributes[k], structure_attributes[k]
+                )
+            elif k.startswith("_"):
+                # ugly way of checking if a substring exists in the initial structure
+                for i in range(len(k)):
+                    subkey = k[i:]
+                    if subkey in structure_attributes:
+                        assert (
+                            reconverted_structure_attributes[k]
+                            == structure_attributes[subkey]
+                        )
+                        break
+                else:
+                    raise ValueError(f"No subkey of {k} was found in initial structure")
+
+            else:
+                assert reconverted_structure_attributes[k] == structure_attributes[k]
+
+
+@pytest.mark.parametrize(
+    "format",
+    [k for k in Structure._type_ingesters.keys() if k in Structure._type_converters],
+)
+def test_two_way_conversion(RAW_STRUCTURES, format):
+    for structure in RAW_STRUCTURES:
+        new_structure = Structure(structure)
+        converted_structure = new_structure.convert(format)
+        if converted_structure is None:
+            continue
+        reconverted_structure = Structure.ingest_from(
+            converted_structure, format
+        ).entry.dict()
+        compare_lossy_conversion(
+            structure["attributes"], reconverted_structure["attributes"]
+        )
+
+
+@pytest.mark.parametrize(
+    "format",
+    [k for k in Structure._type_ingesters.keys() if k in Structure._type_converters],
+)
+def test_two_way_conversion_with_implicit_type(RAW_STRUCTURES, format):
+    for structure in RAW_STRUCTURES:
+        new_structure = Structure(structure)
+        converted_structure = new_structure.convert(format)
+        if converted_structure is None:
+            continue
+        reconverted_structure = Structure.ingest_from(
+            converted_structure, format=None
+        ).entry.dict()
+
+        compare_lossy_conversion(
+            structure["attributes"], reconverted_structure["attributes"]
+        )
diff --git a/tests/adapters/structures/test_utils.py b/tests/adapters/structures/test_utils.py
index 5cb1e763..107c90b1 100644
--- a/tests/adapters/structures/test_utils.py
+++ b/tests/adapters/structures/test_utils.py
@@ -162,3 +162,19 @@ def test_species_from_species_at_sites():
         ],
         key=lambda _: _["name"],
     )
+
+
+def test_elements_ratios_from_sites():
+    import numpy as np
+
+    from optimade.adapters.structures.utils import elements_ratios_from_species_at_sites
+
+    assert np.allclose(elements_ratios_from_species_at_sites(["Si"]), [1.0])
+    assert np.allclose(elements_ratios_from_species_at_sites(["Si", "Ge"]), [0.5, 0.5])
+    assert np.allclose(
+        elements_ratios_from_species_at_sites(["Si", "Si", "Ge"]), [1 / 3, 2 / 3]
+    )
+    assert np.allclose(
+        elements_ratios_from_species_at_sites(["Si", "Si", "Ge", "C", "C"]),
+        [0.4, 0.2, 0.4],
+    )
diff --git a/tests/models/test_structures.py b/tests/models/test_structures.py
index 60f81156..213dfd5e 100644
--- a/tests/models/test_structures.py
+++ b/tests/models/test_structures.py
@@ -158,15 +158,15 @@ deformities = (
     ),
     (
         {"chemical_formula_reduced": "Ge2Si2"},
-        "chemical_formula_reduced 'Ge2Si2' is not properly reduced: greatest common divisor was 2, expected 1.",
+        "chemical_formula_reduced 'Ge2Si2' is not properly reduced: expected 'GeSi'.",
     ),
     (
         {"chemical_formula_reduced": "Ge144Si60V24"},
-        "chemical_formula_reduced 'Ge144Si60V24' is not properly reduced: greatest common divisor was 12, expected 1.",
+        "chemical_formula_reduced 'Ge144Si60V24' is not properly reduced: expected 'Ge12Si5V2'.",
     ),
     (
         {"chemical_formula_anonymous": "A10B5C5"},
-        "chemical_formula_anonymous 'A10B5C5' is not properly reduced: greatest common divisor was 5, expected 1.",
+        "chemical_formula_anonymous 'A10B5C5' is not properly reduced: expected 'A2BC'",
     ),
     (
         {"chemical_formula_anonymous": "A44B15C9D4E3F2GHI0J0K0L0"},
diff --git a/tests/models/test_utils.py b/tests/models/test_utils.py
index 9374b16d..1b4cbaf5 100644
--- a/tests/models/test_utils.py
+++ b/tests/models/test_utils.py
@@ -116,3 +116,25 @@ def test_formula_regexp():
     for formula in bad_formulae:
         with pytest.raises(ValidationError):
             assert DummyModel(formula=formula)
+
+
+def test_reduce_formula():
+    from optimade.models.utils import reduce_formula
+
+    assert reduce_formula("Si1O2") == "O2Si"
+    assert reduce_formula("Si11O2") == "O2Si11"
+    assert reduce_formula("Si10O2C4") == "C2OSi5"
+    assert reduce_formula("Li1") == "Li"
+    assert reduce_formula("Li1Ge1") == "GeLi"
+
+
+def test_anonymize_formula():
+    from optimade.models.utils import anonymize_formula
+
+    assert anonymize_formula("Si1O2") == "A2B"
+    assert anonymize_formula("Si11O2") == "A11B2"
+    assert anonymize_formula("Si10O2C4") == "A5B2C"
+
+    assert anonymize_formula("Si1 O2") == "A2B"
+    assert anonymize_formula("Si11 O2") == "A11B2"
+    assert anonymize_formula("Si10 O2C4") == "A5B2C"