Repository: ESMValGroup/ESMValCore

Issue:
new xch4 derivation script
Some observational datasets are providing the variable xch4 (column-averaged dry air mole fractions of CH4).
This variable is not provided in the CMIP model simulations, so it needs to be derived to be able to compare CMIP simulations with the observations.

I started working on a derivation script in the branch "xch4_derivation_branch".


Discussion:
bouweandela: >  There is 1 codacy issue left, which I could not fix: codacy complains about a missing blank line after line 104 in esmvalcore/preprocessor/_derive/xch4.py even though there is one. The typical DOS/Unix problem LF vs CRLF does not seem to be the reason here. @bouweandela @schlunma do you have an idea?

Running prospector locally gives more useful information:
```
$ prospector esmvalcore/preprocessor/_derive/xch4.py
```
gives:
```
Messages
========

esmvalcore/preprocessor/_derive/xch4.py
  Line: 104
    pep257: D410 / Missing blank line after section ('Parameters')
    pep257: D411 / Missing blank line before section ('Parameters')
    pep257: D417 / Missing argument descriptions in the docstring (argument(s) ch4_cube, ps_cube, top_limit are missing descriptions in '_pressure_level_widths' docstring)
    pep257: D411 / Missing blank line before section ('Returns')
```

But note that numpy style docstrings [are not required](https://docs.esmvaltool.org/projects/esmvalcore/en/latest/contributing.html#what-should-be-documented) for code that is not part of the [public API](https://docs.esmvaltool.org/projects/esmvalcore/en/latest/api/esmvalcore.html) (i.e. can be seen on docs.esmvaltool.org), so it would also be fine to not describe all parameters in detail.

Maybe you were already planning to do this, but I noticed that it looks like a lot of the code has been copy/pasted from the file `esmvalcore/preprocessor/_derive/toz.py`. It would be better to just import the function you need from there, i.e. `from .toz import _pressure_level_widths`.
axel-lauer: All "helper functions" are now imported from toz.py. This also solved the codacy issues. @bouweandela thanks for pointing this out! The tests failing now do not seem to be related to this PR but rather to tests/integration/cmor/_fixes/cmip6/test_cesm2.py and tests/integration/cmor/_fixes/cmip6/test_cesm2_waccm.py. Not sure where this is coming from.
bouweandela: Thats probably #644, usually just re-running the tests by clicking the 'Re-run from failed' button on CircleCI works. I haven't had time to figure out why that is happening and I'm hoping that the problem will just go away by itself again. So far no users have reported encountering this issue while using the tool in real life, until that happens I don't really want to invest a lot of time in it.
bettina-gier: Any news on this? I'd like to do a pull request for xco2 soon, which should build on xch4, as it's almost the same computation.
hb326: > 
> 
> Any news on this? I'd like to do a pull request for xco2 soon, which should build on xch4, as it's almost the same computation.

@schlunma wanted to take a look at this. Not sure how far he got with it during the workshop. I think he changed some things in the calculation of the column and wanted that to be tested to go ahead with this pull request.
schlunma: Yes, the calculation of the pressure level width for `toz` needs to be tested, see [here](https://github.com/ESMValGroup/ESMValCore/pull/880#issuecomment-733671826) :grimacing: 
schlunma: I added unit tests and cleaned the code so similar variables (e.g. `xco2`) can be derived easier. This is ready for review now.

@hb326 can you test it with real CMIP5 **and** CMIP6 data please? I noticed that `hus` has different units in CMIP5 (`1`) and CMIP6 (`%`) and adapted the code to account for this.

Patch:
diff --git a/doc/api/esmvalcore.api.config.rst b/doc/api/esmvalcore.api.config.rst
new file mode 100644
index 000000000..5c694c8d7
--- /dev/null
+++ b/doc/api/esmvalcore.api.config.rst
@@ -0,0 +1,141 @@
+.. _api_config:
+
+Configuration
+=============
+
+This section describes the config submodule of the API.
+
+Config
+******
+
+Configuration of ESMValCore/Tool is done via the ``Config`` object.
+The global configuration can be imported from the ``esmvalcore.experimental`` module as ``CFG``:
+
+.. code-block:: python
+
+    >>> from esmvalcore.experimental import CFG
+    >>> CFG
+    Config({'auxiliary_data_dir': PosixPath('/home/user/auxiliary_data'),
+            'compress_netcdf': False,
+            'config_developer_file': None,
+            'config_file': PosixPath('/home/user/.esmvaltool/config-user.yml'),
+            'drs': {'CMIP5': 'default', 'CMIP6': 'default'},
+            'exit_on_warning': False,
+            'log_level': 'info',
+            'max_parallel_tasks': None,
+            'output_dir': PosixPath('/home/user/esmvaltool_output'),
+            'output_file_type': 'png',
+            'profile_diagnostic': False,
+            'remove_preproc_dir': True,
+            'rootpath': {'CMIP5': '~/default_inputpath',
+                         'CMIP6': '~/default_inputpath',
+                         'default': '~/default_inputpath'},
+            'save_intermediary_cubes': False,
+            'write_netcdf': True,
+            'write_plots': True})
+
+The parameters for the user configuration file are listed `here <https://docs.esmvaltool.org/projects/ESMValCore/en/latest/quickstart/configure.html#user-configuration-file>`__.
+
+``CFG`` is essentially a python dictionary with a few extra functions, similar to ``matplotlib.rcParams``.
+This means that values can be updated like this:
+
+.. code-block:: python
+
+    >>> CFG['output_dir'] = '~/esmvaltool_output'
+    >>> CFG['output_dir']
+    PosixPath('/home/user/esmvaltool_output')
+
+Notice that ``CFG`` automatically converts the path to an instance of ``pathlib.Path`` and expands the home directory.
+All values entered into the config are validated to prevent mistakes, for example, it will warn you if you make a typo in the key:
+
+.. code-block:: python
+
+    >>> CFG['otoptu_dri'] = '~/esmvaltool_output'
+    InvalidConfigParameter: `otoptu_dri` is not a valid config parameter.
+
+Or, if the value entered cannot be converted to the expected type:
+
+.. code-block:: python
+
+    >>> CFG['max_years'] = '🐜'
+    InvalidConfigParameter: Key `max_years`: Could not convert '🐜' to int
+
+``Config`` is also flexible, so it tries to correct the type of your input if possible:
+
+.. code-block:: python
+
+    >>> CFG['max_years'] = '123'  # str
+    >>> type(CFG['max_years'])
+    int
+
+By default, the config is loaded from the default location (``/home/user/.esmvaltool/config-user.yml``).
+If it does not exist, it falls back to the default values.
+to load a different file:
+
+.. code-block:: python
+
+    >>> CFG.load_from_file('~/my-config.yml')
+
+Or to reload the current config:
+
+.. code-block:: python
+
+    >>> CFG.reload()
+
+
+Session
+*******
+
+Recipes and diagnostics will be run in their own directories.
+This behaviour can be controlled via the ``Session`` object.
+A ``Session`` can be initiated from the global ``Config``.
+
+.. code-block:: python
+
+    >>> session = CFG.start_session(name='my_session')
+
+A ``Session`` is very similar to the config.
+It is also a dictionary, and copies all the keys from the ``Config``.
+At this moment, ``session`` is essentially a copy of ``CFG``:
+
+.. code-block:: python
+
+    >>> print(session == CFG)
+    True
+    >>> session['output_dir'] = '~/my_output_dir'
+    >>> print(session == CFG)  # False
+    False
+
+A ``Session`` also knows about the directories where the data will stored.
+The session name is used to prefix the directories.
+
+.. code-block:: python
+
+    >>> session.session_dir
+    /home/user/my_output_dir/my_session_20201203_155821
+    >>> session.run_dir
+    /home/user/my_output_dir/my_session_20201203_155821/run
+    >>> session.work_dir
+    /home/user/my_output_dir/my_session_20201203_155821/work
+    >>> session.preproc_dir
+    /home/user/my_output_dir/my_session_20201203_155821/preproc
+    >>> session.plot_dir
+    /home/user/my_output_dir/my_session_20201203_155821/plots
+
+Unlike the global configuration, of which only one can exist, multiple sessions can be initiated from the ``Config``.
+
+
+API reference
+*************
+
+.. autoclass:: esmvalcore.experimental.config.CFG
+    :no-inherited-members:
+    :no-show-inheritance:
+
+.. autoclass:: esmvalcore.experimental.config.Config
+    :no-inherited-members:
+    :no-show-inheritance:
+
+.. autoclass:: esmvalcore.experimental.config.Session
+    :no-inherited-members:
+    :no-show-inheritance:
diff --git a/doc/api/esmvalcore.api.recipe_info.rst b/doc/api/esmvalcore.api.recipe_info.rst
new file mode 100644
index 000000000..e73ceb25a
--- /dev/null
+++ b/doc/api/esmvalcore.api.recipe_info.rst
@@ -0,0 +1,22 @@
+.. _api_recipe_info:
+
+Recipe Metadata
+===============
+
+This section describes the :py:mod:`esmvalcore.experimental.recipe_info` submodule of the API.
+
+Recipe metadata
+***************
+
+:py:class:`esmvalcore.experimental.recipe_info.RecipeInfo` info is a class that holds metadata from a recipe.
+
+.. code-block:: python
+
+    >>> RecipeInfo('path/to/my_recipe.yml')
+    recipe = RecipeInfo('My recipe')
+
+
+API reference
+*************
+
+.. automodule:: esmvalcore.experimental.recipe_info
diff --git a/doc/api/esmvalcore.api.rst b/doc/api/esmvalcore.api.rst
index 8870ef67e..0f23a0ebd 100644
--- a/doc/api/esmvalcore.api.rst
+++ b/doc/api/esmvalcore.api.rst
@@ -7,137 +7,8 @@ This page describes the new ESMValCore API.
 The API module is available in the submodule ``esmvalcore.experimental``.
 The API is under development, so use at your own risk!
 
-Config
-******
+.. toctree::
 
-Configuration of ESMValCore/Tool is done via the ``Config`` object.
-The global configuration can be imported from the ``esmvalcore.experimental`` module as ``CFG``:
-
-.. code-block:: python
-
-    >>> from esmvalcore.experimental import CFG
-    >>> CFG
-    Config({'auxiliary_data_dir': PosixPath('/home/user/auxiliary_data'),
-            'compress_netcdf': False,
-            'config_developer_file': None,
-            'config_file': PosixPath('/home/user/.esmvaltool/config-user.yml'),
-            'drs': {'CMIP5': 'default', 'CMIP6': 'default'},
-            'exit_on_warning': False,
-            'log_level': 'info',
-            'max_parallel_tasks': None,
-            'output_dir': PosixPath('/home/user/esmvaltool_output'),
-            'output_file_type': 'png',
-            'profile_diagnostic': False,
-            'remove_preproc_dir': True,
-            'rootpath': {'CMIP5': '~/default_inputpath',
-                         'CMIP6': '~/default_inputpath',
-                         'default': '~/default_inputpath'},
-            'save_intermediary_cubes': False,
-            'write_netcdf': True,
-            'write_plots': True})
-
-The parameters for the user configuration file are listed `here <https://docs.esmvaltool.org/projects/ESMValCore/en/latest/quickstart/configure.html#user-configuration-file>`__.
-
-``CFG`` is essentially a python dictionary with a few extra functions, similar to ``matplotlib.rcParams``.
-This means that values can be updated like this:
-
-.. code-block:: python
-
-    >>> CFG['output_dir'] = '~/esmvaltool_output'
-    >>> CFG['output_dir']
-    PosixPath('/home/user/esmvaltool_output')
-
-Notice that ``CFG`` automatically converts the path to an instance of ``pathlib.Path`` and expands the home directory.
-All values entered into the config are validated to prevent mistakes, for example, it will warn you if you make a typo in the key:
-
-.. code-block:: python
-
-    >>> CFG['otoptu_dri'] = '~/esmvaltool_output'
-    InvalidConfigParameter: `otoptu_dri` is not a valid config parameter.
-
-Or, if the value entered cannot be converted to the expected type:
-
-.. code-block:: python
-
-    >>> CFG['max_years'] = '🐜'
-    InvalidConfigParameter: Key `max_years`: Could not convert '🐜' to int
-
-``Config`` is also flexible, so it tries to correct the type of your input if possible:
-
-.. code-block:: python
-
-    >>> CFG['max_years'] = '123'  # str
-    >>> type(CFG['max_years'])
-    int
-
-By default, the config is loaded from the default location (``/home/user/.esmvaltool/config-user.yml``).
-If it does not exist, it falls back to the default values.
-to load a different file:
-
-.. code-block:: python
-
-    >>> CFG.load_from_file('~/my-config.yml')
-
-Or to reload the current config:
-
-.. code-block:: python
-
-    >>> CFG.reload()
-
-
-Session
-*******
-
-Recipes and diagnostics will be run in their own directories.
-This behaviour can be controlled via the ``Session`` object.
-A ``Session`` can be initiated from the global ``Config``.
-
-.. code-block:: python
-
-    >>> session = CFG.start_session(name='my_session')
-
-A ``Session`` is very similar to the config.
-It is also a dictionary, and copies all the keys from the ``Config``.
-At this moment, ``session`` is essentially a copy of ``CFG``:
-
-.. code-block:: python
-
-    >>> print(session == CFG)
-    True
-    >>> session['output_dir'] = '~/my_output_dir'
-    >>> print(session == CFG)  # False
-    False
-
-A ``Session`` also knows about the directories where the data will stored.
-The session name is used to prefix the directories.
-
-.. code-block:: python
-
-    >>> session.session_dir
-    /home/user/my_output_dir/my_session_20201203_155821
-    >>> session.run_dir
-    /home/user/my_output_dir/my_session_20201203_155821/run
-    >>> session.work_dir
-    /home/user/my_output_dir/my_session_20201203_155821/work
-    >>> session.preproc_dir
-    /home/user/my_output_dir/my_session_20201203_155821/preproc
-    >>> session.plot_dir
-    /home/user/my_output_dir/my_session_20201203_155821/plots
-
-Unlike the global configuration, of which only one can exist, multiple sessions can be initiated from the ``Config``.
-
-
-API reference
-*************
-
-.. autoclass:: esmvalcore.experimental.config.CFG
-    :no-inherited-members:
-    :no-show-inheritance:
-
-.. autoclass:: esmvalcore.experimental.config.Config
-    :no-inherited-members:
-    :no-show-inheritance:
-
-.. autoclass:: esmvalcore.experimental.config.Session
-    :no-inherited-members:
-    :no-show-inheritance:
+   esmvalcore.api.config
+   esmvalcore.api.recipe_info
+   esmvalcore.api.utils
diff --git a/doc/api/esmvalcore.api.utils.rst b/doc/api/esmvalcore.api.utils.rst
new file mode 100644
index 000000000..554f14148
--- /dev/null
+++ b/doc/api/esmvalcore.api.utils.rst
@@ -0,0 +1,63 @@
+.. _api_utils:
+
+
+
+Utils
+=====
+
+This section describes the utilities submodule of the API.
+
+Finding recipes
+***************
+
+One of the first thing we may want to do, is to simply get one of the recipes available in ``ESMValTool``
+
+If you already know which recipe you want to load, call :py:func:`esmvalcore.experimental.utils.get_recipe`.
+
+.. code-block:: python
+
+    from esmvalcore.experimental import get_recipe
+    >>> get_recipe('examples/recipe_python')
+    RecipeInfo('Recipe python')
+
+Call the :py:func:`esmvalcore.experimental.utils.get_all_recipes` function to get a list of all available recipes.
+
+.. code-block:: python
+
+    >>> from esmvalcore.experimental import get_all_recipes
+    >>> recipes = get_all_recipes()
+    >>> recipes
+    [RecipeInfo('Recipe perfmetrics cmip5 4cds'),
+     RecipeInfo('Recipe martin18grl'),
+     ...
+     RecipeInfo('Recipe wflow'),
+     RecipeInfo('Recipe pcrglobwb')]
+
+To search for a specific recipe, you can use the :py:meth:`esmvalcore.experimental.utils.RecipeList.find` method. This takes a search query that looks through the recipe metadata and returns any matches. The query can be a regex pattern, so you can make it as complex as you like.
+
+.. code-block:: python
+
+    >>> results = recipes.find('climwip')
+    [RecipeInfo('Recipe climwip')]
+
+The recipes are loaded in a :py:class:`esmvalcore.experimental.recipe_info.RecipeInfo` object, which knows about the documentation, authors, project, and related references of the recipe. It resolves all the tags, so that it knows which institute an author belongs to and which references are associated with the recipe.
+
+This means you can search for something like this:
+
+.. code-block:: python
+
+    >>> recipes.find('Geophysical Research Letters')
+    [RecipeInfo('Recipe martin18grl'),
+     RecipeInfo('Recipe climwip'),
+     RecipeInfo('Recipe ecs constraints'),
+     RecipeInfo('Recipe ecs scatter'),
+     RecipeInfo('Recipe ecs'),
+     RecipeInfo('Recipe seaice')]
+
+
+API reference
+*************
+
+.. automodule:: esmvalcore.experimental.utils
+    :no-inherited-members:
+    :no-show-inheritance:
diff --git a/doc/requirements.txt b/doc/requirements.txt
index 7d67ccad6..0094f9fec 100644
--- a/doc/requirements.txt
+++ b/doc/requirements.txt
@@ -6,6 +6,7 @@ numpy
 pandas
 pillow
 prov[dot]
+pybtex
 pyyaml
 scipy
 shapely[vectorized]
diff --git a/esmvalcore/experimental/__init__.py b/esmvalcore/experimental/__init__.py
index 5614ce777..e7078ae06 100644
--- a/esmvalcore/experimental/__init__.py
+++ b/esmvalcore/experimental/__init__.py
@@ -8,7 +8,13 @@ warnings.warn(
     '\n  More info: https://github.com/ESMValGroup/ESMValCore/issues/498', )
 
 from .config import CFG  # noqa: E402
+from .recipe_info import RecipeInfo  # noqa: E402
+from .utils import RecipeList, get_all_recipes, get_recipe  # noqa: E402
 
 __all__ = [
     'CFG',
+    'get_all_recipes',
+    'get_recipe',
+    'RecipeInfo',
+    'RecipeList',
 ]
diff --git a/esmvalcore/experimental/recipe_info.py b/esmvalcore/experimental/recipe_info.py
new file mode 100644
index 000000000..5fe7ac830
--- /dev/null
+++ b/esmvalcore/experimental/recipe_info.py
@@ -0,0 +1,275 @@
+"""Recipe metadata."""
+
+import textwrap
+from pathlib import Path
+
+import pybtex
+import yaml
+from pybtex.database.input import bibtex
+
+from esmvalcore._citation import REFERENCES_PATH
+from esmvalcore._config import TAGS
+
+
+class RenderError(BaseException):
+    """Error during rendering of object."""
+
+
+class Contributor:
+    """Contains contributor (author or maintainer) information."""
+
+    def __init__(self, name: str, institute: str, orcid: str = None):
+        self.name = name
+        self.institute = institute
+        self.orcid = orcid
+
+    def __repr__(self) -> str:
+        """Return canonical string representation."""
+        return (f'{self.__class__.__name__}({self.name!r},'
+                f' institute={self.institute!r}, orcid={self.orcid!r})')
+
+    def __str__(self) -> str:
+        """Return string representation."""
+        string = f'{self.name} ({self.institute}'
+        if self.orcid:
+            string += f'; {self.orcid}'
+        string += ')'
+        return string
+
+    def _repr_markdown_(self):
+        """Represent using markdown renderer in a notebook environment."""
+        return str(self)
+
+    @classmethod
+    def from_tag(cls, tag: str):
+        """Return an instance of Contributor from a tag (``TAGS``).
+
+        Contributors are defined by author tags in ``config-
+        references.yml``.
+        """
+        mapping = TAGS['authors'][tag]
+
+        name = ' '.join(reversed(mapping['name'].split(', ')))
+        institute = mapping.get('institute', 'No affiliation')
+        orcid = mapping['orcid']
+
+        return cls(name=name, institute=institute, orcid=orcid)
+
+
+class Project:
+    """Contains project information."""
+
+    def __init__(self, project: str):
+        self.project = project
+
+    def __repr__(self) -> str:
+        """Return canonical string representation."""
+        return f'{self.__class__.__name__}({self.project!r})'
+
+    def __str__(self) -> str:
+        """Return string representation."""
+        string = f'{self.project}'
+        return string
+
+    @classmethod
+    def from_tag(cls, tag: str):
+        """Return an instance of Project from a tag (``TAGS``).
+
+        The project tags are defined in ``config-references.yml``.
+        """
+        project = TAGS['projects'][tag]
+        return cls(project=project)
+
+
+class Reference:
+    """Contains reference information."""
+
+    def __init__(self, filename):
+        parser = bibtex.Parser(strict=False)
+        bib_data = parser.parse_file(filename)
+
+        if len(bib_data.entries) > 1:
+            raise NotImplementedError(
+                f'{self.__class__.__name__} cannot handle bibtex files '
+                'with more than 1 entry.')
+
+        self._bib_data = bib_data
+        self._key, self._entry = list(bib_data.entries.items())[0]
+        self._filename = filename
+
+    @classmethod
+    def from_tag(cls, tag: str):
+        """Return an instance of Reference from a bibtex tag.
+
+        The bibtex tags resolved as
+        ``esmvaltool/references/{tag}.bibtex``.
+        """
+        filename = Path(REFERENCES_PATH, f'{tag}.bibtex')
+        return cls(filename)
+
+    def __repr__(self):
+        """Return canonical string representation."""
+        return f'{self.__class__.__name__}({self._key!r})'
+
+    def __str__(self):
+        """Return string representation."""
+        return self.render(renderer='plaintext')
+
+    def _repr_markdown_(self):
+        """Represent using markdown renderer in a notebook environment."""
+        return self.render(renderer='markdown')
+
+    def render(self, renderer: str = 'plaintext') -> str:
+        """Render the reference.
+
+        Parameters
+        ----------
+        renderer : str
+            Choose the renderer for the string representation.
+            Must be one of: 'plaintext', 'markdown', 'html', 'latex'
+
+        Returns
+        -------
+        str
+            Rendered reference
+        """
+        style = 'plain'  # alpha, plain, unsrt, unsrtalpha
+        backend = pybtex.plugin.find_plugin('pybtex.backends', renderer)()
+        style = pybtex.plugin.find_plugin('pybtex.style.formatting', style)()
+
+        try:
+            formatter = style.format_entry(self._key, self._entry)
+            rendered = formatter.text.render(backend)
+        except Exception as err:
+            raise RenderError(
+                f'Could not render {self._key!r}: {err}') from None
+
+        return rendered
+
+
+class RecipeInfo():
+    """Contains Recipe metadata.
+
+    Parameters
+    ----------
+    path : pathlike
+        Path to the recipe.
+    """
+
+    def __init__(self, path: str):
+        self.path = Path(path)
+        if not self.path.exists():
+            raise FileNotFoundError(f'Cannot find recipe: `{path}`.')
+
+        self._data = None
+        self._authors = None
+        self._maintainers = None
+        self._projects = None
+        self._references = None
+        self._description = None
+
+    def __repr__(self) -> str:
+        """Return canonical string representation."""
+        return f'{self.__class__.__name__}({self.name!r})'
+
+    def _repr_markdown_(self) -> str:
+        """Represent using markdown renderer in a notebook environment."""
+        return self.render('markdown')
+
+    def __str__(self) -> str:
+        """Return string representation."""
+        return self.render('plaintext')
+
+    def to_markdown(self) -> str:
+        """Return markdown formatted string."""
+        return self.render('markdown')
+
+    def render(self, renderer: str = 'plaintext') -> str:
+        """Return formatted string.
+
+        Parameters
+        ----------
+        renderer : str
+            Choose the renderer for the string representation.
+            Must be one of 'plaintext', 'markdown'
+        """
+        bullet = '\n - '
+        string = f'## {self.name}'
+
+        string += '\n\n'
+        string += f'{self.description}'
+
+        string += '\n\n### Authors'
+        for author in self.authors:
+            string += f'{bullet}{author}'
+
+        string += '\n\n### Maintainers'
+        for maintainer in self.maintainers:
+            string += f'{bullet}{maintainer}'
+
+        if self.projects:
+            string += '\n\n### Projects'
+            for project in self.projects:
+                string += f'{bullet}{project}'
+
+        if self.references:
+            string += '\n\n### References'
+            for reference in self.references:
+                string += bullet + reference.render(renderer)
+
+        string += '\n'
+
+        return string
+
+    @property
+    def data(self) -> dict:
+        """Return dictionary representation of the recipe."""
+        if self._data is None:
+            self._data = yaml.safe_load(open(self.path, 'r'))
+        return self._data
+
+    @property
+    def name(self) -> str:
+        """Name of the recipe."""
+        return self.path.stem.replace('_', ' ').capitalize()
+
+    @property
+    def description(self) -> str:
+        """Recipe description."""
+        if self._description is None:
+            description = self.data['documentation']['description']
+            self._description = '\n'.join(textwrap.wrap(description))
+        return self._description
+
+    @property
+    def authors(self) -> tuple:
+        """List of recipe authors."""
+        if self._authors is None:
+            tags = self.data['documentation']['authors']
+            self._authors = tuple(Contributor.from_tag(tag) for tag in tags)
+        return self._authors
+
+    @property
+    def maintainers(self) -> tuple:
+        """List of recipe maintainers."""
+        if self._maintainers is None:
+            tags = self.data['documentation']['maintainer']
+            self._maintainers = tuple(
+                Contributor.from_tag(tag) for tag in tags)
+        return self._maintainers
+
+    @property
+    def projects(self) -> tuple:
+        """List of recipe projects."""
+        if self._projects is None:
+            tags = self.data['documentation'].get('projects', [])
+            self._projects = tuple(Project.from_tag(tag) for tag in tags)
+        return self._projects
+
+    @property
+    def references(self) -> tuple:
+        """List of project references."""
+        if self._references is None:
+            tags = self.data['documentation'].get('references', [])
+            self._references = tuple(Reference.from_tag(tag) for tag in tags)
+        return self._references
diff --git a/esmvalcore/experimental/utils.py b/esmvalcore/experimental/utils.py
new file mode 100644
index 000000000..3e2edb864
--- /dev/null
+++ b/esmvalcore/experimental/utils.py
@@ -0,0 +1,85 @@
+"""ESMValCore utilities."""
+
+import re
+from pathlib import Path
+
+from esmvalcore._config import DIAGNOSTICS_PATH
+
+from .recipe_info import RecipeInfo
+
+
+class RecipeList(list):
+    """Container for recipes."""
+
+    def find(self, query: str):
+        """Search for recipes matching the search query or pattern.
+
+        This function will search the recipe description, author list, and
+        project information.
+
+        Parameters
+        ----------
+        query : str
+            String to search for, e.g. ``find_recipes('righi')`` will return
+            all matching that author. Can be a `regex` pattern.
+
+        Returns
+        -------
+        RecipeList
+            List of recipes matching the search query.
+        """
+        query = re.compile(query, flags=re.IGNORECASE)
+
+        matches = RecipeList()
+
+        for recipe in self:
+            match = re.search(query, str(recipe))
+            if match:
+                matches.append(recipe)
+
+        return matches
+
+
+def get_all_recipes(subdir: str = None) -> list:
+    """Return a list of all available recipes.
+
+    Parameters
+    ----------
+    subdir : str
+        Sub-directory of the ``DIAGNOSTICS_PATH`` to look for
+        recipes, e.g. ``get_all_recipes(subdir='examples')``.
+
+    Returns
+    -------
+    RecipeList
+        List of available recipes
+    """
+    if not subdir:
+        subdir = '**'
+    rootdir = Path(DIAGNOSTICS_PATH, 'recipes')
+    files = rootdir.glob(f'{subdir}/*.yml')
+    return RecipeList(RecipeInfo(file) for file in files)
+
+
+def get_recipe(name: str) -> 'RecipeInfo':
+    """Get a recipe by its name.
+
+    Parameters
+    ----------
+    name : str
+        Name of the recipe, i.e. ``examples/recipe_python.yml``
+
+    Returns
+    -------
+    RecipeInfo
+    """
+    locations = Path(), Path(DIAGNOSTICS_PATH, 'recipes')
+    filenames = name, name + '.yml'
+
+    for location in locations:
+        for filename in filenames:
+            try_path = Path(location, filename)
+            if try_path.exists():
+                return RecipeInfo(try_path)
+
+    raise FileNotFoundError(f'Could not find `{name}')
diff --git a/esmvalcore/preprocessor/_derive/_shared.py b/esmvalcore/preprocessor/_derive/_shared.py
index 17556fc9b..38162b6f3 100644
--- a/esmvalcore/preprocessor/_derive/_shared.py
+++ b/esmvalcore/preprocessor/_derive/_shared.py
@@ -5,6 +5,7 @@ from copy import deepcopy
 
 import iris
 import numpy as np
+from scipy import constants
 
 from esmvalcore.iris_helpers import var_name_constraint
 
@@ -29,6 +30,82 @@ def cloud_area_fraction(cubes, tau_constraint, plev_constraint):
     return new_cube
 
 
+def column_average(cube, hus_cube, zg_cube, ps_cube):
+    """Calculate column-averaged mole fractions.
+
+    The calculation follows the method described in the obs4MIPs technical note
+    "Merged SCIMACHY/ENVISAT and TANSO-FTS/GOSAT atmospheric column-average
+    dry-air mole fraction of CH4 (XCH4)".
+    by M. Buchwitz and M. Reuter, available at:
+    http://esgf-data1.ceda.ac.uk/thredds/fileServer/esg_obs4mips/
+        esacci/ghg/data/obs4mips/crdp_3/CH4/v100/
+        TechNote_SCIAGOSAT_L3_CRDP3_001_XCH4_FINAL.pdf
+
+    Parameters
+    ----------
+    cube : iris.cube.Cube
+        Cube containing a mole fraction of a gas (must be 4D with coordinates
+        ``time``, ``air_pressure``, ``latitude`` and ``longitude``).
+        ``latitude`` and ``air_pressure``).
+    hus_cube : iris.cube.Cube
+        Cube containing ``specific_humidity`` (must be 4D with coordinates
+        ``time``, ``air_pressure``, ``latitude`` and ``longitude``).
+        ``latitude`` and ``air_pressure``).
+    zg_cube : iris.cube.Cube
+        Cube containing ``geopotential_height`` (must be 3D with coordinates
+        ``time``, ``latitude`` and ``longitude``).
+    ps_cube : iris.cube.Cube
+        Cube containing ``surface_air_pressure`` (must be 3D with coordinates
+        ``time``, ``latitude`` and ``longitude``).
+
+    Returns
+    -------
+    iris.cube.Cube
+        Column-averaged mole fraction of a gas.
+
+    """
+    # Convert units of data
+    hus_cube.convert_units('1')
+    zg_cube.convert_units('m')
+    ps_cube.convert_units('Pa')
+
+    # Level thickness (note: Buchwitz & Reuter use hPa but we use Pa; in fact,
+    # this does not matter as units cancel out when calculating column-average
+    p_layer_widths = pressure_level_widths(cube, ps_cube, top_limit=0.0)
+
+    # Latitudes (1-dim array)
+    lat = cube.coord('latitude').points
+
+    # Gravitational acceleration g_0 on the geoid approximated by the
+    # international gravity formula depending only on the latitude
+    g_0 = np.array(lat)
+    g_0 = 9.780327 * (1.0 + 0.0053024 * (np.sin(lat / 180.0 * np.pi))**2 -
+                      0.0000058 * (np.sin(2.0 * lat / 180.0 * np.pi))**2)
+
+    # Approximation of the gravitational acceleration including the
+    # free air correction
+    # Note: the formula for g given in Buchwitz & Reuter contains a typo and
+    # should read: g_0**2 - 2*f*zg (i.e. minus instead of +)
+    fair_cor = 3.0825958e-6
+    g_4d_array = iris.util.broadcast_to_shape(g_0**2, zg_cube.shape, [2])
+    g_4d_array = np.sqrt(g_4d_array.data - 2.0 * fair_cor * zg_cube.data)
+
+    # Number of dry air particles (air molecules excluding water vapor) within
+    # each layer
+    mw_air = 28.9644e-3
+    n_dry = ((hus_cube * -1.0 + 1.0) * constants.value('Avogadro constant') *
+             p_layer_widths.data / (mw_air * g_4d_array))
+
+    # Number of gas molecules per layer
+    cube = cube * n_dry
+
+    # Column-average
+    x_cube = (
+        cube.collapsed('air_pressure', iris.analysis.SUM) /
+        n_dry.collapsed('air_pressure', iris.analysis.SUM))
+    return x_cube
+
+
 def pressure_level_widths(cube, ps_cube, top_limit=0.0):
     """Create a cube with pressure level widths.
 
@@ -37,7 +114,7 @@ def pressure_level_widths(cube, ps_cube, top_limit=0.0):
     Parameters
     ----------
     cube : iris.cube.Cube
-        Cube containing ``air_pressure`` coordiante.
+        Cube containing ``air_pressure`` coordinate.
     ps_cube : iris.cube.Cube
         Cube containing ``surface_air_pressure``.
     top_limit : float
diff --git a/esmvalcore/preprocessor/_derive/xch4.py b/esmvalcore/preprocessor/_derive/xch4.py
new file mode 100644
index 000000000..8082c46ec
--- /dev/null
+++ b/esmvalcore/preprocessor/_derive/xch4.py
@@ -0,0 +1,36 @@
+"""Derivation of variable ``xch4``."""
+
+from iris import Constraint
+
+from ._baseclass import DerivedVariableBase
+from ._shared import column_average
+
+
+class DerivedVariable(DerivedVariableBase):
+    """Derivation of variable ``xch4``."""
+
+    @staticmethod
+    def required(project):
+        """Declare the variables needed for derivation."""
+        required = [
+            {'short_name': 'ch4'},
+            {'short_name': 'hus'},
+            {'short_name': 'zg'},
+            {'short_name': 'ps'},
+        ]
+        return required
+
+    @staticmethod
+    def calculate(cubes):
+        """Calculate the column-averaged atmospheric CH4 [1e-9]."""
+        ch4_cube = cubes.extract_strict(
+            Constraint(name='mole_fraction_of_methane_in_air'))
+        hus_cube = cubes.extract_strict(Constraint(name='specific_humidity'))
+        zg_cube = cubes.extract_strict(Constraint(name='geopotential_height'))
+        ps_cube = cubes.extract_strict(Constraint(name='surface_air_pressure'))
+
+        # Column-averaged CH4
+        xch4_cube = column_average(ch4_cube, hus_cube, zg_cube, ps_cube)
+        xch4_cube.convert_units('1')
+
+        return xch4_cube
diff --git a/package/meta.yaml b/package/meta.yaml
index f5ee7471e..d69e26856 100644
--- a/package/meta.yaml
+++ b/package/meta.yaml
@@ -51,6 +51,7 @@ requirements:
     - numpy
     - prov
     - psutil
+    - pybtex
     - pydot
     - pyyaml
     - requests
@@ -82,7 +83,7 @@ test:
     - esmvalcore.cmor.check
     - esmvalcore.cmor.fix
     - esmvalcore.preprocessor
-
+    - esmvalcore.experimental
 
 about:
   home: https://www.esmvaltool.org
diff --git a/setup.py b/setup.py
index 3554240a6..03e4482cc 100755
--- a/setup.py
+++ b/setup.py
@@ -39,6 +39,7 @@ REQUIREMENTS = {
         'numpy',
         'prov[dot]',
         'psutil',
+        'pybtex',
         'pyyaml',
         'requests',
         'scitools-iris>=2.2',


Test patch:
diff --git a/tests/unit/experimental/test_recipe_info.py b/tests/unit/experimental/test_recipe_info.py
new file mode 100644
index 000000000..887b70c5f
--- /dev/null
+++ b/tests/unit/experimental/test_recipe_info.py
@@ -0,0 +1,48 @@
+import pytest
+
+from esmvalcore.experimental import get_recipe
+from esmvalcore.experimental.recipe_info import Contributor, Project, Reference
+
+pytest.importorskip(
+    'esmvaltool',
+    reason='The behaviour of these tests depends on what ``DIAGNOSTICS_PATH``'
+    'points to. This is defined by a forward-reference to ESMValTool, which'
+    'is not installed in the CI, but likely to be available in a developer'
+    'or user installation.')
+
+
+def test_contributor():
+    """Coverage test for Contributor."""
+    contributor = Contributor.from_tag('righi_mattia')
+
+    assert contributor.name == 'Mattia Righi'
+    assert contributor.institute == 'DLR, Germany'
+    assert contributor.orcid.startswith('https://orcid.org/')
+    assert isinstance(repr(contributor), str)
+    assert isinstance(str(contributor), str)
+
+
+def test_reference():
+    """Coverage test for Reference."""
+    reference = Reference.from_tag('acknow_project')
+
+    assert isinstance(repr(reference), str)
+    assert isinstance(str(reference), str)
+    assert isinstance(reference.render('markdown'), str)
+
+
+def test_project():
+    """Coverage test for Project."""
+    project = Project.from_tag('esmval')
+
+    assert isinstance(repr(project), str)
+    assert isinstance(str(project), str)
+
+
+def test_recipe_info():
+    """Coverage test for RecipeInfo."""
+    recipe = get_recipe('examples/recipe_python')
+
+    assert isinstance(repr(recipe), str)
+    assert isinstance(str(recipe), str)
+    assert isinstance(recipe.to_markdown(), str)
diff --git a/tests/unit/experimental/test_utils.py b/tests/unit/experimental/test_utils.py
new file mode 100644
index 000000000..33dcc0d14
--- /dev/null
+++ b/tests/unit/experimental/test_utils.py
@@ -0,0 +1,40 @@
+import pytest
+
+from esmvalcore.experimental.recipe_info import RecipeInfo
+from esmvalcore.experimental.utils import (
+    RecipeList,
+    get_all_recipes,
+    get_recipe,
+)
+
+pytest.importorskip(
+    'esmvaltool',
+    reason='The behaviour of these tests depends on what ``DIAGNOSTICS_PATH``'
+    'points to. This is defined by a forward-reference to ESMValTool, which'
+    'is not installed in the CI, but likely to be available in a developer'
+    'or user installation.')
+
+
+def test_get_recipe():
+    """Get single recipe."""
+    recipe = get_recipe('examples/recipe_python.yml')
+    assert isinstance(recipe, RecipeInfo)
+
+
+def test_get_all_recipes():
+    """Get all recipes."""
+    recipes = get_all_recipes()
+    assert isinstance(recipes, list)
+
+    recipes.find('')
+
+
+def test_recipe_list_find():
+    """Get all recipes."""
+    recipes = get_all_recipes(subdir='examples')
+
+    assert isinstance(recipes, RecipeList)
+
+    result = recipes.find('python')
+
+    assert isinstance(result, RecipeList)
diff --git a/tests/unit/preprocessor/_derive/test_shared.py b/tests/unit/preprocessor/_derive/test_shared.py
index 77e4b84f6..4c3fe686d 100644
--- a/tests/unit/preprocessor/_derive/test_shared.py
+++ b/tests/unit/preprocessor/_derive/test_shared.py
@@ -1,9 +1,60 @@
 """Unit tests for shared variable derivation functions."""
 
+import iris
 import numpy as np
 import pytest
 
-from esmvalcore.preprocessor._derive._shared import _get_pressure_level_widths
+from esmvalcore.preprocessor._derive._shared import (
+    _get_pressure_level_widths,
+    column_average,
+)
+
+
+def get_cube(data, air_pressure_coord=True, **kwargs):
+    """Get sample cube."""
+    time_coord = iris.coords.DimCoord([0.0], standard_name='time',
+                                      var_name='time',
+                                      units='days since 1950-01-01 00:00:00')
+    plev_coord = iris.coords.DimCoord([90000.0, 80000.0],
+                                      standard_name='air_pressure',
+                                      var_name='plev', units='Pa')
+    lat_coord = iris.coords.DimCoord([45.0], standard_name='latitude',
+                                     var_name='lat', units='degrees')
+    lon_coord = iris.coords.DimCoord([10.0], standard_name='longitude',
+                                     var_name='lon', units='degrees')
+    if air_pressure_coord:
+        coord_specs = [(time_coord, 0), (plev_coord, 1), (lat_coord, 2),
+                       (lon_coord, 3)]
+    else:
+        coord_specs = [(time_coord, 0), (lat_coord, 1), (lon_coord, 2)]
+    cube = iris.cube.Cube(data, dim_coords_and_dims=coord_specs, **kwargs)
+    return cube
+
+
+def test_column_average():
+    """Test calculation of column-average."""
+    cube = get_cube([[[[1.0]], [[2.0]]]], air_pressure_coord=True,
+                    var_name='ch4', units='1')
+    hus_cube = get_cube([[[[0.2]], [[0.2]]]], air_pressure_coord=True,
+                        var_name='hus', units='1')
+    zg_cube = get_cube([[[100.0]]], air_pressure_coord=False, var_name='zg',
+                       units='m')
+    ps_cube = get_cube([[[100000.0]]], air_pressure_coord=False, var_name='ps',
+                       units='Pa')
+    x_cube = column_average(cube, hus_cube, zg_cube, ps_cube)
+    assert x_cube.shape == (1, 1, 1)
+    assert x_cube.units == '1'
+    assert x_cube.coords('time')
+    assert x_cube.coords('air_pressure')
+    assert x_cube.coords('latitude')
+    assert x_cube.coords('longitude')
+    np.testing.assert_allclose(x_cube.data, [[[1.85]]])
+    np.testing.assert_allclose(x_cube.coord('time').points, [0.0])
+    np.testing.assert_allclose(x_cube.coord('air_pressure').points, 85000.0)
+    np.testing.assert_allclose(x_cube.coord('air_pressure').bounds,
+                               [[80000.0, 90000.0]])
+    np.testing.assert_allclose(x_cube.coord('latitude').points, [45.0])
+    np.testing.assert_allclose(x_cube.coord('longitude').points, [10.0])
 
 
 def test_col_is_not_monotonic():
diff --git a/tests/unit/preprocessor/_derive/test_xch4.py b/tests/unit/preprocessor/_derive/test_xch4.py
new file mode 100644
index 000000000..e9980fbca
--- /dev/null
+++ b/tests/unit/preprocessor/_derive/test_xch4.py
@@ -0,0 +1,58 @@
+"""Test derivation of ``xch4``."""
+import iris
+import numpy as np
+import pytest
+
+import esmvalcore.preprocessor._derive.xch4 as xch4
+
+from .test_shared import get_cube
+
+
+@pytest.fixture
+def cubes():
+    """Input cubes for derivation of ``xch4``."""
+    xch4_cube = get_cube([[[[1.0]], [[2.0]]]], air_pressure_coord=True,
+                         standard_name='mole_fraction_of_methane_in_air',
+                         var_name='ch4', units='1e-3')
+    hus_cube = get_cube([[[[0.2]], [[0.2]]]], air_pressure_coord=True,
+                        standard_name='specific_humidity', var_name='hus',
+                        units='%')
+    zg_cube = get_cube([[[100.0]]], air_pressure_coord=False,
+                       standard_name='geopotential_height', var_name='zg',
+
+                       units='m')
+    ps_cube = get_cube([[[100000.0]]], air_pressure_coord=False,
+                       standard_name='surface_air_pressure', var_name='ps',
+                       units='Pa')
+    return iris.cube.CubeList([xch4_cube, hus_cube, zg_cube, ps_cube])
+
+
+def test_xch4_calculate(cubes):
+    """Test function ``calculate``."""
+    derived_var = xch4.DerivedVariable()
+    out_cube = derived_var.calculate(cubes)
+    assert out_cube.shape == (1, 1, 1)
+    assert out_cube.units == '1'
+    assert out_cube.coords('time')
+    assert out_cube.coords('air_pressure')
+    assert out_cube.coords('latitude')
+    assert out_cube.coords('longitude')
+    np.testing.assert_allclose(out_cube.data, [[[1.85e-3]]])
+    np.testing.assert_allclose(out_cube.coord('time').points, [0.0])
+    np.testing.assert_allclose(out_cube.coord('air_pressure').points, 85000.0)
+    np.testing.assert_allclose(out_cube.coord('air_pressure').bounds,
+                               [[80000.0, 90000.0]])
+    np.testing.assert_allclose(out_cube.coord('latitude').points, [45.0])
+    np.testing.assert_allclose(out_cube.coord('longitude').points, [10.0])
+
+
+def test_xch4_required():
+    """Test function ``required``."""
+    derived_var = xch4.DerivedVariable()
+    output = derived_var.required(None)
+    assert output == [
+        {'short_name': 'ch4'},
+        {'short_name': 'hus'},
+        {'short_name': 'zg'},
+        {'short_name': 'ps'},
+    ]