Repository: ESMValGroup/ESMValCore

Issue:
Specify directory stucture per path instead of per project in config-user.yml
At the moment all drs need to be the same per project, while it would make much more sense to be able to specify the drs per rootpath.

To make this more clear, now we have this:
```yaml
rootpath:
  # Rootpath to CMIP5 data
  CMIP5:
    - /badc/cmip5/data/cmip5/output1
    - ~/some_other_path

# Directory structure for input data: [default]/BADC/DKRZ/ETHZ/etc
# See config-developer.yml for definitions.
drs:
  CMIP5: BADC
```
while I think this would be much more useful:
```yaml
# Directory structure for input data in the format
# input_data:
#   project_a:
#     /path/to/dir_1: drs_1
#     /path/to/dir_2: drs_2
#   project_b:
#     /path/to/dir_3: drs_2
# See config-developer.yml for valid drs definitions per project.
input_data:
  CMIP5:
    /badc/cmip5/data/cmip5/output1: BADC
    ~/some_other_path: default
```

Discussion:
codecov[bot]: # [Codecov](https://codecov.io/gh/ESMValGroup/ESMValCore/pull/1894?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ESMValGroup) Report
> Merging [#1894](https://codecov.io/gh/ESMValGroup/ESMValCore/pull/1894?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ESMValGroup) (86acb34) into [main](https://codecov.io/gh/ESMValGroup/ESMValCore/commit/de1a27d40e35a215d614fe73a6c3c7f7ac81ae73?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ESMValGroup) (de1a27d) will **increase** coverage by `0.00%`.
> The diff coverage is `97.01%`.

```diff
@@           Coverage Diff           @@
##             main    #1894   +/-   ##
=======================================
  Coverage   92.02%   92.03%           
=======================================
  Files         234      234           
  Lines       12032    12054   +22     
=======================================
+ Hits        11073    11094   +21     
- Misses        959      960    +1     
```


| [Impacted Files](https://codecov.io/gh/ESMValGroup/ESMValCore/pull/1894?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ESMValGroup) | Coverage Δ | |
|---|---|---|
| [esmvalcore/config/\_config\_validators.py](https://codecov.io/gh/ESMValGroup/ESMValCore/pull/1894?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ESMValGroup#diff-ZXNtdmFsY29yZS9jb25maWcvX2NvbmZpZ192YWxpZGF0b3JzLnB5) | `92.30% <83.33%> (-0.45%)` | :arrow_down: |
| [esmvalcore/local.py](https://codecov.io/gh/ESMValGroup/ESMValCore/pull/1894?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ESMValGroup#diff-ZXNtdmFsY29yZS9sb2NhbC5weQ==) | `97.94% <98.36%> (+0.10%)` | :arrow_up: |

Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ESMValGroup). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ESMValGroup)

bouweandela: The alternative would be to completely redesign how we configure finding data. There are some good ideas out there already, see e.g. https://github.com/ESMValGroup/ESMValCore/issues/795#issuecomment-698219457, the issues linked therein, and https://github.com/ESMValGroup/ESMValCore/issues/1165. However, none of these options would be backward compatible.
valeriupredoi: > The alternative would be to completely redesign how we configure finding data. There are some good ideas out there already, see e.g. [#795 (comment)](https://github.com/ESMValGroup/ESMValCore/issues/795#issuecomment-698219457), the issues linked therein, and #1165. However, none of these options would be backward compatible. That's why I opted for keeping the existing functionality and extending it with yet another option.

yes! Redesign is the key here - at the end of the day, the user will still get their data, but going through much less a pain how to configure their environment. If we go on like this, even us, the devs, will start getting lost in configuration - I, for one, am starting to already, and only have about 6 years experience with the Tool :grin: 
remi-kazeroni: Hi @bouweandela, sorry for being late in reviewing this. I will take a look once the comment's from @valeriupredoi are addressed, would that be ok? Please let us know what is needed to bring this PR to the finish line 🍺 
bouweandela: > Redesign is the key here

How about adding a new item to config-user.yml that can be used to directly construct [`esmvalcore.local.DataSource`](https://esmvaltool--1894.org.readthedocs.build/projects/ESMValCore/en/1894/api/esmvalcore.local.html#esmvalcore.local.DataSource) objects from? We could then easily make that extensible by allowing the user to provide custom DataSource classes, so they could instead use e.g. intake-esm #1218 or whatever they want.
 
```yml
data_sources:
  - type: esmvalcore.local.DataSource
    project: CMIP6
    rootpath: /work/bd0854/DATA/ESMValTool2/CMIP6_DKRZ
    dirname_template: '{activity}/{institute}/{dataset}/{exp}/{ensemble}/{mip}/{short_name}/{grid}/{version}'
    filename_template: '{short_name}_{mip}_{dataset}_{exp}_{ensemble}_{grid}*.nc'
```
the `type` key would only be needed if it's something else than `esmvalcore.local.DataSource`. 

This has some advantages:
- It does not hide where the path templates are defined, making it easier to understand
- No nested dicts with many possible entries, just a single list of DataSource objects
- Extensible if we provide an abstract base class for DataSource. It should have a `find_files` method and the kind of things that it can return, current examples: `esmvalcore.local.LocalFile` and `esmvalcore.esgf.ESGFFile`

There's also a few disadvantages:
- It may look a bit more intimidating at first glance
- For user-friendliness, it we would still to finally do something with the proposal of defining sites (e.g. Jasmin, Levante, etc) and having some simple way of getting a default configuration for those
- No solution for where the CMOR tables or output filenames are defined, those would still be in config-developer.yml
- I'm not sure if I have time to implement this, having this in v2.8 seems unlikely
schlunma: I personally like this new feature very much (the way it is implemented right now)! Specifying DRS per rootpath will make my daily work much easier (I don't know how much time I wasted in the past with creating nested directories when I wanted to use data from two different sources). So I am definitely in favor of having this in v2.8.

About the key-value order: since rootpaths are unique and DRS are not (each rootpath has a single DRS, but each DRS can be used for multiple rootpaths) the order that @bouweandela proposed (e.g., `/path/to/cmip6: DKRZ`) makes much more sense to me.

About the customized data sources: I also like this idea very much! Being able to define custom file and dir templates quickly in the config-user file without messing with a custom config-developer file is great! However, I agree that this might be more intimidating  for new users. Maybe we could add some defaults in the config-developer file and the user has the possibility to add custom entries in the config-user file? This could mean that the config-developer and config-user files contain identical options with the config-developer as machine/default configuration, and the config-user as the user-defined configuration. The current distinction of these two files feels rather arbitrary to me. I think this has been proposed in the past, but I cannot find the corresponding issue. Maybe something for a v3.0?

TLDR; I am in favor of merging this as is, and think about allowing custom data sources in a future version.
valeriupredoi: So here's the thing - just like Eisenhower once said, I like all this, as long as it's not in the config-user file 😁 Are you guys not realizing that we're slowly beginning to bena tool with such a complicated configuration that one needs less time to learn to fly a Boeing 737 than to learn to correctly configure ESMValTool? I'll bark more on the call today :beer:
schlunma: To be honest, I think the current version of specifying DRS+rootpath in two different places (`drs` and `rootpath`) is more complicated than what Bouwe proposes here, which is "for rootpath XYZ, use DRS XZY".


valeriupredoi: > To be honest, I think the current version of specifying DRS+rootpath in two different places (`drs` and `rootpath`) is more complicated than what Bouwe proposes here, which is "for rootpath XYZ, use DRS XZY".

this is not decommissioning the other config option, it adds to it it as an extra option. At any rate, let's talk on the call, we can add here and in #1609 an executive summary of our discussion on the call, later on
bouweandela: The @ESMValGroup/technical-lead-development-team decided to postpone this feature to the next release.
remi-kazeroni: Thanks @bouweandela for all your work on this already and agreeing to shift this to the next release. Let's make sure that we have a release planning meeting soon after v2.8 is released to discuss this PR. Or even a specific meeting to discuss how we could revamp the configuration of ESMValTool in order to increase the user-friendliness of the Tool
schlunma: I'd really like to revive this discussion here. The changes proposed in this PR, i.e., being able to do something like this

```yaml
download_dir: /work/bd0854/DATA/ESMValTool2/download
rootpath:
  CMIP6:
    /work/bd0854/DATA/ESMValTool2/CMIP6_DKRZ: DKRZ
    /work/bd0854/DATA/ESMValTool2/download: ESGF
```

would be really helpful, especially when dealing with native model output. I really hope we can include this in v2.9.
bouweandela: At the workshop it was agreed that a new configuration file format is acceptable and I'll write a proposal. Dropping this from the v2.10 milestone because it seems unlikely that we can agree on the new format before that time.
schlunma: > At the workshop it was agreed that a new configuration file format is acceptable and I'll write a proposal. Dropping this from the v2.10 milestone because it seems unlikely that we can agree on the new format before that time.

It's been 8 months since the workshop, and there was no progress regarding the new configuration. Is there any way we can merge this feature as it is proposed in this PR? I know that it would be much cleaner to refactor our configuration, but this just doesn't seem realistic at this point.

On the other hand, this issue has been open since 2018 (!), and many people would benefit a lot from this.
bouweandela: Sorry for being slow with the writeup for the new configuration, I'll try to get it done next week. About merging this as-is, maybe that would be something to discuss at the next @ESMValGroup/technical-lead-development-team meeting?
schlunma: Good idea, I just added it to the agenda.
valeriupredoi: don't blame me for holding this up for too long and now we have a bazillion conflicts :grin: 
schlunma: @bouweandela I would really like to have this in v2.12. Would it be fine for you if I solve the merge conflicts and finalize this PR?
bouweandela: Yes, that would be great!
schlunma: Thanks for testing @axel-lauer! 🚀 

The reason for this behavior is the [default entry for `drs`](https://github.com/ESMValGroup/ESMValCore/blob/405300886cfb2314eadc6ff2f9b137371ada0dfe/esmvalcore/config-user.yml#L105-L110), which says `ESGF` for the obs4mips data. If you don't specify `drs` in your own configuration file, this default is used. If you specify `drs` in your configuration file but omit obs4mips, then `default` instead of `ESGF` is used.

I agree that this behavior is confusing, but this was not introduced with this PR. With the current main branch you get the same result.

Patch:
diff --git a/doc/quickstart/configure.rst b/doc/quickstart/configure.rst
index 834109e2d..cb5c73340 100644
--- a/doc/quickstart/configure.rst
+++ b/doc/quickstart/configure.rst
@@ -26,7 +26,7 @@ User configuration file
 
 
 The ``config-user.yml`` configuration file contains all the global level
-information needed by ESMValTool. It can be reused as many times the user needs
+information needed by ESMValCore. It can be reused as many times the user needs
 to before changing any of the options stored in it. This file is essentially
 the gateway between the user and the machine-specific instructions to
 ``esmvaltool``. By default, esmvaltool looks for it in the home directory,
@@ -189,7 +189,7 @@ and memory usage.
 A detailed explanation of the data finding-related sections of the
 ``config-user.yml`` (``rootpath`` and ``drs``) is presented in the
 :ref:`data-retrieval` section. This section relates directly to the data
-finding capabilities  of ESMValTool and are very important to be understood by
+finding capabilities of ESMValCore and are very important to be understood by
 the user.
 
 .. note::
@@ -945,7 +945,7 @@ addition of more details per project, dataset, mip table, and variable name.
 
 More precisely, one can provide this information in an extra yaml file, named
 `{project}-something.yml`, where `{project}` corresponds to the project as used
-by ESMValTool in :ref:`Datasets` and "something" is arbitrary.
+by ESMValCore in :ref:`Datasets` and "something" is arbitrary.
 
 Format of the extra facets files
 --------------------------------
@@ -998,7 +998,7 @@ variable of any CMIP5 dataset that does not have a ``product`` key yet:
 Location of the extra facets files
 ----------------------------------
 Extra facets files can be placed in several different places. When we use them
-to support a particular use-case within the ESMValTool project, they will be
+to support a particular use-case within the ESMValCore project, they will be
 provided in the sub-folder `extra_facets` inside the package
 :mod:`esmvalcore.config`. If they are used from the user side, they can be either
 placed in `~/.esmvaltool/extra_facets` or in any other directory of the users
diff --git a/doc/quickstart/find_data.rst b/doc/quickstart/find_data.rst
index 131bf8f99..5692a9f0b 100644
--- a/doc/quickstart/find_data.rst
+++ b/doc/quickstart/find_data.rst
@@ -7,7 +7,7 @@ Input data
 Overview
 ========
 Data discovery and retrieval is the first step in any evaluation process;
-ESMValTool uses a `semi-automated` data finding mechanism with inputs from both
+ESMValCore uses a `semi-automated` data finding mechanism with inputs from both
 the user configuration file and the recipe file: this means that the user will
 have to provide the tool with a set of parameters related to the data needed
 and once these parameters have been provided, the tool will automatically find
@@ -31,7 +31,7 @@ standard for naming files and structured paths; the `DRS
 <https://www.ecmwf.int/sites/default/files/elibrary/2014/13713-data-reference-syntax-governing-standards-within-climate-research-data-archived-esgf.pdf>`_
 ensures that files and paths to them are named according to a
 standardized convention. Examples of this convention, also used by
-ESMValTool for file discovery and data retrieval, include:
+ESMValCore for file discovery and data retrieval, include:
 
 * CMIP6 file: ``{variable_short_name}_{mip}_{dataset_name}_{experiment}_{ensemble}_{grid}_{start-date}-{end-date}.nc``
 * CMIP5 file: ``{variable_short_name}_{mip}_{dataset_name}_{experiment}_{ensemble}_{start-date}-{end-date}.nc``
@@ -44,7 +44,7 @@ ESGF data nodes, these paths differ slightly, for example:
   {variable_short_name}/{grid}``;
 * CMIP6 path for ETHZ: ``ROOT-ETHZ/{experiment}/{mip}/{variable_short_name}/{dataset_name}/{ensemble}/{grid}``
 
-From the ESMValTool user perspective the number of data input parameters is
+From the ESMValCore user perspective the number of data input parameters is
 optimized to allow for ease of use. We detail this procedure in the next
 section.
 
@@ -163,7 +163,7 @@ dedicated projects instead of the project ``native6``.
 CESM
 ^^^^
 
-ESMValTool is able to read native `CESM <https://www.cesm.ucar.edu/>`__ model
+ESMValCore is able to read native `CESM <https://www.cesm.ucar.edu/>`__ model
 output.
 
 .. warning::
@@ -252,7 +252,7 @@ Key                  Description                            Default value if not
 EMAC
 ^^^^
 
-ESMValTool is able to read native `EMAC
+ESMValCore is able to read native `EMAC
 <https://www.dlr.de/pa/en/desktopdefault.aspx/tabid-8859/15306_read-37415/>`_
 model output.
 
@@ -274,7 +274,7 @@ Thus, example dataset entries could look like this:
     - {project: EMAC, dataset: EMAC, exp: historical, mip: Amon, short_name: ta, raw_name: tm1_p39_cav, start_year: 2000, end_year: 2014}
 
 Please note the duplication of the name ``EMAC`` in ``project`` and
-``dataset``, which is necessary to comply with ESMValTool's data finding and
+``dataset``, which is necessary to comply with ESMValCore's data finding and
 CMORizing functionalities.
 A variable-specific default for the facet ``channel`` is given in the extra
 facets (see next paragraph) for many variables, but this can be overwritten in
@@ -285,7 +285,7 @@ facets<extra_facets>`.
 By default, the file :download:`emac-mappings.yml
 </../esmvalcore/config/extra_facets/emac-mappings.yml>` is used for that
 purpose.
-For some variables, extra facets are necessary; otherwise ESMValTool cannot
+For some variables, extra facets are necessary; otherwise ESMValCore cannot
 read them properly.
 Supported keys for extra facets are:
 
@@ -326,7 +326,7 @@ Key                  Description                            Default value if not
 ICON
 ^^^^
 
-ESMValTool is able to read native `ICON
+ESMValCore is able to read native `ICON
 <https://code.mpimet.mpg.de/projects/iconpublic>`_ model output.
 
 The default naming conventions for input directories and files for ICON are
@@ -349,7 +349,7 @@ Thus, example dataset entries could look like this:
        end_year: 2014}
 
 Please note the duplication of the name ``ICON`` in ``project`` and
-``dataset``, which is necessary to comply with ESMValTool's data finding and
+``dataset``, which is necessary to comply with ESMValCore's data finding and
 CMORizing functionalities.
 A variable-specific default for the facet ``var_type`` is given in the extra
 facets (see below) for many variables, but this can be overwritten in the
@@ -460,7 +460,7 @@ facets<extra_facets>`.
 By default, the file :download:`icon-mappings.yml
 </../esmvalcore/config/extra_facets/icon-mappings.yml>` is used for that
 purpose.
-For some variables, extra facets are necessary; otherwise ESMValTool cannot
+For some variables, extra facets are necessary; otherwise ESMValCore cannot
 read them properly.
 Supported keys for extra facets are:
 
@@ -569,7 +569,7 @@ files must also undergo some data selection.
 
 Data retrieval
 ==============
-Data retrieval in ESMValTool has two main aspects from the user's point of
+Data retrieval in ESMValCore has two main aspects from the user's point of
 view:
 
 * data can be found by the tool, subject to availability on disk or `ESGF <https://esgf.llnl.gov/>`_;
@@ -577,7 +577,7 @@ view:
 
 The first point is self-explanatory: if the user runs the tool on a machine
 that has access to a data repository or multiple data repositories, then
-ESMValTool will look for and find the available data requested by the user.
+ESMValCore will look for and find the available data requested by the user.
 If the files are not found locally, the tool can search the ESGF_ and download
 the missing files, provided that they are available.
 
@@ -598,7 +598,7 @@ the :ref:`user configuration file`.
 
 Setting the correct root paths
 ------------------------------
-The first step towards providing ESMValTool the correct set of parameters for
+The first step towards providing ESMValCore the correct set of parameters for
 data retrieval is setting the root paths to the data. This is done in the user
 configuration file ``config-user.yml``. The two sections where the user will
 set the paths are ``rootpath`` and ``drs``. ``rootpath`` contains pointers to
@@ -608,24 +608,11 @@ first discuss the ``drs`` parameter: as we've seen in the previous section, the
 DRS as a standard is used for both file naming conventions and for directory
 structures.
 
-Synda
------
-
-If the `synda install <https://prodiguer.github.io/synda/sdt/user_guide.html#synda-install>`_ command is used to download data,
-it maintains the directory structure as on ESGF. To find data downloaded by
-synda, use the ``SYNDA`` ``drs`` parameter.
-
-.. code-block:: yaml
-
- drs:
-   CMIP6: SYNDA
-   CMIP5: SYNDA
-
 .. _config-user-drs:
 
 Explaining ``config-user/drs: CMIP5:`` or ``config-user/drs: CMIP6:``
 ---------------------------------------------------------------------
-Whereas ESMValTool will **always** use the CMOR standard for file naming (please
+Whereas ESMValCore will by default use the CMOR standard for file naming (please
 refer above), by setting the ``drs`` parameter the user tells the tool what
 type of root paths they need the data from, e.g.:
 
@@ -655,10 +642,17 @@ is another way to retrieve data from a ``ROOT`` directory that has no DRS-like
 structure; ``default`` indicates that the data lies in a directory that
 contains all the files without any structure.
 
+The names of the directories trees that can be used under `drs` are defined in
+:ref:`config-developer`.
+
 .. note::
-   When using ``CMIP6: default`` or ``CMIP5: default`` it is important to
-   remember that all the needed files must be in the same top-level directory
-   set by ``default`` (see below how to set ``default``).
+   When using ``CMIP6: default`` or ``CMIP5: default``, all the needed files
+   must be in the same top-level directory specified under ``rootpath``.
+   However, it is not recommended to use this, as it makes it impossible for
+   the tool to read the facets from the directory tree.
+   Moreover, this way of organizing data makes it impossible to store multiple
+   versions of the same file because the files typically have the same name
+   for different versions.
 
 .. _config-user-rootpath:
 
@@ -668,27 +662,37 @@ Explaining ``config-user/rootpath:``
 ``rootpath`` identifies the root directory for different data types (``ROOT`` as we used it above):
 
 * ``CMIP`` e.g. ``CMIP5`` or ``CMIP6``: this is the `root` path(s) to where the
-  CMIP files are stored; it can be a single path or a list of paths; it can
+  CMIP files are stored; it can be a single path, a list of paths, or a mapping
+  with paths as keys and `drs` names as values; it can
   point to an ESGF node or it can point to a user private repository. Example
-  for a CMIP5 root path pointing to the ESGF node on CEDA-Jasmin (formerly
+  for a CMIP5 root path pointing to the ESGF node mounted on CEDA-Jasmin (formerly
   known as BADC):
 
   .. code-block:: yaml
 
-    CMIP5: /badc/cmip5/data/cmip5/output1
+    rootpath:
+      CMIP5: /badc/cmip5/data/cmip5/output1
 
   Example for a CMIP6 root path pointing to the ESGF node on CEDA-Jasmin:
 
   .. code-block:: yaml
 
-    CMIP6: /badc/cmip6/data/CMIP6/CMIP
+    rootpath:
+      CMIP6: /badc/cmip6/data/CMIP6
 
   Example for a mix of CMIP6 root path pointing to the ESGF node on CEDA-Jasmin
   and a user-specific data repository for extra data:
 
   .. code-block:: yaml
 
-    CMIP6: [/badc/cmip6/data/CMIP6/CMIP, /home/users/johndoe/cmip_data]
+    rootpath:
+      CMIP6:
+        /badc/cmip6/data/CMIP6: BADC
+        ~/climate_data: ESGF
+
+  Note that this notation combines the ``rootpath`` and ``drs`` settings, so it
+  is not necessary to specify the directory structure in under ``drs`` in this
+  case.
 
 * ``OBS``: this is the `root` path(s) to where the observational datasets are
   stored; again, this could be a single path or a list of paths, just like for
@@ -697,7 +701,8 @@ Explaining ``config-user/rootpath:``
 
   .. code-block:: yaml
 
-    OBS: /gws/nopw/j04/esmeval/obsdata-v2
+    rootpath:
+      OBS: /gws/nopw/j04/esmeval/obsdata-v2
 
 * ``default``: this is the `root` path(s) where the tool will look for data
   from projects that do not have their own rootpath set.
@@ -705,9 +710,22 @@ Explaining ``config-user/rootpath:``
 * ``RAWOBS``: this is the `root` path(s) to where the raw observational data
   files are stored; this is used by ``esmvaltool data format``.
 
+Synda
+-----
+
+If the `synda install <https://prodiguer.github.io/synda/sdt/user_guide.html#synda-install>`_ command is used to download data,
+it maintains the directory structure as on ESGF. To find data downloaded by
+synda, use the ``SYNDA`` ``drs`` parameter.
+
+.. code-block:: yaml
+
+ drs:
+   CMIP6: SYNDA
+   CMIP5: SYNDA
+
 Dataset definitions in ``recipe``
 ---------------------------------
-Once the correct paths have been established, ESMValTool collects the
+Once the correct paths have been established, ESMValCore collects the
 information on the specific datasets that are needed for the analysis. This
 information, together with the CMOR convention for naming files (see CMOR-DRS_)
 will allow the tool to search and find the right files. The specific
diff --git a/esmvalcore/cmor/_fixes/icon/_base_fixes.py b/esmvalcore/cmor/_fixes/icon/_base_fixes.py
index e4196b035..9c242ef73 100644
--- a/esmvalcore/cmor/_fixes/icon/_base_fixes.py
+++ b/esmvalcore/cmor/_fixes/icon/_base_fixes.py
@@ -19,7 +19,7 @@ from iris.cube import Cube, CubeList
 from iris.experimental.ugrid import Connectivity, Mesh
 
 from esmvalcore.cmor._fixes.native_datasets import NativeDatasetFix
-from esmvalcore.local import _get_rootpath, _replace_tags, _select_drs
+from esmvalcore.local import _get_data_sources
 
 logger = logging.getLogger(__name__)
 
@@ -297,12 +297,12 @@ class IconFix(NativeDatasetFix):
 
     def _get_grid_from_rootpath(self, grid_name: str) -> CubeList | None:
         """Try to get grid from the ICON rootpath."""
-        rootpaths = _get_rootpath('ICON')
-        dirname_template = _select_drs('input_dir', 'ICON')
-        dirname_globs = _replace_tags(dirname_template, self.extra_facets)
-        possible_grid_paths = [
-            r / d / grid_name for r in rootpaths for d in dirname_globs
-        ]
+        glob_patterns: list[Path] = []
+        for data_source in _get_data_sources('ICON'):
+            glob_patterns.extend(
+                data_source.get_glob_patterns(**self.extra_facets)
+            )
+        possible_grid_paths = [d.parent / grid_name for d in glob_patterns]
         for grid_path in possible_grid_paths:
             if grid_path.is_file():
                 logger.debug("Using ICON grid file '%s'", grid_path)
diff --git a/esmvalcore/config-user.yml b/esmvalcore/config-user.yml
index c102928db..ecdee818f 100644
--- a/esmvalcore/config-user.yml
+++ b/esmvalcore/config-user.yml
@@ -109,13 +109,25 @@ drs:
   CORDEX: ESGF
   obs4MIPs: ESGF
 
-# Example rootpaths and directory structure that showcases the different
-# projects and also the use of lists
+# Example rootpaths and directory structure names for different projects.
+# For each project, the entry can be a single path, a list of paths, or a
+# mapping from paths to directory structure names.
+# For single paths and list of paths, the directory structure names can be
+# defined under 'drs'.
+# If no path is defined for a project, the tool will look in the 'default'
+# path.
+# If no directory structure name is given, the name 'default' will be used.
+# Directory structures corresponding to the names are defined in the file
+# config-developer.yml.
 # For site-specific entries, see below.
 #rootpath:
-#  CMIP3: [~/cmip3_inputpath1, ~/cmip3_inputpath2]
-#  CMIP5: [~/cmip5_inputpath1, ~/cmip5_inputpath2]
-#  CMIP6: [~/cmip6_inputpath1, ~/cmip6_inputpath2]
+#  CMIP6:
+#    /path/to/data: DKRZ
+#    ~/path/to/more/data: ESGF
+#  CMIP5:
+#    - ~/cmip5_inputpath1
+#    - ~/cmip5_inputpath2
+#  CMIP3: ~/cmip6_inputpath
 #  OBS: ~/obs_inputpath
 #  OBS6: ~/obs6_inputpath
 #  obs4MIPs: ~/obs4mips_inputpath
@@ -124,11 +136,10 @@ drs:
 #  RAWOBS: ~/rawobs_inputpath
 #  default: ~/default_inputpath
 #drs:
-#  CMIP3: default
-#  CMIP5: default
-#  CMIP6: default
-#  CORDEX: default
-#  obs4MIPs: default
+#  CMIP3: ESGF
+#  CMIP5: ESGF
+#  CORDEX: ESGF
+#  obs4MIPs: ESGF
 
 # Directory tree created by automatically downloading from ESGF
 # Uncomment the lines below to locate data that has been automatically
@@ -175,22 +186,27 @@ drs:
 # Uncomment the lines below to locate data on Levante at DKRZ.
 #auxiliary_data_dir: /work/bd0854/DATA/ESMValTool2/AUX
 #rootpath:
-#  CMIP6: /work/bd0854/DATA/ESMValTool2/CMIP6_DKRZ
-#  CMIP5: /work/bd0854/DATA/ESMValTool2/CMIP5_DKRZ
-#  CMIP3: /work/bd0854/DATA/ESMValTool2/CMIP3
-#  CORDEX: /work/ik1017/C3SCORDEX/data/c3s-cordex/output
+#  CMIP6:
+#    /work/bd0854/DATA/ESMValTool2/CMIP6_DKRZ: DKRZ
+#    /work/bd0854/DATA/ESMValTool2/download: ESGF
+#  CMIP5:
+#    /work/bd0854/DATA/ESMValTool2/CMIP5_DKRZ: DKRZ
+#    /work/bd0854/DATA/ESMValTool2/download: ESGF
+#  CMIP3:
+#    /work/bd0854/DATA/ESMValTool2/CMIP3: DKRZ
+#    /work/bd0854/DATA/ESMValTool2/download: ESGF
+#  CORDEX:
+#    /work/ik1017/C3SCORDEX/data/c3s-cordex/output: BADC
+#    /work/bd0854/DATA/ESMValTool2/download: ESGF
 #  OBS: /work/bd0854/DATA/ESMValTool2/OBS
 #  OBS6: /work/bd0854/DATA/ESMValTool2/OBS
-#  obs4MIPs: /work/bd0854/DATA/ESMValTool2/OBS
+#  obs4MIPs:
+#    /work/bd0854/DATA/ESMValTool2/OBS: default
+#    /work/bd0854/DATA/ESMValTool2/download: ESGF
 #  ana4mips: /work/bd0854/DATA/ESMValTool2/OBS
 #  native6: /work/bd0854/DATA/ESMValTool2/RAWOBS
 #  RAWOBS: /work/bd0854/DATA/ESMValTool2/RAWOBS
 #drs:
-#  CMIP6: DKRZ
-#  CMIP5: DKRZ
-#  CMIP3: DKRZ
-#  CORDEX: BADC
-#  obs4MIPs: default
 #  ana4mips: default
 #  OBS: default
 #  OBS6: default
diff --git a/esmvalcore/config/_config_validators.py b/esmvalcore/config/_config_validators.py
index 867370ae7..d6489862d 100644
--- a/esmvalcore/config/_config_validators.py
+++ b/esmvalcore/config/_config_validators.py
@@ -205,7 +205,17 @@ def validate_rootpath(value):
                 "Correcting capitalization, project 'obs4mips' should be "
                 "written as 'obs4MIPs' in 'rootpath' in config-user.yml")
             key = 'obs4MIPs'
-        new_mapping[key] = validate_pathlist(paths)
+        if isinstance(paths, Path):
+            paths = str(paths)
+        if isinstance(paths, (str, list)):
+            new_mapping[key] = validate_pathlist(paths)
+        else:
+            validate_dict(paths)
+            new_mapping[key] = {
+                validate_path(path): validate_string(drs)
+                for path, drs in paths.items()
+            }
+
     return new_mapping
 
 
diff --git a/esmvalcore/local.py b/esmvalcore/local.py
index 5cb499dbb..539679f68 100644
--- a/esmvalcore/local.py
+++ b/esmvalcore/local.py
@@ -5,6 +5,7 @@ import itertools
 import logging
 import os
 import re
+from dataclasses import dataclass
 from glob import glob
 from pathlib import Path
 from typing import TYPE_CHECKING, Any, Union
@@ -376,78 +377,88 @@ def _apply_caps(original, lower, upper):
     return original
 
 
-def _select_drs(input_type, project):
+def _select_drs(input_type: str, project: str, structure: str) -> list[str]:
     """Select the directory structure of input path."""
     cfg = get_project_config(project)
-    input_path = cfg[input_type]
-    if isinstance(input_path, str):
-        return input_path
+    input_path_patterns = cfg[input_type]
+    if isinstance(input_path_patterns, str):
+        return [input_path_patterns]
 
-    structure = CFG['drs'].get(project, 'default')
-    if structure in input_path:
-        return input_path[structure]
+    if structure in input_path_patterns:
+        value = input_path_patterns[structure]
+        if isinstance(value, str):
+            value = [value]
+        return value
 
     raise KeyError(
         'drs {} for {} project not specified in config-developer file'.format(
             structure, project))
 
 
-_ROOTPATH_WARNED = set()
+@dataclass(order=True, frozen=True)
+class DataSource:
+    """Class for storing a data source and finding the associated files."""
 
+    rootpath: Path
+    dirname_template: str
+    filename_template: str
 
-def _get_rootpath(project):
-    """Select the rootpath."""
-    rootpath = CFG['rootpath']
-    for key in (project, 'default'):
-        if key in rootpath:
-            nonexistent = tuple(p for p in rootpath[key]
-                                if not os.path.exists(p))
-            if nonexistent and (key, nonexistent) not in _ROOTPATH_WARNED:
-                logger.warning(
-                    "'%s' rootpaths '%s' set in config-user.yml do not exist",
-                    key, ', '.join(str(p) for p in nonexistent))
-                _ROOTPATH_WARNED.add((key, nonexistent))
-            return rootpath[key]
-    raise KeyError(f'The "{project}" option is missing from the "rootpath" '
-                   'section in the config-user.yml file.')
+    def get_glob_patterns(self, **facets) -> list[Path]:
+        """Compose the globs that will be used to look for files."""
+        dirname_globs = _replace_tags(self.dirname_template, facets)
+        filename_globs = _replace_tags(self.filename_template, facets)
+        return sorted(self.rootpath / d / f for d in dirname_globs
+                      for f in filename_globs)
 
+    def find_files(self, **facets) -> list[LocalFile]:
+        """Find files."""
+        globs = self.get_glob_patterns(**facets)
+        logger.debug("Looking for files matching %s", globs)
 
-def _get_globs(variable):
-    """Compose the globs that will be used to look for files."""
-    project = variable['project']
+        files = []
+        for glob_ in globs:
+            for filename in glob(str(glob_)):
+                file = LocalFile(filename)
+                file.facets.update(_path2facets(file, self.dirname_template))
+                files.append(file)
+        files.sort()  # sorting makes it easier to see what was found
 
-    rootpaths = _get_rootpath(project)
+        if 'timerange' in facets:
+            files = _select_files(files, facets['timerange'])
+        return files
 
-    dirname_template = _select_drs('input_dir', project)
-    dirname_globs = _replace_tags(dirname_template, variable)
 
-    filename_template = _select_drs('input_file', project)
-    filename_globs = _replace_tags(filename_template, variable)
-
-    globs = sorted(r / d / f for r in rootpaths for d in dirname_globs
-                   for f in filename_globs)
-    return globs
-
-
-def _get_input_filelist(variable):
-    """Return the full path to input files."""
-    variable = dict(variable)
-    if 'original_short_name' in variable:
-        variable['short_name'] = variable['original_short_name']
-
-    globs = _get_globs(variable)
-    logger.debug(
-        "Looking for files matching:\n%s",
-        "\n".join(str(g) for g in globs),
-    )
+_ROOTPATH_WARNED = set()
 
-    files = list(Path(file) for glob_ in globs for file in glob(str(glob_)))
-    files.sort()  # sorting makes it easier to see what was found
 
-    if 'timerange' in variable:
-        files = _select_files(files, variable['timerange'])
+def _get_data_sources(project: str) -> list[DataSource]:
+    """Get a list of data sources."""
+    rootpaths = CFG['rootpath']
+    for key in (project, 'default'):
+        if key in rootpaths:
+            paths = rootpaths[key]
+            nonexistent = tuple(p for p in paths if not os.path.exists(p))
+            if nonexistent and (key, nonexistent) not in _ROOTPATH_WARNED:
+                logger.warning(
+                    "'%s' rootpaths '%s' set in config-user.yml do not exist",
+                    key, ', '.join(str(p) for p in nonexistent))
+                _ROOTPATH_WARNED.add((key, nonexistent))
+            if isinstance(paths, list):
+                structure = CFG['drs'].get(project, 'default')
+                paths = {p: structure for p in paths}
+            sources: list[DataSource] = []
+            for path, structure in paths.items():
+                dir_templates = _select_drs('input_dir', project, structure)
+                file_templates = _select_drs('input_file', project, structure)
+                sources.extend(
+                    DataSource(path, d, f)
+                    for d in dir_templates for f in file_templates
+                )
+            return sources
 
-    return files, globs
+    raise KeyError(
+        f"No '{project}' or 'default' path specified under 'rootpath' in "
+        "the user configuration.")
 
 
 def _get_output_file(variable: dict[str, Any], preproc_dir: Path) -> Path:
@@ -626,19 +637,18 @@ def find_files(
     list[LocalFile]
         The files that were found.
     """  # pylint: disable=line-too-long
-    filenames, globs = _get_input_filelist(facets)
-    drs = _select_drs('input_dir', facets['project'])
-    if isinstance(drs, list):
-        # Not sure how to handle a list of DRSs
-        drs = ''
+    facets = dict(facets)
+    if 'original_short_name' in facets:
+        facets['short_name'] = facets['original_short_name']
+
     files = []
     filter_latest = False
-    for filename in filenames:
-        file = LocalFile(filename)
-        file.facets.update(_path2facets(file, drs))
-        if file.facets.get('version') == 'latest':
-            filter_latest = True
-        files.append(file)
+    data_sources = _get_data_sources(facets['project'])  # type: ignore
+    for data_source in data_sources:
+        for file in data_source.find_files(**facets):
+            if file.facets.get('version') == 'latest':
+                filter_latest = True
+            files.append(file)
 
     if filter_latest:
         files = _filter_versions_called_latest(files)
@@ -646,8 +656,13 @@ def find_files(
     if 'version' not in facets:
         files = _select_latest_version(files)
 
+    files.sort()  # sorting makes it easier to see what was found
+
     if debug:
-        return files, globs
+        globs = []
+        for data_source in data_sources:
+            globs.extend(data_source.get_glob_patterns(**facets))
+        return files, sorted(globs)
     return files
 
 


Test patch:
diff --git a/tests/integration/cmor/_fixes/icon/test_icon.py b/tests/integration/cmor/_fixes/icon/test_icon.py
index 24fd1d4ba..b1607d07b 100644
--- a/tests/integration/cmor/_fixes/icon/test_icon.py
+++ b/tests/integration/cmor/_fixes/icon/test_icon.py
@@ -516,6 +516,7 @@ def test_get_areacella_fix():
 def test_areacella_fix(cubes_grid):
     """Test fix."""
     fix = get_allvars_fix('fx', 'areacella')
+    fix.extra_facets['var_type'] = 'fx'
     fixed_cubes = fix.fix_metadata(cubes_grid)
 
     assert len(fixed_cubes) == 1
@@ -538,6 +539,7 @@ def test_get_areacello_fix():
 def test_areacello_fix(cubes_grid):
     """Test fix."""
     fix = get_allvars_fix('Ofx', 'areacello')
+    fix.extra_facets['var_type'] = 'fx'
     fixed_cubes = fix.fix_metadata(cubes_grid)
 
     assert len(fixed_cubes) == 1
diff --git a/tests/integration/conftest.py b/tests/integration/conftest.py
index 4bc1f406e..fbddec7fe 100644
--- a/tests/integration/conftest.py
+++ b/tests/integration/conftest.py
@@ -19,7 +19,7 @@ from esmvalcore.local import (
 def session(tmp_path: Path, monkeypatch):
     CFG.clear()
     CFG.update(CFG_DEFAULT)
-    monkeypatch.setitem(CFG, 'rootpath', {'default': str(tmp_path)})
+    monkeypatch.setitem(CFG, 'rootpath', {'default': {tmp_path: 'default'}})
 
     session = CFG.start_session('recipe_test')
     session['output_dir'] = tmp_path / 'esmvaltool_output'
@@ -56,8 +56,8 @@ def _get_files(root_path, facets, tracking_id):
         all_facets = [facets]
 
     # Globs without expanded facets
-    dir_template = _select_drs('input_dir', facets['project'])
-    file_template = _select_drs('input_file', facets['project'])
+    dir_template = _select_drs('input_dir', facets['project'], 'default')
+    file_template = _select_drs('input_file', facets['project'], 'default')
     dir_globs = _replace_tags(dir_template, facets)
     file_globs = _replace_tags(file_template, facets)
     globs = sorted(
@@ -67,8 +67,12 @@ def _get_files(root_path, facets, tracking_id):
     files = []
     for expanded_facets in all_facets:
         filenames = []
-        dir_template = _select_drs('input_dir', expanded_facets['project'])
-        file_template = _select_drs('input_file', expanded_facets['project'])
+        dir_template = _select_drs(
+            'input_dir', expanded_facets['project'], 'default'
+        )
+        file_template = _select_drs(
+            'input_file', expanded_facets['project'], 'default'
+        )
         dir_globs = _replace_tags(dir_template, expanded_facets)
         file_globs = _replace_tags(file_template, expanded_facets)
         filename = (
diff --git a/tests/integration/recipe/test_recipe.py b/tests/integration/recipe/test_recipe.py
index c50d837b6..e467ae5c4 100644
--- a/tests/integration/recipe/test_recipe.py
+++ b/tests/integration/recipe/test_recipe.py
@@ -2023,7 +2023,7 @@ def test_weighting_landsea_fraction_exclude(tmp_path, patched_datafinder,
                 additional_datasets:
                   - {dataset: CanESM2}
                   - {dataset: GFDL-CM3}
-                  - {dataset: TEST, project: obs4MIPs,
+                  - {dataset: TEST, project: obs4MIPs, tier: 1,
                      supplementary_variables: [{short_name: sftlf, mip: fx}]}
             scripts: null
         """)
diff --git a/tests/integration/test_local.py b/tests/integration/test_local.py
index 811431b33..74830130c 100644
--- a/tests/integration/test_local.py
+++ b/tests/integration/test_local.py
@@ -91,8 +91,8 @@ def test_find_files(monkeypatch, root, cfg):
     ref_globs = [
         Path(root, d, f) for d in cfg['dirs'] for f in cfg['file_patterns']
     ]
-    assert sorted([Path(f) for f in input_filelist]) == sorted(ref_files)
-    assert sorted([Path(g) for g in globs]) == sorted(ref_globs)
+    assert [Path(f) for f in input_filelist] == sorted(ref_files)
+    assert [Path(g) for g in globs] == sorted(ref_globs)
 
 
 def test_find_files_with_facets(monkeypatch, root):
diff --git a/tests/unit/config/test_config.py b/tests/unit/config/test_config.py
index c84002779..ec88e281e 100644
--- a/tests/unit/config/test_config.py
+++ b/tests/unit/config/test_config.py
@@ -6,7 +6,7 @@ import yaml
 
 import esmvalcore
 from esmvalcore.cmor.check import CheckLevels
-from esmvalcore.config import CFG, _config
+from esmvalcore.config import CFG, _config, _config_validators
 from esmvalcore.config._config import (
     _deep_update,
     _load_extra_facets,
@@ -248,6 +248,9 @@ def test_project_obs4mips_case_correction(tmp_path, monkeypatch, mocker):
     assert 'obs4mips' not in _config.CFG
     assert _config.CFG['obs4MIPs'] == project_cfg
 
+    # Restore config-developer file
+    _config_validators.validate_config_developer(None)
+
 
 def test_load_config_developer_custom(tmp_path, monkeypatch, mocker):
     monkeypatch.setattr(_config, 'CFG', {})
@@ -261,6 +264,9 @@ def test_load_config_developer_custom(tmp_path, monkeypatch, mocker):
 
     assert 'custom' in _config.CFG
 
+    # Restore config-developer file
+    _config_validators.validate_config_developer(None)
+
 
 @pytest.mark.parametrize(
     'project,step',
diff --git a/tests/unit/config/test_config_validator.py b/tests/unit/config/test_config_validator.py
index b4d40b512..36dcd763f 100644
--- a/tests/unit/config/test_config_validator.py
+++ b/tests/unit/config/test_config_validator.py
@@ -21,6 +21,7 @@ from esmvalcore.config._config_validators import (
     validate_path,
     validate_path_or_none,
     validate_positive,
+    validate_rootpath,
     validate_search_esgf,
     validate_string,
     validate_string_or_none,
@@ -121,6 +122,7 @@ def generate_validator_testcases(valid):
                 ('a/b/c', Path.cwd() / 'a' / 'b' / 'c'),
                 ('/a/b/c/', Path('/', 'a', 'b', 'c')),
                 ('~/', Path.home()),
+                (Path.home(), Path.home()),
             ),
             'fail': (
                 (None, ValueError),
@@ -143,6 +145,49 @@ def generate_validator_testcases(valid):
                 ([], ValueError),
             ),
         },
+        {
+            'validator':
+            validate_rootpath,
+            'success': (
+                # Test a single path
+                ({
+                    'default': '/a'
+                }, {
+                    'default': [Path('/a')]
+                }),
+                ({
+                    'default': Path('/a')
+                }, {
+                    'default': [Path('/a')]
+                }),
+                # Test a list of paths
+                ({
+                    'CMIP6': ['/a', '/b']
+                }, {
+                    'CMIP6': [Path('/a'), Path('/b')]
+                }),
+                ({
+                    'CMIP6': [Path('/a'), Path('/b')]
+                }, {
+                    'CMIP6': [Path('/a'), Path('/b')]
+                }),
+                # Test a dict of paths
+                (
+                    {
+                        'CMIP6': {
+                            '/a': 'DKRZ',
+                            '/b': 'ESGF',
+                        },
+                    },
+                    {
+                        'CMIP6': {
+                            Path('/a'): 'DKRZ',
+                            Path('/b'): 'ESGF',
+                        },
+                    },
+                )),
+            'fail': (),
+        },
         {
             'validator': validate_positive,
             'success': (
@@ -287,3 +332,6 @@ def test_validate_config_developer(tmp_path):
 
     path = validate_config_developer(cfg_dev_file)
     assert path == cfg_dev_file
+
+    # Restore original config-developer file
+    validate_config_developer(None)
diff --git a/tests/unit/local/test_get_data_sources.py b/tests/unit/local/test_get_data_sources.py
new file mode 100644
index 000000000..3def03462
--- /dev/null
+++ b/tests/unit/local/test_get_data_sources.py
@@ -0,0 +1,65 @@
+from pathlib import Path
+
+import pytest
+
+from esmvalcore.config import CFG
+from esmvalcore.config._config_validators import validate_config_developer
+from esmvalcore.local import DataSource, _get_data_sources
+
+
+@pytest.mark.parametrize('rootpath_drs', [
+    (
+        {
+            'CMIP6': {
+                '/climate_data': 'ESGF'
+            }
+        },
+        {},
+    ),
+    (
+        {
+            'CMIP6': ['/climate_data']
+        },
+        {
+            'CMIP6': 'ESGF'
+        },
+    ),
+    (
+        {
+            'default': ['/climate_data']
+        },
+        {
+            'CMIP6': 'ESGF'
+        },
+    ),
+])
+def test_get_data_sources(monkeypatch, rootpath_drs):
+    # Make sure that default config-developer file is used
+    validate_config_developer(None)
+
+    rootpath, drs = rootpath_drs
+    monkeypatch.setitem(CFG, 'rootpath', rootpath)
+    monkeypatch.setitem(CFG, 'drs', drs)
+    sources = _get_data_sources('CMIP6')
+    source = sources[0]
+    assert isinstance(source, DataSource)
+    assert source.rootpath == Path('/climate_data')
+    assert '{project}' in source.dirname_template
+    assert '{short_name}' in source.filename_template
+
+
+def test_get_data_sources_nodefault(monkeypatch):
+    # Make sure that default config-developer file is used
+    validate_config_developer(None)
+
+    monkeypatch.setitem(
+        CFG,
+        'rootpath',
+        {
+            'CMIP5': {
+                '/climate_data': 'default'
+            },
+        },
+    )
+    with pytest.raises(KeyError):
+        _get_data_sources('CMIP6')
diff --git a/tests/unit/local/test_get_rootpath.py b/tests/unit/local/test_get_rootpath.py
deleted file mode 100644
index e300db199..000000000
--- a/tests/unit/local/test_get_rootpath.py
+++ /dev/null
@@ -1,28 +0,0 @@
-"""Tests for ``_get_rootpath`` in ``esmvalcore.local``."""
-from unittest import mock
-
-import pytest
-
-from esmvalcore import local
-
-
-@mock.patch("os.path.exists")
-def test_get_rootpath_exists(mexists):
-    mexists.return_value = True
-    cfg = {"rootpath": {"CMIP5": ["/path1"], "CMIP6": ["/path2"]}}
-    project = "CMIP5"
-    with mock.patch.dict(local.CFG, cfg):
-        output = local._get_rootpath(project)
-    # 'output' is a list containing a PosixPath:
-    assert str(output[0]) == cfg["rootpath"][project][0]
-
-
-@mock.patch("os.path.exists")
-def test_get_rootpath_does_not_exist(mexists):
-    mexists.return_value = False
-    cfg = {"rootpath": {"CMIP5": ["path1"], "CMIP6": ["path2"]}}
-    project = "OBS"
-    with mock.patch.dict(local.CFG, cfg):
-        msg = rf"The \"{project}\" option is missing.*"
-        with pytest.raises(KeyError, match=msg):
-            local._get_rootpath(project)
diff --git a/tests/unit/test_dataset.py b/tests/unit/test_dataset.py
index 951f3f17f..25129dbf5 100644
--- a/tests/unit/test_dataset.py
+++ b/tests/unit/test_dataset.py
@@ -1189,28 +1189,26 @@ def test_from_recipe_with_glob(tmp_path, session, mocker):
           tas:
             project: CMIP5
             mip: Amon
+            exp: rcp85
+            ensemble: r1i1p1
             additional_datasets:
-              - {dataset: '*'}
+              - {dataset: '*', institute: '*'}
     """)
     recipe = tmp_path / 'recipe_test.yml'
     recipe.write_text(recipe_txt, encoding='utf-8')
 
     session['drs']['CMIP5'] = 'ESGF'
-
+    CFG['rootpath']['CMIP5'] = [tmp_path]
     filenames = [
-        "cmip5/output1/CSIRO-QCCCE/CSIRO-Mk3-6-0/rcp85/mon/atmos/Amon/r4i1p1/"
-        "v20120323/tas_Amon_CSIRO-Mk3-6-0_rcp85_r4i1p1_200601-210012.nc",
-        "cmip5/output1/NIMR-KMA/HadGEM2-AO/historical/mon/atmos/Amon/r1i1p1/"
-        "v20130815/tas_Amon_HadGEM2-AO_historical_r1i1p1_186001-200512.nc",
+        "cmip5/output1/CSIRO-QCCCE/CSIRO-Mk3-6-0/rcp85/mon/atmos/Amon/r1i1p1/"
+        "v20120323/tas_Amon_CSIRO-Mk3-6-0_rcp85_r1i1p1_200601-210012.nc",
+        "cmip5/output1/NIMR-KMA/HadGEM2-AO/rcp85/mon/atmos/Amon/r1i1p1/"
+        "v20130815/tas_Amon_HadGEM2-AO_rcp85_r1i1p1_186001-200512.nc",
     ]
-
-    mocker.patch.object(
-        esmvalcore.local,
-        '_get_input_filelist',
-        autospec=True,
-        spec_set=True,
-        return_value=(filenames, []),
-    )
+    for filename in filenames:
+        path = tmp_path / filename
+        path.parent.mkdir(parents=True, exist_ok=True)
+        path.write_text('')
 
     definitions = [
         {
@@ -1222,6 +1220,9 @@ def test_from_recipe_with_glob(tmp_path, session, mocker):
             'short_name': 'tas',
             'alias': 'CSIRO-Mk3-6-0',
             'recipe_dataset_index': 0,
+            'exp': 'rcp85',
+            'ensemble': 'r1i1p1',
+            'institute': 'CSIRO-QCCCE',
         },
         {
             'diagnostic': 'diagnostic1',
@@ -1232,6 +1233,9 @@ def test_from_recipe_with_glob(tmp_path, session, mocker):
             'short_name': 'tas',
             'alias': 'HadGEM2-AO',
             'recipe_dataset_index': 1,
+            'exp': 'rcp85',
+            'ensemble': 'r1i1p1',
+            'institute': 'NIMR-KMA',
         },
     ]
     expected = []