Repository: ESMValGroup/ESMValCore

Issue:
ICON: specify horizontal grid file in recipe
Currently, the ICON grid file is read from the file attribute `grid_file_uri`. However, some ICON files do not have this attribute. Thus, it would be good to be able to specify the location of this grid file (absolute or relative to `auxiliary_data_dir`) in the recipe.

Discussion:
codecov[bot]: ## [Codecov](https://app.codecov.io/gh/ESMValGroup/ESMValCore/pull/2078?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ESMValGroup) Report
> Merging [#2078](https://app.codecov.io/gh/ESMValGroup/ESMValCore/pull/2078?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ESMValGroup) (475de6d) into [main](https://app.codecov.io/gh/ESMValGroup/ESMValCore/commit/6e1438803130581c7ad6138db4c162bfef0906b9?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ESMValGroup) (6e14388) will **increase** coverage by `0.00%`.
> The diff coverage is `100.00%`.

```diff
@@           Coverage Diff           @@
##             main    #2078   +/-   ##
=======================================
  Coverage   92.97%   92.97%           
=======================================
  Files         235      235           
  Lines       12604    12620   +16     
=======================================
+ Hits        11718    11734   +16     
  Misses        886      886           
```


| [Impacted Files](https://app.codecov.io/gh/ESMValGroup/ESMValCore/pull/2078?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ESMValGroup) | Coverage Δ | |
|---|---|---|
| [esmvalcore/cmor/\_fixes/icon/\_base\_fixes.py](https://app.codecov.io/gh/ESMValGroup/ESMValCore/pull/2078?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ESMValGroup#diff-ZXNtdmFsY29yZS9jbW9yL19maXhlcy9pY29uL19iYXNlX2ZpeGVzLnB5) | `100.00% <100.00%> (ø)` | |



Patch:
diff --git a/doc/quickstart/find_data.rst b/doc/quickstart/find_data.rst
index 5a810081b..20da29ac0 100644
--- a/doc/quickstart/find_data.rst
+++ b/doc/quickstart/find_data.rst
@@ -370,6 +370,17 @@ is always disabled.
    See :ref:`timerange_examples` for more information on the ``timerange``
    option.
 
+Usually, ESMValCore will need the corresponding ICON grid file of your
+simulation to work properly (examples: setting latitude/longitude coordinates
+if these are not yet present, UGRIDization [see below], etc.).
+This grid file can either be specified as absolute or relative (to
+``auxiliary_data_dir`` as defined in the :ref:`user configuration file`) path
+with the facet ``horizontal_grid`` in the recipe or the extra facets (see
+below), or retrieved automatically from the `grid_file_uri` attribute of the
+input files.
+In the latter case, the file is downloaded once and then cached.
+The cached file is valid for 7 days.
+
 ESMValCore can automatically make native ICON data `UGRID
 <https://ugrid-conventions.github.io/ugrid-conventions/>`__-compliant when
 loading the data.
@@ -393,8 +404,7 @@ algorithm <https://earthsystemmodeling.org/regrid/#regridding-methods>`__:
        regrid:
          target_grid: 1x1
          scheme:
-           reference: esmf_regrid.experimental.unstructured_scheme:regrid_unstructured_to_rectilinear
-           method: conservative
+           reference: esmf_regrid.schemes:ESMFAreaWeighted
 
 This automatic UGRIDization is enabled by default, but can be switched off with
 the facet ``ugrid: false`` in the recipe or the extra facets (see below).
@@ -445,6 +455,11 @@ Supported keys for extra facets are:
 =================== ================================ ===================================
 Key                 Description                      Default value if not specified
 =================== ================================ ===================================
+``horizontal_grid`` Absolute or relative (to         If not given, use file attribute
+                    ``auxiliary_data_dir`` defined   ``grid_file_uri`` to retrieve ICON
+                    in the                           grid file
+                    :ref:`user configuration file`)
+                    path to the ICON grid file
 ``latitude``        Standard name of the latitude    ``latitude``
                     coordinate in the raw input
                     file
@@ -454,13 +469,13 @@ Key                 Description                      Default value if not specif
 ``raw_name``        Variable name of the             CMOR variable name of the
                     variable in the raw input        corresponding variable
                     file
-``shift_time``      Shift time points back by 1/2 of ``True``
-                    the corresponding output time
-                    interval
 ``raw_units``       Units of the variable in the     If specified, the value given by
                     raw input file                   the ``units`` attribute in the
                                                      raw input file; otherwise
                                                      ``unknown``
+``shift_time``      Shift time points back by 1/2 of ``True``
+                    the corresponding output time
+                    interval
 ``ugrid``           Automatic UGRIDization of        ``True``
                     the input data
 ``var_type``        Variable type of the             No default (needs to be specified
@@ -498,6 +513,18 @@ Key                 Description                      Default value if not specif
    default naming conventions listed above).
    Make sure that no other variable uses this ``var_type``.
 
+   If you want to use the :func:`~esmvalcore.preprocessor.area_statistics`
+   preprocessor on *regridded* ICON data, make sure to **not** use the cell area
+   files by using the ``skip: true`` syntax in the recipe as described in
+   :ref:`preprocessors_using_supplementary_variables`, e.g.,
+
+   .. code-block:: yaml
+
+     datasets:
+       - {project: ICON, dataset: ICON, exp: amip,
+          supplementary_variables: [{short_name: areacella, skip: true}]}
+
+
 .. _read_ipsl-cm6:
 
 IPSL-CM6
diff --git a/environment.yml b/environment.yml
index fed703cce..fbe3cc990 100644
--- a/environment.yml
+++ b/environment.yml
@@ -21,7 +21,7 @@ dependencies:
   - humanfriendly
   - importlib_resources
   - iris>=3.6.0
-  - iris-esmf-regrid >=0.6.0  # to work with latest esmpy
+  - iris-esmf-regrid >=0.7.0  # to work with latest esmpy
   - isodate
   - jinja2
   - libnetcdf!=4.9.1 # to avoid hdf5 warnings
diff --git a/esmvalcore/cmor/_fixes/icon/_base_fixes.py b/esmvalcore/cmor/_fixes/icon/_base_fixes.py
index cfe6ab949..4fc67f490 100644
--- a/esmvalcore/cmor/_fixes/icon/_base_fixes.py
+++ b/esmvalcore/cmor/_fixes/icon/_base_fixes.py
@@ -248,41 +248,29 @@ class IconFix(NativeDatasetFix):
 
         return cubes
 
-    def get_horizontal_grid(self, cube):
-        """Get copy of ICON horizontal grid from global attribute of cube.
-
-        Note
-        ----
-        If necessary, this functions downloads the grid file to a cache
-        directory. The downloaded file is valid for 7 days until it is
-        downloaded again.
-
-        Parameters
-        ----------
-        cube: iris.cube.Cube
-            Cube for which the ICON horizontal grid is retrieved. This cube
-            needs to have a global attribute that specifies the download
-            location of the ICON horizontal grid file (see
-            ``self.GRID_FILE_ATTR``).
+    def _get_grid_from_facet(self):
+        """Get horizontal grid from user-defined facet `horizontal_grid`."""
+        grid_path = self._get_path_from_facet(
+            'horizontal_grid', 'Horizontal grid file'
+        )
+        grid_name = grid_path.name
 
-        Returns
-        -------
-        iris.cube.CubeList
-            Copy of ICON horizontal grid.
+        # If already loaded, return the horizontal grid
+        if grid_name in self._horizontal_grids:
+            return self._horizontal_grids[grid_name]
 
-        Raises
-        ------
-        ValueError
-            Input cube does not contain the necessary attribute that specifies
-            the download location of the ICON horizontal grid file (see
-            ``self.GRID_FILE_ATTR``).
+        # Load file
+        self._horizontal_grids[grid_name] = self._load_cubes(grid_path)
+        logger.debug("Loaded ICON grid file from %s", grid_path)
+        return self._horizontal_grids[grid_name]
 
-        """
+    def _get_grid_from_cube_attr(self, cube):
+        """Get horizontal grid from `grid_file_uri` attribute of cube."""
         (grid_url, grid_name) = self._get_grid_url(cube)
 
-        # If already loaded, return the horizontal grid (cube)
+        # If already loaded, return the horizontal grid
         if grid_name in self._horizontal_grids:
-            return self._horizontal_grids[grid_name].copy()
+            return self._horizontal_grids[grid_name]
 
         # Check if grid file has recently been downloaded and load it if
         # possible
@@ -302,7 +290,7 @@ class IconFix(NativeDatasetFix):
                     self._horizontal_grids[grid_name] = self._load_cubes(
                         grid_path
                     )
-                    return self._horizontal_grids[grid_name].copy()
+                    return self._horizontal_grids[grid_name]
                 logger.debug("Existing cached ICON grid file '%s' is outdated",
                              grid_path)
 
@@ -325,7 +313,51 @@ class IconFix(NativeDatasetFix):
 
             self._horizontal_grids[grid_name] = self._load_cubes(grid_path)
 
-        return self._horizontal_grids[grid_name].copy()
+        return self._horizontal_grids[grid_name]
+
+    def get_horizontal_grid(self, cube):
+        """Get copy of ICON horizontal grid.
+
+        If given, retrieve grid from `horizontal_grid` facet specified by the
+        user. Otherwise, try to download the file from the location given by
+        the global attribute `grid_file_uri` of the cube.
+
+        Note
+        ----
+        If necessary, this functions downloads the grid file to a cache
+        directory. The downloaded file is valid for 7 days until it is
+        downloaded again.
+
+        Parameters
+        ----------
+        cube: iris.cube.Cube
+            Cube for which the ICON horizontal grid is retrieved. If the facet
+            `horizontal_grid` is not specified by the user, this cube needs to
+            have a global attribute `grid_file_uri` that specifies the download
+            location of the ICON horizontal grid file.
+
+        Returns
+        -------
+        iris.cube.CubeList
+            Copy of ICON horizontal grid.
+
+        Raises
+        ------
+        FileNotFoundError
+            Path specified by `horizontal_grid` facet (absolute or relative to
+            `auxiliary_data_dir`) does not exist.
+        ValueError
+            Input cube does not contain the necessary attribute `grid_file_uri`
+            that specifies the download location of the ICON horizontal grid
+            file.
+
+        """
+        if 'horizontal_grid' in self.extra_facets:
+            grid = self._get_grid_from_facet()
+        else:
+            grid = self._get_grid_from_cube_attr(cube)
+
+        return grid.copy()
 
     def get_mesh(self, cube):
         """Get mesh.
@@ -338,9 +370,10 @@ class IconFix(NativeDatasetFix):
         Parameters
         ----------
         cube: iris.cube.Cube
-            Cube for which the mesh is retrieved. This cube needs to have a
-            global attribute that specifies the download location of the ICON
-            horizontal grid file (see ``self.GRID_FILE_ATTR``).
+            Cube for which the mesh is retrieved. If the facet
+            `horizontal_grid` is not specified by the user, this cube needs to
+            have the global attribute `grid_file_uri` that specifies the
+            download location of the ICON horizontal grid file.
 
         Returns
         -------
@@ -349,17 +382,32 @@ class IconFix(NativeDatasetFix):
 
         Raises
         ------
+        FileNotFoundError
+            Path specified by `horizontal_grid` facet (absolute or relative to
+            `auxiliary_data_dir`) does not exist.
         ValueError
-            Input cube does not contain the necessary attribute that specifies
-            the the ICON horizontal grid file (see ``self.GRID_FILE_ATTR``).
+            Input cube does not contain the necessary attribute `grid_file_uri`
+            that specifies the download location of the ICON horizontal grid
+            file.
 
         """
-        (_, grid_name) = self._get_grid_url(cube)
+        # If specified by the user, use `horizontal_grid` facet to determine
+        # grid name; otherwise, use the `grid_file_uri` attribute of the cube
+        if 'horizontal_grid' in self.extra_facets:
+            grid_path = self._get_path_from_facet(
+                'horizontal_grid', 'Horizontal grid file'
+            )
+            grid_name = grid_path.name
+        else:
+            (_, grid_name) = self._get_grid_url(cube)
+
+        # Re-use mesh if possible
         if grid_name in self._meshes:
             logger.debug("Reusing ICON mesh for grid %s", grid_name)
         else:
             logger.debug("Creating ICON mesh for grid %s", grid_name)
             self._meshes[grid_name] = self._create_mesh(cube)
+
         return self._meshes[grid_name]
 
     @staticmethod


Test patch:
diff --git a/tests/integration/cmor/_fixes/icon/test_icon.py b/tests/integration/cmor/_fixes/icon/test_icon.py
index 61a382bf6..297058a9a 100644
--- a/tests/integration/cmor/_fixes/icon/test_icon.py
+++ b/tests/integration/cmor/_fixes/icon/test_icon.py
@@ -1174,22 +1174,40 @@ def test_add_coord_from_grid_fail_two_unnamed_dims(cubes_2d):
         fix._add_coord_from_grid_file(tas_cube, 'latitude')
 
 
+# Test get_horizontal_grid
+
+
+@mock.patch.object(IconFix, '_get_grid_from_facet', autospec=True)
 @mock.patch('esmvalcore.cmor._fixes.icon._base_fixes.requests', autospec=True)
-def test_get_horizontal_grid_cached_in_dict(mock_requests):
+def test_get_horizontal_grid_from_attr_cached_in_dict(
+    mock_requests,
+    mock_get_grid_from_facet,
+):
     """Test fix."""
     cube = Cube(0, attributes={'grid_file_uri': 'cached_grid_url.nc'})
     grid_cube = Cube(0)
     fix = get_allvars_fix('Amon', 'tas')
     fix._horizontal_grids['cached_grid_url.nc'] = grid_cube
+    fix._horizontal_grids['grid_from_facet.nc'] = mock.sentinel.wrong_grid
 
     grid = fix.get_horizontal_grid(cube)
+    assert len(fix._horizontal_grids) == 2
+    assert 'cached_grid_url.nc' in fix._horizontal_grids
+    assert 'grid_from_facet.nc' in fix._horizontal_grids  # has not been used
+    assert fix._horizontal_grids['cached_grid_url.nc'] == grid
     assert grid == grid_cube
     assert grid is not grid_cube
     assert mock_requests.mock_calls == []
+    mock_get_grid_from_facet.assert_not_called()
 
 
+@mock.patch.object(IconFix, '_get_grid_from_facet', autospec=True)
 @mock.patch('esmvalcore.cmor._fixes.icon._base_fixes.requests', autospec=True)
-def test_get_horizontal_grid_cached_in_file(mock_requests, tmp_path):
+def test_get_horizontal_grid_from_attr_cached_in_file(
+    mock_requests,
+    mock_get_grid_from_facet,
+    tmp_path,
+):
     """Test fix."""
     cube = Cube(0, attributes={
         'grid_file_uri': 'https://temporary.url/this/is/the/grid_file.nc'})
@@ -1206,10 +1224,17 @@ def test_get_horizontal_grid_cached_in_file(mock_requests, tmp_path):
     assert grid[0].var_name == 'grid'
     assert len(fix._horizontal_grids) == 1
     assert 'grid_file.nc' in fix._horizontal_grids
+    assert fix._horizontal_grids['grid_file.nc'] == grid
     assert mock_requests.mock_calls == []
+    mock_get_grid_from_facet.assert_not_called()
 
 
-def test_get_horizontal_grid_cache_file_too_old(tmp_path, monkeypatch):
+@mock.patch.object(IconFix, '_get_grid_from_facet', autospec=True)
+def test_get_horizontal_grid_from_attr_cache_file_too_old(
+    mock_get_grid_from_facet,
+    tmp_path,
+    monkeypatch,
+):
     """Test fix."""
     cube = Cube(0, attributes={'grid_file_uri': TEST_GRID_FILE_URI})
     fix = get_allvars_fix('Amon', 'tas')
@@ -1233,6 +1258,88 @@ def test_get_horizontal_grid_cache_file_too_old(tmp_path, monkeypatch):
     assert 'vertex_of_cell' in var_names
     assert len(fix._horizontal_grids) == 1
     assert TEST_GRID_FILE_NAME in fix._horizontal_grids
+    assert fix._horizontal_grids[TEST_GRID_FILE_NAME] == grid
+    mock_get_grid_from_facet.assert_not_called()
+
+
+@mock.patch.object(IconFix, '_get_grid_from_cube_attr', autospec=True)
+def test_get_horizontal_grid_from_facet_cached_in_dict(
+    mock_get_grid_from_cube_attr,
+    tmp_path,
+):
+    """Test fix."""
+    session = CFG.start_session('my session')
+    session['auxiliary_data_dir'] = tmp_path
+
+    # Save temporary grid file (this will not be used; however, it is necessary
+    # to not raise a FileNotFoundError)
+    grid_path = 'grid.nc'
+    wrong_grid_cube = Cube(0, var_name='wrong_grid')
+    iris.save(wrong_grid_cube, tmp_path / 'grid.nc')
+
+    # Make sure that grid specified by cube attribute is NOT used
+    cube = Cube(0, attributes={'grid_file_uri': 'cached_grid_url.nc'})
+    grid_cube = Cube(0, var_name='grid')
+    fix = get_allvars_fix('Amon', 'tas', session=session)
+    fix.extra_facets['horizontal_grid'] = grid_path
+    fix._horizontal_grids['cached_grid_url.nc'] = mock.sentinel.wrong_grid
+    fix._horizontal_grids['grid.nc'] = grid_cube
+
+    grid = fix.get_horizontal_grid(cube)
+    assert len(fix._horizontal_grids) == 2
+    assert 'cached_grid_url.nc' in fix._horizontal_grids  # has not been used
+    assert 'grid.nc' in fix._horizontal_grids
+    assert fix._horizontal_grids['grid.nc'] == grid
+    assert grid == grid_cube
+    assert grid is not grid_cube
+    mock_get_grid_from_cube_attr.assert_not_called()
+
+
+@pytest.mark.parametrize('grid_path', ['{tmp_path}/grid.nc', 'grid.nc'])
+@mock.patch.object(IconFix, '_get_grid_from_cube_attr', autospec=True)
+def test_get_horizontal_grid_from_facet(
+    mock_get_grid_from_cube_attr,
+    grid_path,
+    tmp_path,
+):
+    """Test fix."""
+    session = CFG.start_session('my session')
+    session['auxiliary_data_dir'] = tmp_path
+
+    # Make sure that grid specified by cube attribute is NOT used
+    cube = Cube(0, attributes={'grid_file_uri': 'cached_grid_url.nc'})
+
+    # Save temporary grid file
+    grid_path = grid_path.format(tmp_path=tmp_path)
+    grid_cube = Cube(0, var_name='grid')
+    iris.save(grid_cube, tmp_path / 'grid.nc')
+
+    fix = get_allvars_fix('Amon', 'tas', session=session)
+    fix.extra_facets['horizontal_grid'] = grid_path
+    fix._horizontal_grids['cached_grid_url.nc'] = mock.sentinel.wrong_grid
+
+    grid = fix.get_horizontal_grid(cube)
+    assert isinstance(grid, CubeList)
+    assert len(grid) == 1
+    assert grid[0].var_name == 'grid'
+    assert len(fix._horizontal_grids) == 2
+    assert 'cached_grid_url.nc' in fix._horizontal_grids  # has not been used
+    assert 'grid.nc' in fix._horizontal_grids
+    assert fix._horizontal_grids['grid.nc'] == grid
+    mock_get_grid_from_cube_attr.assert_not_called()
+
+
+def test_get_horizontal_grid_from_facet_fail(tmp_path):
+    """Test fix."""
+    session = CFG.start_session('my session')
+    session['auxiliary_data_dir'] = tmp_path
+
+    cube = Cube(0)
+    fix = get_allvars_fix('Amon', 'tas', session=session)
+    fix.extra_facets['horizontal_grid'] = '/this/does/not/exist.nc'
+
+    with pytest.raises(FileNotFoundError):
+        fix.get_horizontal_grid(cube)
 
 
 # Test with single-dimension cubes
@@ -1872,7 +1979,7 @@ def test_get_grid_url_fail():
 # Test get_mesh
 
 
-def test_get_mesh_cached(monkeypatch):
+def test_get_mesh_cached_from_attr(monkeypatch):
     """Test fix."""
     cube = Cube(0, attributes={'grid_file_uri': TEST_GRID_FILE_URI})
     fix = get_allvars_fix('Amon', 'tas')
@@ -1883,7 +1990,7 @@ def test_get_mesh_cached(monkeypatch):
     fix._create_mesh.assert_not_called()
 
 
-def test_get_mesh_not_cached(monkeypatch):
+def test_get_mesh_not_cached_from_attr(monkeypatch):
     """Test fix."""
     cube = Cube(0, attributes={'grid_file_uri': TEST_GRID_FILE_URI})
     fix = get_allvars_fix('Amon', 'tas')
@@ -1892,6 +1999,52 @@ def test_get_mesh_not_cached(monkeypatch):
     fix._create_mesh.assert_called_once_with(cube)
 
 
+def test_get_mesh_cached_from_facet(monkeypatch, tmp_path):
+    """Test fix."""
+    session = CFG.start_session('my session')
+    session['auxiliary_data_dir'] = tmp_path
+
+    # Save temporary grid file (this will not be used; however, it is necessary
+    # to not raise a FileNotFoundError)
+    grid_path = 'grid.nc'
+    grid_cube = Cube(0, var_name='grid')
+    iris.save(grid_cube, tmp_path / 'grid.nc')
+
+    cube = Cube(0, attributes={'grid_file_uri': TEST_GRID_FILE_URI})
+    fix = get_allvars_fix('Amon', 'tas', session=session)
+    fix.extra_facets['horizontal_grid'] = grid_path
+    monkeypatch.setattr(fix, '_create_mesh', mock.Mock())
+    fix._meshes[TEST_GRID_FILE_NAME] = mock.sentinel.wrong_mesh
+    fix._meshes['grid.nc'] = mock.sentinel.mesh
+
+    mesh = fix.get_mesh(cube)
+
+    assert mesh == mock.sentinel.mesh
+    fix._create_mesh.assert_not_called()
+
+
+def test_get_mesh_not_cached_from_facet(monkeypatch, tmp_path):
+    """Test fix."""
+    session = CFG.start_session('my session')
+    session['auxiliary_data_dir'] = tmp_path
+
+    # Save temporary grid file (this will not be used; however, it is necessary
+    # to not raise a FileNotFoundError)
+    grid_path = 'grid.nc'
+    grid_cube = Cube(0, var_name='grid')
+    iris.save(grid_cube, tmp_path / 'grid.nc')
+
+    cube = Cube(0, attributes={'grid_file_uri': TEST_GRID_FILE_URI})
+    fix = get_allvars_fix('Amon', 'tas', session=session)
+    fix.extra_facets['horizontal_grid'] = grid_path
+    monkeypatch.setattr(fix, '_create_mesh', mock.Mock())
+    fix._meshes[TEST_GRID_FILE_NAME] = mock.sentinel.wrong_mesh
+
+    fix.get_mesh(cube)
+
+    fix._create_mesh.assert_called_once_with(cube)
+
+
 # Test _get_path_from_facet