Repository: ESMValGroup/ESMValCore

Issue:
`ignore_warnings` in cfg-developer.yml is not considered anymore
The `ignore_warnings` key ([example](https://github.com/ESMValGroup/ESMValCore/blob/8b6be3c3e994cae1609d8acc1cc0a7007a4d697d/esmvalcore/config-developer.yml#L138-L143)) in the `config-developer.yml` file is not considered anymore, which leads to a cluttered log when processing EMAC files.

The reason for this is that the new `Dataset.load` method does not pass this keyword argument to the `load` function:

https://github.com/ESMValGroup/ESMValCore/blob/8b6be3c3e994cae1609d8acc1cc0a7007a4d697d/esmvalcore/dataset.py#L731

Discussion:
codecov[bot]: ## [Codecov](https://app.codecov.io/gh/ESMValGroup/ESMValCore/pull/2046?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ESMValGroup) Report
> Merging [#2046](https://app.codecov.io/gh/ESMValGroup/ESMValCore/pull/2046?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ESMValGroup) (a21ef73) into [main](https://app.codecov.io/gh/ESMValGroup/ESMValCore/commit/8b6be3c3e994cae1609d8acc1cc0a7007a4d697d?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ESMValGroup) (8b6be3c) will **decrease** coverage by `0.01%`.
> The diff coverage is `100.00%`.

```diff
@@            Coverage Diff             @@
##             main    #2046      +/-   ##
==========================================
- Coverage   92.79%   92.79%   -0.01%     
==========================================
  Files         236      236              
  Lines       12484    12482       -2     
==========================================
- Hits        11585    11583       -2     
  Misses        899      899              
```


| [Impacted Files](https://app.codecov.io/gh/ESMValGroup/ESMValCore/pull/2046?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ESMValGroup) | Coverage Δ | |
|---|---|---|
| [esmvalcore/preprocessor/\_\_init\_\_.py](https://app.codecov.io/gh/ESMValGroup/ESMValCore/pull/2046?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ESMValGroup#diff-ZXNtdmFsY29yZS9wcmVwcm9jZXNzb3IvX19pbml0X18ucHk=) | `95.22% <ø> (-0.02%)` | :arrow_down: |
| [esmvalcore/dataset.py](https://app.codecov.io/gh/ESMValGroup/ESMValCore/pull/2046?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ESMValGroup#diff-ZXNtdmFsY29yZS9kYXRhc2V0LnB5) | `100.00% <100.00%> (ø)` | |



Patch:
diff --git a/doc/quickstart/configure.rst b/doc/quickstart/configure.rst
index 3ca78b9f2..eb1f618df 100644
--- a/doc/quickstart/configure.rst
+++ b/doc/quickstart/configure.rst
@@ -613,7 +613,7 @@ Currently supported preprocessor steps:
 * :func:`~esmvalcore.preprocessor.load`
 
 Here is an example on how to ignore specific warnings during the preprocessor
-step ``load`` for all datasets of project  ``EMAC`` (taken from the default
+step ``load`` for all datasets of project ``EMAC`` (taken from the default
 ``config-developer.yml`` file):
 
 .. code-block:: yaml
diff --git a/doc/recipe/preprocessor.rst b/doc/recipe/preprocessor.rst
index 8ebabc40e..3a087bef8 100644
--- a/doc/recipe/preprocessor.rst
+++ b/doc/recipe/preprocessor.rst
@@ -1381,16 +1381,17 @@ See also :func:`esmvalcore.preprocessor.decadal_statistics`.
 ----------------------
 
 This function produces statistics for the whole dataset. It can produce scalars
-(if the full period is chosen) or daily, monthly or seasonal statistics.
+(if the full period is chosen) or hourly, daily, monthly or seasonal
+statistics.
 
 Parameters:
     * operator: operation to apply. Accepted values are 'mean', 'median',
       'std_dev', 'min', 'max', 'sum' and 'rms'. Default is 'mean'
 
     * period: define the granularity of the statistics: get values for the
-      full period, for each month or day of year.
+      full period, for each month, day of year or hour of day.
       Available periods: 'full', 'season', 'seasonal', 'monthly', 'month',
-      'mon', 'daily', 'day'. Default is 'full'
+      'mon', 'daily', 'day', 'hourly', 'hour', 'hr'. Default is 'full'
 
     * seasons: if period 'seasonal' or 'season' allows to set custom seasons.
       Default is '[DJF, MAM, JJA, SON]'
@@ -1502,14 +1503,14 @@ See also :func:`esmvalcore.preprocessor.resample_hours`.
 ----------------------
 
 This function computes the anomalies for the whole dataset. It can compute
-anomalies from the full, seasonal, monthly and daily climatologies. Optionally
-standardized anomalies can be calculated.
+anomalies from the full, seasonal, monthly, daily and hourly climatologies.
+Optionally standardized anomalies can be calculated.
 
 Parameters:
     * period: define the granularity of the climatology to use:
-      full period, seasonal, monthly or daily.
+      full period, seasonal, monthly, daily or hourly.
       Available periods: 'full', 'season', 'seasonal', 'monthly', 'month',
-      'mon', 'daily', 'day'. Default is 'full'
+      'mon', 'daily', 'day', 'hourly', 'hour', 'hr'. Default is 'full'
     * reference: Time slice to use as the reference to compute the climatology
       on. Can be 'null' to use the full cube or a dictionary with the
       parameters from extract_time_. Default is null
@@ -1886,7 +1887,7 @@ that can be:
 * left closed ``[z_min, z_max)``, in which the extracted range includes ``z_min`` but not ``z_max``.
 * right closed ``(z_min, z_max]``, in which the extracted range includes ``z_max`` but not ``z_min``.
 
-The extraction is performed by applying a constraint on the coordinate values, without any kind of interpolation. 
+The extraction is performed by applying a constraint on the coordinate values, without any kind of interpolation.
 
 This function takes four arguments:
 
diff --git a/esmvalcore/_recipe/recipe.py b/esmvalcore/_recipe/recipe.py
index 4644ef524..38b7426c0 100644
--- a/esmvalcore/_recipe/recipe.py
+++ b/esmvalcore/_recipe/recipe.py
@@ -549,16 +549,6 @@ def _update_multi_dataset_settings(facets, settings):
         _exclude_dataset(settings, facets, step)
 
 
-def _update_warning_settings(settings, project):
-    """Update project-specific warning settings."""
-    cfg = get_project_config(project)
-    if 'ignore_warnings' not in cfg:
-        return
-    for (step, ignored_warnings) in cfg['ignore_warnings'].items():
-        if step in settings:
-            settings[step]['ignore_warnings'] = ignored_warnings
-
-
 def _get_tag(step, identifier, statistic):
     # Avoid . in filename for percentiles
     statistic = statistic.replace('.', '-')
@@ -697,7 +687,6 @@ def _get_preprocessor_products(
 
     for dataset in datasets:
         settings = _get_default_settings(dataset)
-        _update_warning_settings(settings, dataset.facets['project'])
         _apply_preprocessor_profile(settings, profile)
         _update_multi_dataset_settings(dataset.facets, settings)
         _update_preproc_functions(settings, dataset, datasets, missing_vars)
diff --git a/esmvalcore/config/_config.py b/esmvalcore/config/_config.py
index 4cd0b57aa..deb2f7be8 100644
--- a/esmvalcore/config/_config.py
+++ b/esmvalcore/config/_config.py
@@ -1,4 +1,6 @@
 """Functions dealing with config-user.yml / config-developer.yml."""
+from __future__ import annotations
+
 import collections.abc
 import fnmatch
 import logging
@@ -11,6 +13,7 @@ import yaml
 
 from esmvalcore.cmor.table import CMOR_TABLES, read_cmor_tables
 from esmvalcore.exceptions import RecipeError
+from esmvalcore.typing import FacetValue
 
 logger = logging.getLogger(__name__)
 
@@ -139,3 +142,15 @@ def get_activity(variable):
         return CMOR_TABLES[project].activities[exp][0]
     except (KeyError, AttributeError):
         return None
+
+
+def get_ignored_warnings(project: FacetValue, step: str) -> None | list:
+    """Get ignored warnings for a given preprocessing step."""
+    if project not in CFG:
+        return None
+    project_cfg = CFG[project]
+    if 'ignore_warnings' not in project_cfg:
+        return None
+    if step not in project_cfg['ignore_warnings']:
+        return None
+    return project_cfg['ignore_warnings'][step]
diff --git a/esmvalcore/dataset.py b/esmvalcore/dataset.py
index 5710ef665..35c858840 100644
--- a/esmvalcore/dataset.py
+++ b/esmvalcore/dataset.py
@@ -22,6 +22,7 @@ from esmvalcore.config import CFG, Session
 from esmvalcore.config._config import (
     get_activity,
     get_extra_facets,
+    get_ignored_warnings,
     get_institutes,
 )
 from esmvalcore.exceptions import InputFilesNotFound, RecipeError
@@ -729,7 +730,12 @@ class Dataset:
             'session': self.session,
             **self.facets,
         }
-        settings['load'] = {'callback': callback}
+        settings['load'] = {
+            'callback': callback,
+            'ignore_warnings': get_ignored_warnings(
+                self.facets['project'], 'load'
+            ),
+        }
         settings['fix_metadata'] = {
             'check_level': self.session['check_level'],
             'session': self.session,
diff --git a/esmvalcore/preprocessor/_supplementary_vars.py b/esmvalcore/preprocessor/_supplementary_vars.py
index 96be17d52..881373509 100644
--- a/esmvalcore/preprocessor/_supplementary_vars.py
+++ b/esmvalcore/preprocessor/_supplementary_vars.py
@@ -12,6 +12,7 @@ import iris.cube
 from esmvalcore.cmor.check import cmor_check_data, cmor_check_metadata
 from esmvalcore.cmor.fix import fix_data, fix_metadata
 from esmvalcore.config import CFG
+from esmvalcore.config._config import get_ignored_warnings
 from esmvalcore.exceptions import ESMValCoreDeprecationWarning
 from esmvalcore.preprocessor._io import concatenate, load
 from esmvalcore.preprocessor._time import clip_timerange
@@ -59,7 +60,8 @@ def _load_fx(var_cube, fx_info, check_level):
     freq = fx_info['frequency']
 
     for fx_file in fx_info['filename']:
-        loaded_cube = load(fx_file)
+        ignored_warnings = get_ignored_warnings(project, 'load')
+        loaded_cube = load(fx_file, ignore_warnings=ignored_warnings)
         loaded_cube = fix_metadata(loaded_cube,
                                    check_level=check_level,
                                    **fx_info)
diff --git a/esmvalcore/preprocessor/_time.py b/esmvalcore/preprocessor/_time.py
index 0ba0a953f..5de60f35a 100644
--- a/esmvalcore/preprocessor/_time.py
+++ b/esmvalcore/preprocessor/_time.py
@@ -704,30 +704,31 @@ def climate_statistics(cube,
     """Compute climate statistics with the specified granularity.
 
     Computes statistics for the whole dataset. It is possible to get them for
-    the full period or with the data grouped by day, month or season
+    the full period or with the data grouped by hour, day, month or season.
 
     Parameters
     ----------
     cube: iris.cube.Cube
-        input cube.
+        Input cube.
 
     operator: str, optional
         Select operator to apply.
         Available operators: 'mean', 'median', 'std_dev', 'sum', 'min',
-        'max', 'rms'
+        'max', 'rms'.
 
     period: str, optional
         Period to compute the statistic over.
         Available periods: 'full', 'season', 'seasonal', 'monthly', 'month',
-        'mon', 'daily', 'day'
+        'mon', 'daily', 'day', 'hourly', 'hour', 'hr'.
 
     seasons: list or tuple of str, optional
-        Seasons to use if needed. Defaults to ('DJF', 'MAM', 'JJA', 'SON')
+        Seasons to use if needed. Defaults to ('DJF', 'MAM', 'JJA', 'SON').
 
     Returns
     -------
     iris.cube.Cube
-        Monthly statistics cube
+        Climate statistics cube.
+
     """
     original_dtype = cube.dtype
     period = period.lower()
@@ -775,34 +776,35 @@ def anomalies(cube,
               seasons=('DJF', 'MAM', 'JJA', 'SON')):
     """Compute anomalies using a mean with the specified granularity.
 
-    Computes anomalies based on daily, monthly, seasonal or yearly means for
-    the full available period
+    Computes anomalies based on hourly, daily, monthly, seasonal or yearly
+    means for the full available period.
 
     Parameters
     ----------
     cube: iris.cube.Cube
-        input cube.
+        Input cube.
 
     period: str
         Period to compute the statistic over.
         Available periods: 'full', 'season', 'seasonal', 'monthly', 'month',
-        'mon', 'daily', 'day'
+        'mon', 'daily', 'day', 'hourly', 'hour', 'hr'.
 
     reference: list int, optional, default: None
         Period of time to use a reference, as needed for the 'extract_time'
-        preprocessor function
-        If None, all available data is used as a reference
+        preprocessor function. If ``None``, all available data is used as a
+        reference.
 
     standardize: bool, optional
-        If True standardized anomalies are calculated
+        If ``True`` standardized anomalies are calculated.
 
     seasons: list or tuple of str, optional
-        Seasons to use if needed. Defaults to ('DJF', 'MAM', 'JJA', 'SON')
+        Seasons to use if needed. Defaults to ('DJF', 'MAM', 'JJA', 'SON').
 
     Returns
     -------
     iris.cube.Cube
-        Anomalies cube
+        Anomalies cube.
+
     """
     if reference is None:
         reference_cube = cube
@@ -868,6 +870,10 @@ def _compute_anomalies(cube, reference, period, seasons):
 
 def _get_period_coord(cube, period, seasons):
     """Get periods."""
+    if period in ['hourly', 'hour', 'hr']:
+        if not cube.coords('hour'):
+            iris.coord_categorisation.add_hour(cube, 'time')
+        return cube.coord('hour')
     if period in ['daily', 'day']:
         if not cube.coords('day_of_year'):
             iris.coord_categorisation.add_day_of_year(cube, 'time')


Test patch:
diff --git a/tests/unit/config/test_config.py b/tests/unit/config/test_config.py
index cb7619970..dbb1e8d27 100644
--- a/tests/unit/config/test_config.py
+++ b/tests/unit/config/test_config.py
@@ -10,6 +10,7 @@ from esmvalcore.config._config import (
     _deep_update,
     _load_extra_facets,
     get_extra_facets,
+    get_ignored_warnings,
     importlib_files,
 )
 from esmvalcore.dataset import Dataset
@@ -243,3 +244,23 @@ def test_project_obs4mips_case_correction(tmp_path, monkeypatch, mocker):
 
     assert 'obs4mips' not in _config.CFG
     assert _config.CFG['obs4MIPs'] == project_cfg
+
+
+@pytest.mark.parametrize(
+    'project,step',
+    [
+        ('invalid_project', 'load'),
+        ('CMIP6', 'load'),
+        ('EMAC', 'save'),
+    ],
+)
+def test_get_ignored_warnings_none(project, step):
+    """Test ``get_ignored_warnings``."""
+    assert get_ignored_warnings(project, step) is None
+
+
+def test_get_ignored_warnings_emac():
+    """Test ``get_ignored_warnings``."""
+    ignored_warnings = get_ignored_warnings('EMAC', 'load')
+    assert isinstance(ignored_warnings, list)
+    assert ignored_warnings
diff --git a/tests/unit/preprocessor/_time/test_time.py b/tests/unit/preprocessor/_time/test_time.py
index e492b34b1..ab9e08653 100644
--- a/tests/unit/preprocessor/_time/test_time.py
+++ b/tests/unit/preprocessor/_time/test_time.py
@@ -623,9 +623,10 @@ class TestClimatology(tests.Test):
                            [151, 181]])
         cube = self._create_cube(data, times, bounds)
 
-        result = climate_statistics(cube, operator='mean', period='season')
-        expected = np.array([1., 1., 1.], dtype=np.float32)
-        assert_array_equal(result.data, expected)
+        for period in ('season', 'seasonal'):
+            result = climate_statistics(cube, operator='mean', period=period)
+            expected = np.array([1., 1., 1.], dtype=np.float32)
+            assert_array_equal(result.data, expected)
 
     def test_custom_season_climatology(self):
         """Test for time avg of a realisitc time axis and 365 day calendar."""
@@ -635,12 +636,13 @@ class TestClimatology(tests.Test):
                            [151, 181], [181, 212], [212, 243]])
         cube = self._create_cube(data, times, bounds)
 
-        result = climate_statistics(cube,
-                                    operator='mean',
-                                    period='season',
-                                    seasons=('jfmamj', 'jasond'))
-        expected = np.array([1., 1.], dtype=np.float32)
-        assert_array_equal(result.data, expected)
+        for period in ('season', 'seasonal'):
+            result = climate_statistics(cube,
+                                        operator='mean',
+                                        period=period,
+                                        seasons=('jfmamj', 'jasond'))
+            expected = np.array([1., 1.], dtype=np.float32)
+            assert_array_equal(result.data, expected)
 
     def test_monthly(self):
         """Test for time avg of a realistic time axis and 365 day calendar."""
@@ -650,9 +652,10 @@ class TestClimatology(tests.Test):
                            [151, 181]])
         cube = self._create_cube(data, times, bounds)
 
-        result = climate_statistics(cube, operator='mean', period='mon')
-        expected = np.ones((6, ), dtype=np.float32)
-        assert_array_equal(result.data, expected)
+        for period in ('monthly', 'month', 'mon'):
+            result = climate_statistics(cube, operator='mean', period=period)
+            expected = np.ones((6, ), dtype=np.float32)
+            assert_array_equal(result.data, expected)
 
     def test_day(self):
         """Test for time avg of a realistic time axis and 365 day calendar."""
@@ -662,9 +665,25 @@ class TestClimatology(tests.Test):
                            [367, 368]])
         cube = self._create_cube(data, times, bounds)
 
-        result = climate_statistics(cube, operator='mean', period='day')
-        expected = np.array([1, 1, 1], dtype=np.float32)
-        assert_array_equal(result.data, expected)
+        for period in ('daily', 'day'):
+            result = climate_statistics(cube, operator='mean', period=period)
+            expected = np.array([1, 1, 1], dtype=np.float32)
+            assert_array_equal(result.data, expected)
+
+    def test_hour(self):
+        """Test for time avg of a realistic time axis and 365 day calendar."""
+        data = np.array([2., 2., 10., 4., 4., 6.], dtype=np.float32)
+        times = np.array([0.5, 1.5, 2.5, 24.5, 25.5, 48.5])
+        bounds = np.array([[0, 1], [1, 2], [2, 3], [3, 4], [4, 5], [5, 6]])
+        cube = self._create_cube(data, times, bounds)
+        cube.coord('time').units = 'hours since 2000-01-01 00:00:00'
+
+        for period in ('hourly', 'hour', 'hr'):
+            result = climate_statistics(cube, operator='mean', period=period)
+            expected = np.array([4., 3., 10.], dtype=np.float32)
+            assert_array_equal(result.data, expected)
+            expected_hours = [0, 1, 2]
+            assert_array_equal(result.coord('hour').points, expected_hours)
 
     def test_period_not_supported(self):
         """Test for time avg of a realistic time axis and 365 day calendar."""
@@ -1752,6 +1771,20 @@ def test_anomalies_custom_season():
     assert_array_equal(result.coord('time').points, cube.coord('time').points)
 
 
+@pytest.mark.parametrize('period', ['hourly', 'hour', 'hr'])
+def test_anomalies_hourly(period):
+    """Test ``anomalies`` with hourly data."""
+    cube = make_map_data(number_years=1)[:48, ...]
+    cube.coord('time').units = 'hours since 2000-01-01 00:00:00'
+    result = anomalies(cube, period)
+    expected = np.concatenate((
+        np.broadcast_to(np.array([[0, -12], [-12, 0]]), (24, 2, 2)),
+        np.broadcast_to(np.array([[0, 12], [12, 0]]), (24, 2, 2)),
+    ))
+    assert_array_equal(result.data, expected)
+    assert result.coord('time') == cube.coord('time')
+
+
 def get_0d_time():
     """Get 0D time coordinate."""
     time = iris.coords.AuxCoord(15.0,
diff --git a/tests/unit/recipe/test_recipe.py b/tests/unit/recipe/test_recipe.py
index b83035131..a8651daf0 100644
--- a/tests/unit/recipe/test_recipe.py
+++ b/tests/unit/recipe/test_recipe.py
@@ -489,34 +489,6 @@ def test_create_diagnostic_tasks(mock_diag_task, tasks_to_run, tasks_run):
         assert expected_call in mock_diag_task.mock_calls
 
 
-def test_update_warning_settings_nonaffected_project():
-    """Test ``_update_warning_settings``."""
-    settings = {'save': {'filename': 'out.nc'}, 'load': {'filename': 'in.nc'}}
-    _recipe._update_warning_settings(settings, 'CMIP5')
-    assert settings == {
-        'save': {'filename': 'out.nc'},
-        'load': {'filename': 'in.nc'},
-    }
-
-
-def test_update_warning_settings_step_not_present():
-    """Test ``_update_warning_settings``."""
-    settings = {'save': {'filename': 'out.nc'}}
-    _recipe._update_warning_settings(settings, 'EMAC')
-    assert settings == {'save': {'filename': 'out.nc'}}
-
-
-def test_update_warning_settings_step_present():
-    """Test ``_update_warning_settings``."""
-    settings = {'save': {'filename': 'out.nc'}, 'load': {'filename': 'in.nc'}}
-    _recipe._update_warning_settings(settings, 'EMAC')
-    assert len(settings) == 2
-    assert settings['save'] == {'filename': 'out.nc'}
-    assert len(settings['load']) == 2
-    assert settings['load']['filename'] == 'in.nc'
-    assert 'ignore_warnings' in settings['load']
-
-
 def test_update_regrid_time():
     """Test `_update_regrid_time."""
     dataset = Dataset(frequency='mon')
diff --git a/tests/unit/test_dataset.py b/tests/unit/test_dataset.py
index bd5c5546e..56baaa60d 100644
--- a/tests/unit/test_dataset.py
+++ b/tests/unit/test_dataset.py
@@ -1684,7 +1684,8 @@ def test_load(mocker, session):
 
     load_args = {
         'load': {
-            'callback': 'default'
+            'callback': 'default',
+            'ignore_warnings': None,
         },
         'fix_file': {
             'add_unique_suffix': True,