Repository: ESMValGroup/ESMValTool
Pull request #3636: Adding HadEX3 CMORizer
Author: login: malininae
Labels: name: observations
name: approved by technical reviewer
name: approved by scientific reviewer
name: IPCC
Created: 2024-05-29T15:09:30Z | Merged: 2025-05-14T07:43:46Z

This is a PR to add HadEX3 CMORizer needed for IPCC AR6 WGI Chapter 3, Cross Chapter Box 3.2. The issue is #3617, custom tables are in the issue [2441](https://github.com/ESMValGroup/ESMValCore/issues/2441) in ESMValCore.


Diff:
diff --git a/doc/sphinx/source/input.rst b/doc/sphinx/source/input.rst
index ce78766594..997fbbbbe6 100644
--- a/doc/sphinx/source/input.rst
+++ b/doc/sphinx/source/input.rst
@@ -373,6 +373,8 @@ A list of the datasets for which a CMORizers is available is provided in the fol
 +------------------------------+------------------------------------------------------------------------------------------------------+------+-----------------+
 | HadCRUT5                     | tas, tasa (Amon)                                                                                     |   2  | Python          |
 +------------------------------+------------------------------------------------------------------------------------------------------+------+-----------------+
+| HadEX3                       | txx, tnn, rx1day, rx5day (Amon, yr)                                                                  |   2  | Python          |
++------------------------------+------------------------------------------------------------------------------------------------------+------+-----------------+
 | HadISST                      | sic (OImon), tos (Omon), ts (Amon)                                                                   |   2  | NCL             |
 +------------------------------+------------------------------------------------------------------------------------------------------+------+-----------------+
 | HALOE                        | tro3, hus (Amon)                                                                                     |   2  | NCL             |
diff --git a/esmvaltool/cmorizers/data/cmor_config/HadEX3.yml b/esmvaltool/cmorizers/data/cmor_config/HadEX3.yml
new file mode 100644
index 0000000000..86166d6128
--- /dev/null
+++ b/esmvaltool/cmorizers/data/cmor_config/HadEX3.yml
@@ -0,0 +1,65 @@
+---
+filename: 'HadEX3_{raw}_{raw_frequency}.nc'
+
+# Common global attributes for CMORizer output
+attributes:
+  dataset_id: HadEX3
+  version: v3.0.4
+  tier: 2
+  modeling_realm: ground
+  project_id: OBS6
+  source: https://www.metoffice.gov.uk/hadobs/hadex3/download.html
+  reference: 'hadex3'
+  comment: 'ETCCDI (Expert Team on Climate Change Detection and Indices)'
+
+# Variables to cmorize
+variables:
+  txx_yr:
+    short_name: txx
+    raw: TXx
+    raw_units: degrees_C
+    raw_frequency: ANN
+    mip: yr
+  txx_mon:
+    short_name: txx
+    raw: TXx
+    raw_units: degrees_C
+    raw_frequency: MON
+    mip: Amon
+  tnn_yr:
+    short_name: tnn
+    raw: TNn
+    raw_units: degrees_C
+    raw_frequency: ANN
+    mip: yr
+  tnn_mon:
+    short_name: tnn
+    raw: TNn
+    raw_units: degrees_C
+    raw_frequency: MON
+    frequency: mon
+    mip: Amon
+  rx1day_yr:
+    short_name: rx1day
+    raw: Rx1day
+    raw_units: mm
+    raw_frequency: ANN
+    mip: yr
+  rx1day_mon:
+    short_name: rx1day
+    raw: Rx1day
+    raw_units: mm
+    raw_frequency: MON
+    mip: Amon
+  rx5day_yr:
+    short_name: rx5day
+    raw: Rx5day
+    raw_units: mm
+    raw_frequency: ANN
+    mip: yr
+  rx5day_mon:
+    short_name: rx5day
+    raw: Rx5day
+    raw_units: mm
+    raw_frequency: MON
+    mip: Amon
diff --git a/esmvaltool/cmorizers/data/datasets.yml b/esmvaltool/cmorizers/data/datasets.yml
index 86e0bfbc6b..6d0198822e 100644
--- a/esmvaltool/cmorizers/data/datasets.yml
+++ b/esmvaltool/cmorizers/data/datasets.yml
@@ -777,6 +777,17 @@ datasets:
         climatology
             [Source]/absolute_v5.nc
 
+  HadEX3:
+    tier: 2
+    source: https://www.metoffice.gov.uk/hadobs/hadex3/download_etccdi.html
+    last_access: 2025-02-05
+    info: |
+      Use automatic download, it will download and unzip all required files:
+      "HadEX3_TXx_MON.nc.gz", "HadEX3_TXx_ANN.nc.gz",
+      "HadEX3_TNn_MON.nc.gz", "HadEX3_TNn_ANN.nc.gz",
+      "HadEX3_Rx1day_MON.nc.gz", "HadEX3_Rx1day_ANN.nc.gz",
+      "HadEX3_Rx5day_MON.nc.gz" and "HadEX3_Rx5day_ANN.nc.gz".
+
   HadISST:
     tier: 2
     source: http://www.metoffice.gov.uk/hadobs/hadisst/data/download.html
diff --git a/esmvaltool/cmorizers/data/downloaders/datasets/hadex3.py b/esmvaltool/cmorizers/data/downloaders/datasets/hadex3.py
new file mode 100644
index 0000000000..f7d3c68055
--- /dev/null
+++ b/esmvaltool/cmorizers/data/downloaders/datasets/hadex3.py
@@ -0,0 +1,57 @@
+"""Script to download HadEX3 from its webpage."""
+
+import logging
+import os
+
+from esmvaltool.cmorizers.data.downloaders.wget import WGetDownloader
+from esmvaltool.cmorizers.data.utilities import unpack_files_in_folder
+
+logger = logging.getLogger(__name__)
+
+
+def download_dataset(
+    config, dataset, dataset_info, start_date, end_date, overwrite
+):
+    """Download dataset.
+
+    Parameters
+    ----------
+    config : dict
+        ESMValTool's user configuration
+    dataset : str
+        Name of the dataset
+    dataset_info : dict
+         Dataset information from the datasets.yml file
+    start_date : datetime
+        Start of the interval to download
+    end_date : datetime
+        End of the interval to download
+    overwrite : bool
+        Overwrite already downloaded files
+    """
+    downloader = WGetDownloader(
+        config=config,
+        dataset=dataset,
+        dataset_info=dataset_info,
+        overwrite=overwrite,
+    )
+
+    os.makedirs(downloader.local_folder, exist_ok=True)
+
+    web_source = "https://www.metoffice.gov.uk/hadobs/hadex3/data/"
+
+    file_list = [
+        "HadEX3_TXx_MON.nc.gz",
+        "HadEX3_TXx_ANN.nc.gz",
+        "HadEX3_TNn_MON.nc.gz",
+        "HadEX3_TNn_ANN.nc.gz",
+        "HadEX3_Rx1day_MON.nc.gz",
+        "HadEX3_Rx1day_ANN.nc.gz",
+        "HadEX3_Rx5day_MON.nc.gz",
+        "HadEX3_Rx5day_ANN.nc.gz",
+    ]
+
+    for file in file_list:
+        downloader.download_file(f"{web_source}{file}", wget_options=[])
+
+    unpack_files_in_folder(downloader.local_folder)
diff --git a/esmvaltool/cmorizers/data/formatters/datasets/hadex3.py b/esmvaltool/cmorizers/data/formatters/datasets/hadex3.py
new file mode 100644
index 0000000000..50df26f961
--- /dev/null
+++ b/esmvaltool/cmorizers/data/formatters/datasets/hadex3.py
@@ -0,0 +1,106 @@
+"""ESMValTool CMORizer for HadEX3 data.
+
+Tier
+    Tier 2: other freely-available dataset.
+
+Source
+    https://www.metoffice.gov.uk/hadobs/hadex3/download.html
+
+Last access
+    2025-02-05
+
+Download and processing instructions
+    Download the following files:
+        Annual.nc.gz and Monthly.nc.gz for TXx, TNn, Rx1day and Rx5day
+    No registration is required for downloading the data.
+
+When using the dataset in a paper, the following are citations to use:
+
+Dunn, R. J. H., et al. (2020), Development of an updated global land
+in-situ-based dataset of temperature and precipitation extremes: HadEX3 JGR-A
+
+Dunn, R. J. H., Donat, M. G., Alexander, L. V., al. (2014), Investigating
+uncertainties in global gridded datasets of climate extremes,
+Climate of the Past, 10, 2171-2199
+
+Donat, M. G., Alexander, L. V., .... Dunn, R. J. H., et al. (2013), Updated
+analyses of temperature and precipitation extreme indices since the beginning
+of the twentieth century: The HadEX2 dataset, J. Geophys. Res. Atmos., 118,
+2098-2118
+"""
+
+import logging
+import os
+
+import iris
+from cf_units import Unit
+from iris import NameConstraint
+
+from esmvaltool.cmorizers.data import utilities as utils
+
+logger = logging.getLogger(__name__)
+
+
+def _fix_time_coord(cube):
+    """Convert the time to the gregorian calendar."""
+    time_coord = cube.coord("time")
+    time_coord.guess_bounds()
+    new_unit = Unit("days since 1850-01-01 00:00:00", calendar="gregorian")
+    new_time_points = time_coord.units.num2date(time_coord.points)
+    time_points = new_unit.date2num(new_time_points)
+    new_time_bounds = time_coord.units.num2date(time_coord.bounds)
+    time_bounds = new_unit.date2num(new_time_bounds)
+
+    cube.coord("time").points = time_points
+    cube.coord("time").bounds = time_bounds
+    cube.coord("time").units = new_unit
+
+
+def _extract_variable(var, var_info, cmor_info, attrs, filepath, out_dir):
+    """Extract variable."""
+    raw_var = var_info.get("raw", var)
+    var = cmor_info.short_name
+    cube = iris.load_cube(filepath, NameConstraint(var_name=raw_var))
+    # Fix units
+    cube.units = var_info.get("raw_units", var)
+    cube.convert_units(cmor_info.units)
+
+    _fix_time_coord(cube)
+    utils.fix_var_metadata(cube, cmor_info)
+    utils.convert_timeunits(cube, 1950)
+    utils.fix_coords(cube)
+    utils.set_global_atts(cube, attrs)
+    utils.save_variable(
+        cube, var, out_dir, attrs, unlimited_dimensions=["time"]
+    )
+
+
+def cmorization(in_dir, out_dir, cfg, cfg_user, start_date, end_date):
+    """Cmorization function call."""
+    glob_attrs = cfg["attributes"]
+    cmor_table = cfg["cmor_table"]
+    filename = cfg["filename"]
+
+    # Run the cmorization
+    for var, var_info in cfg["variables"].items():
+        var_name = var_info["short_name"]
+        var_mip = var_info["mip"]
+        filepath = os.path.join(
+            in_dir,
+            filename.format(
+                raw=var_info["raw"], raw_frequency=var_info["raw_frequency"]
+            ),
+        )
+        if os.path.isfile(filepath):
+            logger.info("Found input file %s", filepath)
+        else:
+            raise ValueError(f"Couldn't find {filepath}")
+        logger.info("CMORizing variable %s in %s table", var_name, var_mip)
+        glob_attrs["mip"] = var_info["mip"]
+        cmor_info = cmor_table.get_variable(var_info["mip"], var_name)
+        _extract_variable(
+            var, var_info, cmor_info, glob_attrs, filepath, out_dir
+        )
+        logger.info(
+            "CMORization of %s in %s table was successful", var_name, var_mip
+        )
diff --git a/esmvaltool/recipes/examples/recipe_check_obs.yml b/esmvaltool/recipes/examples/recipe_check_obs.yml
index cd75f8dfb5..7beb1581e3 100644
--- a/esmvaltool/recipes/examples/recipe_check_obs.yml
+++ b/esmvaltool/recipes/examples/recipe_check_obs.yml
@@ -539,6 +539,21 @@ diagnostics:
     scripts: null
 
 
+  HadEX3:
+    description: HadEX3 check
+    variables:
+      txx:
+      tnn:
+      rx1day:
+      rx5day:
+    additional_datasets:
+      - {dataset: HadEX3, project: OBS6, mip: yr, tier: 2,
+         type: ground, version: v3.0.4, start_year: 1901, end_year: 2018}
+      - {dataset: HadEX3, project: OBS6, mip: Amon, tier: 2,
+         type: ground, version: v3.0.4, start_year: 1901, end_year: 2018}
+    scripts: null
+
+
   HadISST:
     description: HadISST check
     variables:
diff --git a/esmvaltool/references/hadex3.bibtex b/esmvaltool/references/hadex3.bibtex
new file mode 100644
index 0000000000..4262b941d4
--- /dev/null
+++ b/esmvaltool/references/hadex3.bibtex
@@ -0,0 +1,12 @@
+@article{hadex3,
+  title = {Development of an updated global land in situ-based data set of temperature and precipitation extremes: HadEX3},
+  author = {Dunn, Robert JH and Alexander, Lisa V and Donat, Markus G and Zhang, Xuebin and Bador, Margot and Herold, Nicholas and Lippmann, Tanya and Allan, Rob and Aguilar, Enric and Barry, Abdoul Aziz and others},
+  doi = {10.1029/2019jd032263},
+  url = {https://doi.org/10.1029/2019JD032263},
+  journal = {Journal of Geophysical Research: Atmospheres},
+  volume = {125},
+  number = {16},
+  pages = {e2019JD032263},
+  year = {2020},
+  publisher = {Wiley Online Library}
+}


Reviews:
user: login: LisaBock
body: Hi @malininae ! Sorry for the delayed review! And thanks a lot for going ahead with the IPCC stuff.

I run the downloader and cmorizer and they work perfect.
Two comments:
- Please add the dataset to the [table in the documentation](https://github.com/ESMValGroup/ESMValTool/blob/47b806da942572c7c989d69e9829bc934c4f9991/doc/sphinx/source/input.rst).
- Regarding the failing tests: I think you still need to run `pre-commit run --all` for automated fixing (the remaining complains might be okay). 
state: CHANGES_REQUESTED
author_association: MEMBER
submitted_at: 2025-05-12T08:03:59Z
user: login: schlunma
body: Thanks Liza, looks great! Just two tiny comments.

I tested the downloader, CMORizer, and the ran the recipe, everything works fine 🎉 
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-13T12:24:50Z
user: login: malininae
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-13T23:16:45Z
user: login: LisaBock
body: Thanks @malininae !
state: APPROVED
author_association: MEMBER
submitted_at: 2025-05-14T06:47:56Z

Review comments:
diff_hunk: @@ -0,0 +1,115 @@
+"""ESMValTool CMORizer for HadEX3 data.
+
+Tier
+    Tier 2: other freely-available dataset.
+
+Source
+    https://www.metoffice.gov.uk/hadobs/hadex3/download.html
+
+Last access
+    2025-02-05
+
+Download and processing instructions
+    Download the following files:
+        Annual.nc.gz and Monthly.nc.gz for TXx, TNn, Rx1day and Rx5day
+    No registration is required for downloading the data.
+
+When using the dataset in a paper, the following are citations to use:
+
+Dunn, R. J. H., et al. (2020), Development of an updated global land
+in-situ-based dataset of temperature and precipitation extremes: HadEX3 JGR-A
+
+Dunn, R. J. H., Donat, M. G., Alexander, L. V., al. (2014), Investigating
+uncertainties in global gridded datasets of climate extremes,
+Climate of the Past, 10, 2171-2199
+
+Donat, M. G., Alexander, L. V., .... Dunn, R. J. H., et al. (2013), Updated
+analyses of temperature and precipitation extreme indices since the beginning
+of the twentieth century: The HadEX2 dataset, J. Geophys. Res. Atmos., 118,
+2098-2118
+"""
+
+import logging
+import os
+
+import cftime
+import iris
+from cf_units import Unit
+from iris import NameConstraint
+
+from esmvaltool.cmorizers.data import utilities as utils
+
+logger = logging.getLogger(__name__)
+
+
+def _fix_time_coord(cube):
+    """Convert the time to the gregorian calendar."""
+    time_coord = cube.coord("time")
+    time_coord.guess_bounds()
+    new_unit = Unit("days since 1850-01-01 00:00:00", calendar="gregorian")
+    new_time_points = cftime.num2pydate(
+        time_coord.points, time_coord.units.origin, time_coord.units.calendar
+    )
+    time_points = cftime.date2num(
+        new_time_points, new_unit.origin, calendar=new_unit.calendar
+    )
+    new_time_bounds = cftime.num2pydate(
+        time_coord.bounds, time_coord.units.origin, time_coord.units.calendar
+    )
+    time_bounds = cftime.date2num(
+        new_time_bounds, new_unit.origin, calendar=new_unit.calendar
+    )
path: esmvaltool/cmorizers/data/formatters/datasets/hadex3.py
user: login: schlunma
body: ```suggestion
    new_time_points = time_coord.units.num2date(time_coord.points)
    time_points = new_unit.date2num(new_time_points)
    new_time_bounds = time_coord.units.num2date(time_coord.bounds)
    time_bounds = new_unit.date2num(new_time_bounds)
```

should do the same.
created_at: 2025-05-13T12:19:50Z
author_association: CONTRIBUTOR
diff_hunk: @@ -777,6 +777,16 @@ datasets:
         climatology
             [Source]/absolute_v5.nc
 
+  HadEX3:
+    tier: 2
+    source: https://www.metoffice.gov.uk/hadobs/hadex3/download_etccdi.html
+    last_access: 2025-02-05
+    info: |
+      Download and unzip "HadEX3_TXx_MON.nc.gz", "HadEX3_TXx_ANN.nc.gz",
+      "HadEX3_TNn_MON.nc.gz", "HadEX3_TNn_ANN.nc.gz",
+      "HadEX3_Rx1day_MON.nc.gz", "HadEX3_Rx1day_ANN.nc.gz",
+      "HadEX3_Rx5day_MON.nc.gz" and "HadEX3_Rx5day_ANN.nc.gz".
path: esmvaltool/cmorizers/data/datasets.yml
user: login: schlunma
body: Since there is a working downloader script I would just refer to that here.
created_at: 2025-05-13T12:22:31Z
author_association: CONTRIBUTOR
diff_hunk: @@ -777,6 +777,16 @@ datasets:
         climatology
             [Source]/absolute_v5.nc
 
+  HadEX3:
+    tier: 2
+    source: https://www.metoffice.gov.uk/hadobs/hadex3/download_etccdi.html
+    last_access: 2025-02-05
+    info: |
+      Download and unzip "HadEX3_TXx_MON.nc.gz", "HadEX3_TXx_ANN.nc.gz",
+      "HadEX3_TNn_MON.nc.gz", "HadEX3_TNn_ANN.nc.gz",
+      "HadEX3_Rx1day_MON.nc.gz", "HadEX3_Rx1day_ANN.nc.gz",
+      "HadEX3_Rx5day_MON.nc.gz" and "HadEX3_Rx5day_ANN.nc.gz".
path: esmvaltool/cmorizers/data/datasets.yml
user: login: malininae
body: Good idea, done. 
created_at: 2025-05-13T23:16:45Z
author_association: CONTRIBUTOR

PR comments:
user: login: malininae
created_at: 2025-02-05T21:13:27Z
body: Hi all, I'm marking this PR ready for review, but I'm afraid, I might need some help to try to figure out all the codacy fails. 

First of all, the test is failing because the `txx`, `tnn`,  `rx1day` and `rx5day` aren't available in the core yet, however, [this PR](https://github.com/ESMValGroup/ESMValCore/pull/2442) is ready for review and hopefully will be merged in for the 2.12 release. 

Second, codacy is complaining about few things that I have no control over, for example, it's complaining that I have 6 input variables in `cmorization` function, however, that requirement comes from the cmorization class, as far as I understand. I was looking at the other examples like HadCRUT5 and it seems to be similar. Anyway, any advice will be highly appreciated! 
author_association: CONTRIBUTOR
user: login: sloosvel
created_at: 2025-02-27T17:02:12Z
body: Hi, the code freeze for version 2.12 will be on Monday 3rd of March. Is this ready to be included? If this is not ready, let me know and I will move it to the next milestone.
author_association: CONTRIBUTOR
user: login: malininae
created_at: 2025-02-27T17:55:10Z
body: > Hi, the code freeze for version 2.12 will be on Monday 3rd of March. Is this ready to be included? If this is not ready, let me know and I will move it to the next milestone.

Hi @sloosvel, I guess since no one reviewed it, it won't make it into 2.12. 

author_association: CONTRIBUTOR
user: login: malininae
created_at: 2025-05-13T08:35:19Z
body: > Hi @malininae ! Sorry for the delayed review! And thanks a lot for going ahead with the IPCC stuff.
> 
> I run the downloader and cmorizer and they work perfect. Two comments:
> 
> * Please add the dataset to the [table in the documentation](https://github.com/ESMValGroup/ESMValTool/blob/47b806da942572c7c989d69e9829bc934c4f9991/doc/sphinx/source/input.rst).
> * Regarding the failing tests: I think you still need to run `pre-commit run --all` for automated fixing (the remaining complains might be okay).

Thanks so much @LisaBock, done! As I was saying in the workshop, I fixed as much as I could for the codacy, but the 6/5 restriction in most of them are hard to go around, same for the `start_year` in downloading function. Let me know if other questions arise.  
author_association: CONTRIBUTOR
user: login: schlunma
created_at: 2025-05-13T12:25:33Z
body: And maybe one comment for future PRs: could you please use our PR template (the one with the check boxes)? This makes reviewing easier.
author_association: CONTRIBUTOR
user: login: malininae
created_at: 2025-05-13T18:14:16Z
body: > And maybe one comment for future PRs: could you please use our PR template (the one with the check boxes)? This makes reviewing easier.

Thanks so much @schlunma, sorry, will do, have done that for the others already :-)
author_association: CONTRIBUTOR
user: login: LisaBock
created_at: 2025-05-14T07:11:56Z
body: Hi @axel-lauer ! Could you please add the new OBS data to the DKRZ data pool and then merge the PR.
author_association: MEMBER
user: login: axel-lauer
created_at: 2025-05-14T07:43:20Z
body: Just copied the data to the obs pool at DKRZ.
author_association: CONTRIBUTOR