Repository: ESMValGroup/ESMValCore

Issue:
Dataset problems: ACCESS-ESM1-5 air pressure coordinates rounding problem, CanESM5 & CanESM5-CanOE wrong units for co2, gpp incorrectly masked
**Describe the dataset issue**
**ACCESS-ESM1-5**:
Air pressure coordinates for zg and hus in esm-hist are rounded differently, leading to an error when trying to derive xco2 which uses both of these, as well as co2, which has the correct levels in the same version (20191128).
`ValueError: This operation cannot be performed as there are differing coordinates (air_pressure) remaining which cannot be ignored.
`
Also given by a warning for both variables:
```
WARNING There were warnings in variable hus/zg:
plev: does not contain 100000.0 Pa
 plev: does not contain 92500.0 Pa
 plev: does not contain 85000.0 Pa
 plev: does not contain 70000.0 Pa
 plev: does not contain 60000.0 Pa
 plev: does not contain 50000.0 Pa
 plev: does not contain 40000.0 Pa
 plev: does not contain 30000.0 Pa
 plev: does not contain 25000.0 Pa
 plev: does not contain 20000.0 Pa
 plev: does not contain 15000.0 Pa
 plev: does not contain 10000.0 Pa
 plev: does not contain 7000.0 Pa
 plev: does not contain 5000.0 Pa
 plev: does not contain 3000.0 Pa
 plev: does not contain 2000.0 Pa
 plev: does not contain 1000.0 Pa
 plev: does not contain 500.0 Pa
 plev: does not contain 100.0 Pa
```
Actual values in files:
```
 plev = 100000.00000001, 92500.00000001, 85000.00000001, 70000.00000001, 
    60000.00000001, 50000.00000001, 40000.00000001, 30000.00000001, 
    25000.00000001, 20000.00000001, 15000.00000001, 10000.00000001, 
    7000.00000001, 5000.00000001, 3000.00000001, 2000.00000001, 
    1000.00000001, 500.00000001, 100.00000001 ;
```

Files on mistral: 
e.g.
/work/ik1017/CMIP6/data/CMIP6/CMIP/CSIRO/ACCESS-ESM1-5/esm-hist/r1i1p1f1/Amon/zg/gn/v20191128/zg_Amon_ACCESS-ESM1-5_esm-hist_r1i1p1f1_gn_185001-194912.nc
/work/ik1017/CMIP6/data/CMIP6/CMIP/CSIRO/ACCESS-ESM1-5/esm-hist/r1i1p1f1/Amon/hus/gn/v20191128/hus_Amon_ACCESS-ESM1-5_esm-hist_r1i1p1f1_gn_185001-194912.nc

- Full dataset description (fill out as many as you know, please):
  - project: CMIP6 
  - dataset: ACCESS-ESM1-5
  - experiment: esm-hist
  - realization: r1i1p1f1
  - mip: Amon
  - grid: gn
  - data version: v20191128
  - variable used: zg, hus, co2

Note: ps also has the weirdly rounded plev coordinates, but no warning form the ESMValTool and for me it works.

**Can-ESM5/CanESM5-CanOE**
CO2: wrong units, in CMIP6 the units [1] is expected, while they're giving it in ppmv, so data needs to be multiplied by a factor of 1e-6 - **Caveat:** Even though it's the same version as available data, co2 is not on mistral, but available on the ESGF download page and no known errata on ES-DOC.
Gpp: It's a carbon flux on LAND, but has ocean values set to 0 instead of masked, resulting in wrong averages


Files on mistral: 
/work/ik1017/CMIP6/data/CMIP6/CMIP/CCCma/CanESM5/esm-hist/r1i1p1f1/Lmon/gpp/gn/v20190429/gpp_Lmon_CanESM5_esm-hist_r1i1p1f1_gn_185001-201412.nc
/work/ik1017/CMIP6/data/CMIP6/CMIP/CCCma/CanESM5-CanOE/historical/r1i1p2f1/Lmon/gpp/gn/v20190429/gpp_Lmon_CanESM5-CanOE_historical_r1i1p2f1_gn_185001-201412.nc

- Full dataset description (fill out as many as you know, please):
  - project: CMIP6 
  - dataset: CanESM5 / CanOE
  - experiment: esm-hist/historical
  - mip: Amon / Lmon for gpp
  - grid: gn
  - data version: v20190429
  - variable used: co2, gpp

**Assign the right people**
If you are already a member of the ESMValTool GitHub project, please assign Valeriu Predoi (valeriupredoi) and
Klaus Zimmermann (zklaus) to the issue. They will then check the issue raised and propagate it further to the
data model developers.

Cheers :beer:
Pull request:  #947 

Discussion:
sloosvel: Since the fixes for the CanESM models are the same, maybe they could be imported between models? It's how it's being done in [cesm2-waccm fixes](https://github.com/ESMValGroup/ESMValCore/blob/master/esmvalcore/cmor/_fixes/cmip6/cesm2_waccm.py) or in [mpi_esm1_2_xr.py](https://github.com/ESMValGroup/ESMValCore/blob/master/esmvalcore/cmor/_fixes/cmip6/mpi_esm1_2_xr.py). Just to avoid duplicating them.
bettina-gier: Good call, done!
sloosvel: There seem to be some codacy errors in the docstrings, but other than that the fixes work well! I couldn't find data for co2 in CanESM5-CanOE but all the others do what they claim to do. 
bettina-gier: I actually don't understand the remaining codacy errors, as there are blank lines and the arguments are described in the docstring, analogous to already existing fixes. (Or I'm just blind)
sloosvel: I don't see it either. The docstrings look all the same to me, yet some fail and some do not.
bettina-gier: Can we merge this?
Commit failing is not due to this code, and the codacy errors seem nonsensical.

Patch:
diff --git a/esmvalcore/cmor/_fixes/cmip6/access_esm1_5.py b/esmvalcore/cmor/_fixes/cmip6/access_esm1_5.py
index 287f1b16f..da884b5e3 100644
--- a/esmvalcore/cmor/_fixes/cmip6/access_esm1_5.py
+++ b/esmvalcore/cmor/_fixes/cmip6/access_esm1_5.py
@@ -1,7 +1,9 @@
 """Fixes for ACCESS-ESM1-5."""
 import iris
+import numpy as np
 
 from ..common import ClFixHybridHeightCoord
+from ..fix import Fix
 
 
 class Cl(ClFixHybridHeightCoord):
@@ -80,3 +82,51 @@ Cli = Cl
 
 
 Clw = Cl
+
+
+class Hus(Fix):
+    """Fixes for hus."""
+
+    def fix_metadata(self, cubes):
+        """Correctly round air pressure coordinate.
+
+        Parameters
+        ----------
+        cubes : iris.cube.CubeList
+            Input cubes.
+
+        Returns
+        -------
+        iris.cube.Cube
+
+        """
+        cube = self.get_cube_from_list(cubes)
+        cube.coord('air_pressure').points = \
+            np.round(cube.coord('air_pressure').points, 0)
+        cube.coord('air_pressure').bounds = \
+            np.round(cube.coord('air_pressure').bounds, 0)
+        return cubes
+
+
+class Zg(Fix):
+    """Fixes for zg."""
+
+    def fix_metadata(self, cubes):
+        """Correctly round air pressure coordinate.
+
+        Parameters
+        ----------
+        cubes : iris.cube.CubeList
+            Input cubes.
+
+        Returns
+        -------
+        iris.cube.Cube
+
+        """
+        cube = self.get_cube_from_list(cubes)
+        cube.coord('air_pressure').points = \
+            np.round(cube.coord('air_pressure').points, 0)
+        cube.coord('air_pressure').bounds = \
+            np.round(cube.coord('air_pressure').bounds, 0)
+        return cubes
diff --git a/esmvalcore/cmor/_fixes/cmip6/canesm5.py b/esmvalcore/cmor/_fixes/cmip6/canesm5.py
new file mode 100644
index 000000000..d4cdbc95e
--- /dev/null
+++ b/esmvalcore/cmor/_fixes/cmip6/canesm5.py
@@ -0,0 +1,46 @@
+"""Fixes for CanESM5 model."""
+import dask.array as da
+
+from ..fix import Fix
+
+
+class Co2(Fix):
+    """Fixes for co2."""
+
+    def fix_data(self, cube):
+        """Convert units from ppmv to 1.
+
+        Parameters
+        ----------
+        cube : iris.cube.Cube
+            Input cube.
+
+        Returns
+        -------
+        iris.cube.Cube
+
+        """
+        metadata = cube.metadata
+        cube *= 1.e-6
+        cube.metadata = metadata
+        return cube
+
+
+class Gpp(Fix):
+    """Fixes for gpp, ocean values set to 0 instead of masked."""
+
+    def fix_data(self, cube):
+        """Fix masked values.
+
+        Parameters
+        ----------
+        cube: iris.cube.Cube
+            Input cube.
+
+        Returns
+        -------
+        iris.cube.Cube
+
+        """
+        cube.data = da.ma.masked_equal(cube.core_data(), 0.0)
+        return cube
diff --git a/esmvalcore/cmor/_fixes/cmip6/canesm5_canoe.py b/esmvalcore/cmor/_fixes/cmip6/canesm5_canoe.py
new file mode 100644
index 000000000..c879c93da
--- /dev/null
+++ b/esmvalcore/cmor/_fixes/cmip6/canesm5_canoe.py
@@ -0,0 +1,12 @@
+"""Fixes for CanESM5-CanOE model."""
+
+from .canesm5 import Co2 as BaseCO2
+from .canesm5 import Gpp as BaseGpp
+
+
+class Co2(BaseCO2):
+    """Fixes for co2."""
+
+
+class Gpp(BaseGpp):
+    """Fixes for gpp."""


Test patch:
diff --git a/tests/integration/cmor/_fixes/cmip6/test_access_esm1_5.py b/tests/integration/cmor/_fixes/cmip6/test_access_esm1_5.py
index 225087520..d126d3a74 100644
--- a/tests/integration/cmor/_fixes/cmip6/test_access_esm1_5.py
+++ b/tests/integration/cmor/_fixes/cmip6/test_access_esm1_5.py
@@ -5,9 +5,10 @@ import iris
 import numpy as np
 import pytest
 
-from esmvalcore.cmor._fixes.cmip6.access_esm1_5 import Cl, Cli, Clw
+from esmvalcore.cmor._fixes.cmip6.access_esm1_5 import Cl, Cli, Clw, Hus, Zg
 from esmvalcore.cmor._fixes.common import ClFixHybridHeightCoord
 from esmvalcore.cmor.fix import Fix
+from esmvalcore.cmor.table import get_var_info
 
 B_POINTS = [
     0.99771648645401, 0.990881502628326, 0.979542553424835,
@@ -117,3 +118,75 @@ def test_get_clw_fix():
 def test_clw_fix():
     """Test fix for ``clw``."""
     assert Clw is Cl
+
+
+@pytest.fixture
+def cubes_with_wrong_air_pressure():
+    """Cubes with wrong ``air_pressure`` coordinate."""
+    air_pressure_coord = iris.coords.DimCoord(
+        [1000.09, 600.6, 200.0],
+        bounds=[[1200.00001, 800], [800, 400.8], [400.8, 1.9]],
+        var_name='plev',
+        standard_name='air_pressure',
+        units='pa',
+    )
+    hus_cube = iris.cube.Cube(
+        [0.0, 1.0, 2.0],
+        var_name='hus',
+        dim_coords_and_dims=[(air_pressure_coord, 0)],
+    )
+    zg_cube = hus_cube.copy()
+    zg_cube.var_name = 'zg'
+    return iris.cube.CubeList([hus_cube, zg_cube])
+
+
+def test_get_hus_fix():
+    """Test getting of fix."""
+    fix = Fix.get_fixes('CMIP6', 'ACCESS-ESM1-5', 'Amon', 'hus')
+    assert fix == [Hus(None)]
+
+
+def test_hus_fix_metadata(cubes_with_wrong_air_pressure):
+    """Test ``fix_metadata`` for ``hus``."""
+    vardef = get_var_info('CMIP6', 'Amon', 'hus')
+    fix = Hus(vardef)
+    out_cubes = fix.fix_metadata(cubes_with_wrong_air_pressure)
+    assert len(out_cubes) == 2
+    hus_cube = out_cubes.extract_cube('hus')
+    zg_cube = out_cubes.extract_cube('zg')
+    assert hus_cube.var_name == 'hus'
+    assert zg_cube.var_name == 'zg'
+    np.testing.assert_allclose(hus_cube.coord('air_pressure').points,
+                               [1000.0, 601.0, 200.0])
+    np.testing.assert_allclose(hus_cube.coord('air_pressure').bounds,
+                               [[1200.0, 800.0], [800.0, 401.0], [401.0, 2.0]])
+    np.testing.assert_allclose(zg_cube.coord('air_pressure').points,
+                               [1000.09, 600.6, 200.0])
+    np.testing.assert_allclose(zg_cube.coord('air_pressure').bounds,
+                               [[1200.00001, 800], [800, 400.8], [400.8, 1.9]])
+
+
+def test_get_zg_fix():
+    """Test getting of fix."""
+    fix = Fix.get_fixes('CMIP6', 'ACCESS-ESM1-5', 'Amon', 'zg')
+    assert fix == [Zg(None)]
+
+
+def test_zg_fix_metadata(cubes_with_wrong_air_pressure):
+    """Test ``fix_metadata`` for ``zg``."""
+    vardef = get_var_info('CMIP6', 'Amon', 'zg')
+    fix = Zg(vardef)
+    out_cubes = fix.fix_metadata(cubes_with_wrong_air_pressure)
+    assert len(out_cubes) == 2
+    hus_cube = out_cubes.extract_cube('hus')
+    zg_cube = out_cubes.extract_cube('zg')
+    assert hus_cube.var_name == 'hus'
+    assert zg_cube.var_name == 'zg'
+    np.testing.assert_allclose(hus_cube.coord('air_pressure').points,
+                               [1000.09, 600.6, 200.0])
+    np.testing.assert_allclose(hus_cube.coord('air_pressure').bounds,
+                               [[1200.00001, 800], [800, 400.8], [400.8, 1.9]])
+    np.testing.assert_allclose(zg_cube.coord('air_pressure').points,
+                               [1000.0, 601.0, 200.0])
+    np.testing.assert_allclose(zg_cube.coord('air_pressure').bounds,
+                               [[1200.0, 800.0], [800.0, 401.0], [401.0, 2.0]])
diff --git a/tests/integration/cmor/_fixes/cmip6/test_canesm5.py b/tests/integration/cmor/_fixes/cmip6/test_canesm5.py
new file mode 100644
index 000000000..9b9638e5f
--- /dev/null
+++ b/tests/integration/cmor/_fixes/cmip6/test_canesm5.py
@@ -0,0 +1,60 @@
+"""Tests for the fixes of CanESM5."""
+import iris
+import numpy as np
+import pytest
+
+from esmvalcore.cmor._fixes.cmip6.canesm5 import Co2, Gpp
+from esmvalcore.cmor.fix import Fix
+
+
+def test_get_co2_fix():
+    """Test getting of fix."""
+    fix = Fix.get_fixes('CMIP6', 'CanESM5', 'Amon', 'co2')
+    assert fix == [Co2(None)]
+
+
+@pytest.fixture
+def co2_cube():
+    """``co2`` cube."""
+    cube = iris.cube.Cube(
+        [1.0],
+        var_name='co2',
+        standard_name='mole_fraction_of_carbon_dioxide_in_air',
+        units='mol mol-1',
+    )
+    return cube
+
+
+def test_co2_fix_data(co2_cube):
+    """Test ``fix_data`` for ``co2``."""
+    fix = Co2(None)
+    out_cube = fix.fix_data(co2_cube)
+    np.testing.assert_allclose(out_cube.data, [1.e-6])
+
+
+@pytest.fixture
+def gpp_cube():
+    """``gpp`` cube."""
+    cube = iris.cube.Cube(
+        [0, 1],
+        var_name='gpp',
+        standard_name='gross_primary_productivity_of_biomass_expressed_as_'
+        'carbon',
+        units='kg m-2 s-1',
+    )
+    return cube
+
+
+def test_get_gpp_fix():
+    """Test getting of fix."""
+    fix = Fix.get_fixes('CMIP6', 'CanESM5', 'Lmon', 'gpp')
+    assert fix == [Gpp(None)]
+
+
+def test_gpp_fix_data(gpp_cube):
+    """Test ``fix_data`` for ``gpp``."""
+    fix = Gpp(None)
+    out_cube = fix.fix_data(gpp_cube)
+    np.testing.assert_allclose(out_cube.data,
+                               np.ma.masked_invalid([np.nan, 1]))
+    assert np.array_equal(out_cube.data.mask, [True, False])
diff --git a/tests/integration/cmor/_fixes/cmip6/test_canesm5_canoe.py b/tests/integration/cmor/_fixes/cmip6/test_canesm5_canoe.py
new file mode 100644
index 000000000..349444559
--- /dev/null
+++ b/tests/integration/cmor/_fixes/cmip6/test_canesm5_canoe.py
@@ -0,0 +1,27 @@
+"""Test fixes for CanESM5-CanOE."""
+from esmvalcore.cmor._fixes.cmip6.canesm5 import Co2 as BaseCo2
+from esmvalcore.cmor._fixes.cmip6.canesm5 import Gpp as BaseGpp
+from esmvalcore.cmor._fixes.cmip6.canesm5_canoe import Co2, Gpp
+from esmvalcore.cmor._fixes.fix import Fix
+
+
+def test_get_co2_fix():
+    """Test getting of fix."""
+    fix = Fix.get_fixes('CMIP6', 'CanESM5-CanOE', 'Amon', 'co2')
+    assert fix == [Co2(None)]
+
+
+def test_co2_fix():
+    """Test fix for ``co2``."""
+    assert issubclass(Co2, BaseCo2)
+
+
+def test_get_gpp_fix():
+    """Test getting of fix."""
+    fix = Fix.get_fixes('CMIP6', 'CanESM5-CanOE', 'Lmon', 'gpp')
+    assert fix == [Gpp(None)]
+
+
+def test_gpp_fix():
+    """Test fix for ``gpp``."""
+    assert issubclass(Gpp, BaseGpp)