Repository: SciTools/iris
Pull request #6552 (2025-07-08 08:16:51)

Extend Cube Metadata NetCDF zlib compression
## 🚀 Pull Request

### Description
<!-- Provide a clear description about your awesome pull request -->
<!-- Tell us all about your new feature, improvement, or bug fix -->

Extend NetCDF `zlib` compression serialization to also include **auxiliary coordinates** and **ancillary variables** that have the **same shape** as the associated parent cube data payload.

Does not apply to metadata with string dtypes.

The `complevel`, `fletcher32`, `shuffle` and `zlib` kwargs applied to the cube data payload will also be applied to the above applicable metadata.

Closes #6539

Ping @pasc85


---
[Consult Iris pull request check list]( https://scitools-iris.readthedocs.io/en/latest/developers_guide/contributing_pull_request_checklist.html)

---
Add any of the below labels to trigger actions on this PR:

- https://github.com/SciTools/iris/labels/benchmark_this


codecov[bot]: ## [Codecov](https://app.codecov.io/gh/SciTools/iris/pull/6552?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=SciTools) Report
Attention: Patch coverage is `75.00000%` with `5 lines` in your changes missing coverage. Please review.
> Project coverage is 89.89%. Comparing base [(`4e92a5e`)](https://app.codecov.io/gh/SciTools/iris/commit/4e92a5e5bd7f7487a73de8eab7794dd565fae09f?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=SciTools) to head [(`fb05d52`)](https://app.codecov.io/gh/SciTools/iris/commit/fb05d52af9cd1aa3d978737cff80f83c2fa7ebd9?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=SciTools).
> Report is 5 commits behind head on main.

| [Files with missing lines](https://app.codecov.io/gh/SciTools/iris/pull/6552?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=SciTools) | Patch % | Lines |
|---|---|---|
| [lib/iris/fileformats/netcdf/saver.py](https://app.codecov.io/gh/SciTools/iris/pull/6552?src=pr&el=tree&filepath=lib%2Firis%2Ffileformats%2Fnetcdf%2Fsaver.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=SciTools#diff-bGliL2lyaXMvZmlsZWZvcm1hdHMvbmV0Y2RmL3NhdmVyLnB5) | 75.00% | [4 Missing and 1 partial :warning: ](https://app.codecov.io/gh/SciTools/iris/pull/6552?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=SciTools) |

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main    #6552      +/-   ##
==========================================
+ Coverage   89.88%   89.89%   +0.01%     
==========================================
  Files          90       90              
  Lines       23945    24145     +200     
  Branches     4467     4495      +28     
==========================================
+ Hits        21522    21706     +184     
- Misses       1670     1679       +9     
- Partials      753      760       +7     
```

</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/SciTools/iris/pull/6552?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=SciTools).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=SciTools).

<details><summary> :rocket: New features to boost your workflow: </summary>

- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
bjlittle: @pasc85 Brilliant!

Thanks for taking the time and effort to test this and also share the comprehensive test results 🥇 

Looks like this is a good win and you're testing give us confidence that the change is fit for purpose 👍 
pp-mo: > I tested the new feature, and that went really well! The notebook I used works as follows:

This is really useful. 
Just for completeness though, can you explain _what compression settings you applied ?_
If the script is short, it would be great to **just share that**.

Apart from general helpfulness, my reason was ...
I was wondering whether to raise the possible criticism that the same settings might not apply equally well to ancils/aux-coords of different dtypes as the main data.
And I noted what you said that the float32 status did not compress as well as the byte one.

-----
The thing is, this PR is a great quick win, but **I have a bit of a hidden agenda !!**  
I think it would ultimately be much neater and more appropriate to replace our keyword controls for compression, packing and fill-value with a context manager approach as we have done with loading controls -- notably the [load chunking control](https://scitools-iris.readthedocs.io/en/v3.12.2/generated/api/iris.fileformats.netcdf.loader.html#iris.fileformats.netcdf.loader.ChunkControl.set), which allows us to apply different chunking to different file variables.
That type of separate management makes the primary method API tidier, and it is easier to apply more detailed context.  
Whereas, as we have it, the settings apply per cube you are saving, so they must be used for all components of the cube.

@bjlittle what do you think of this ?  
I don't think it can fairly be raised as a criticism of this PR, as that is far too much breaking change to introduce at this point.  But it does show up limitations.  Perhaps we can bank the idea somewhere else.
bjlittle: @pp-mo Great idea, but there's no major benefit given the efforts for this particular use case. I'd be keen to bank this pull-request "as is" without straying off the path too far.

Also, this type of compression is only applicable to NetCDF and not other data formats (easily); we're getting a lot for free from the `netcdf4` package. 

As a compromise why not raise a separate dedicated issue capturing your thoughts, and we can start a conversation there? Perhaps we can agree on a way forward as a separate concern and target `iris` 3.14 ?
bjlittle: @pp-mo Tweaked the non-compression tests to do as I intended, nice spot 👀

Also added some comments to clarify 👍 


pp-mo: > @pp-mo Great idea, but there's no major benefit here given the effort required for this particular use case. I'd be keen to bank this pull-request "as is" without straying off the path too far.
> 

Absolutely

> Also, this type of compression is only applicable to NetCDF and not other data formats (easily); we're getting a lot for free from the `netcdf4` package.

Yes, and in fact the criticism applies more directly to the 'packing' arguments, which aren't covered here.

> As a compromise why not raise a separate dedicated issue capturing your thoughts @pp-mo, and we can start a conversation there? Perhaps we can agree on a way forward as a separate concern and target `iris` 3.14 ?

If anything, given the breaking change nature, perhaps more of an Iris 4 thing?

pp-mo: > I'm talking about netCDF compression here and not straying into the minefield of packing, netCDF or otherwise

Ah ok, wasn't really a clear distinction to me, but point taken.

----

> I wouldn't see what you're proposing beyond this PR as a breaking change, certainly not in the API. Additionally we could still support the compression behaviour proposed here moving forwards as a minimal default.

Well, I was suggesting we might replace the save keywords with a context manager, which I think **_is_** pretty breaking.  I'm not sure we could _combine_ the two effectively, since part of the point would be to be able to compress cube components differently to the cube.  So, I thought a bit future wishlist really.

Patch:
diff --git a/lib/iris/fileformats/netcdf/saver.py b/lib/iris/fileformats/netcdf/saver.py
index a06d4eb9b..95bd92d8a 100644
--- a/lib/iris/fileformats/netcdf/saver.py
+++ b/lib/iris/fileformats/netcdf/saver.py
@@ -586,21 +586,28 @@ class Saver:
         # data-vars in the file.
         cf_mesh_name = self._add_mesh(cube)
 
+        # Group the generic compression keyword arguments together for
+        # convenience, as they will be applied to other cube metadata
+        # as well as the cube data payload.
+        compression_kwargs = {
+            "complevel": complevel,
+            "fletcher32": fletcher32,
+            "shuffle": shuffle,
+            "zlib": zlib,
+        }
+
         # Create the associated cube CF-netCDF data variable.
         cf_var_cube = self._create_cf_data_variable(
             cube,
             cube_dimensions,
-            local_keys,
-            zlib=zlib,
-            complevel=complevel,
-            shuffle=shuffle,
-            fletcher32=fletcher32,
+            local_keys=local_keys,
+            packing=packing,
+            fill_value=fill_value,
             contiguous=contiguous,
             chunksizes=chunksizes,
             endian=endian,
             least_significant_digit=least_significant_digit,
-            packing=packing,
-            fill_value=fill_value,
+            **compression_kwargs,
         )
 
         # Associate any mesh with the data-variable.
@@ -615,7 +622,9 @@ class Saver:
 
         # Add the auxiliary coordinate variables and associate the data
         # variable to them
-        self._add_aux_coords(cube, cf_var_cube, cube_dimensions)
+        self._add_aux_coords(
+            cube, cf_var_cube, cube_dimensions, compression_kwargs=compression_kwargs
+        )
 
         # Add the cell_measures variables and associate the data
         # variable to them
@@ -623,7 +632,9 @@ class Saver:
 
         # Add the ancillary_variables variables and associate the data variable
         # to them
-        self._add_ancillary_variables(cube, cf_var_cube, cube_dimensions)
+        self._add_ancillary_variables(
+            cube, cf_var_cube, cube_dimensions, compression_kwargs=compression_kwargs
+        )
 
         # Add the formula terms to the appropriate cf variables for each
         # aux factory in the cube.
@@ -883,7 +894,14 @@ class Saver:
         return cf_mesh_name
 
     def _add_inner_related_vars(
-        self, cube, cf_var_cube, dimension_names, coordlike_elements
+        self,
+        cube,
+        cf_var_cube,
+        dimension_names,
+        coordlike_elements,
+        /,
+        *,
+        compression_kwargs=None,
     ):
         """Create a set of variables for aux-coords, ancillaries or cell-measures.
 
@@ -913,7 +931,10 @@ class Saver:
                 if cf_name is None:
                     # Not already present : create it
                     cf_name = self._create_generic_cf_array_var(
-                        cube, dimension_names, element
+                        cube,
+                        dimension_names,
+                        element,
+                        compression_kwargs=compression_kwargs,
                     )
                     self._name_coord_map.append(cf_name, element)
 
@@ -929,7 +950,9 @@ class Saver:
                 variable_names = " ".join(sorted(element_names))
                 _setncattr(cf_var_cube, role_attribute_name, variable_names)
 
-    def _add_aux_coords(self, cube, cf_var_cube, dimension_names):
+    def _add_aux_coords(
+        self, cube, cf_var_cube, dimension_names, /, *, compression_kwargs=None
+    ):
         """Add aux. coordinate to the dataset and associate with the data variable.
 
         Parameters
@@ -940,6 +963,9 @@ class Saver:
             A cf variable cube representation.
         dimension_names : list
             Names associated with the dimensions of the cube.
+        compression_kwargs : dict, optional
+            NetCDF data compression keyword arguments.
+
         """
         from iris.mesh.components import (
             MeshEdgeCoords,
@@ -967,6 +993,7 @@ class Saver:
             cf_var_cube,
             dimension_names,
             coords_to_add,
+            compression_kwargs=compression_kwargs,
         )
 
     def _add_cell_measures(self, cube, cf_var_cube, dimension_names):
@@ -988,7 +1015,9 @@ class Saver:
             cube.cell_measures(),
         )
 
-    def _add_ancillary_variables(self, cube, cf_var_cube, dimension_names):
+    def _add_ancillary_variables(
+        self, cube, cf_var_cube, dimension_names, /, *, compression_kwargs=None
+    ):
         """Add ancillary variables measures to the dataset and associate with the data variable.
 
         Parameters
@@ -999,12 +1028,16 @@ class Saver:
             A cf variable cube representation.
         dimension_names : list
             Names associated with the dimensions of the cube.
+        compression_kwargs : dict, optional
+            NetCDF data compression keyword arguments.
+
         """
         return self._add_inner_related_vars(
             cube,
             cf_var_cube,
             dimension_names,
             cube.ancillary_variables(),
+            compression_kwargs=compression_kwargs,
         )
 
     def _add_dim_coords(self, cube, dimension_names):
@@ -1439,7 +1472,7 @@ class Saver:
             values = values.astype(np.int32)
         return values
 
-    def _create_cf_bounds(self, coord, cf_var, cf_name):
+    def _create_cf_bounds(self, coord, cf_var, cf_name, /, *, compression_kwargs=None):
         """Create the associated CF-netCDF bounds variable.
 
         Parameters
@@ -1450,6 +1483,8 @@ class Saver:
             CF-netCDF variable.
         cf_name : str
             Name of the CF-NetCDF variable.
+        compression_kwargs : dict, optional
+            NetCDF data compression keyword arguments.
 
         Returns
         -------
@@ -1457,6 +1492,9 @@ class Saver:
 
         """
         if hasattr(coord, "has_bounds") and coord.has_bounds():
+            if compression_kwargs is None:
+                compression_kwargs = {}
+
             # Get the values in a form which is valid for the file format.
             bounds = self._ensure_valid_dtype(
                 coord.core_bounds(), "the bounds of coordinate", coord
@@ -1489,6 +1527,7 @@ class Saver:
                 boundsvar_name,
                 bounds.dtype.newbyteorder("="),
                 cf_var.dimensions + (bounds_dimension_name,),
+                **compression_kwargs,
             )
             self._lazy_stream_data(data=bounds, cf_var=cf_var_bounds)
 
@@ -1685,8 +1724,11 @@ class Saver:
         cube_or_mesh,
         cube_dim_names,
         element,
+        /,
+        *,
         element_dims=None,
         fill_value=None,
+        compression_kwargs=None,
     ):
         """Create theCF-netCDF variable given dimensional_metadata.
 
@@ -1718,6 +1760,8 @@ class Saver:
             If not set, standard netcdf4-python behaviour : the variable has no
             '_FillValue' property, and uses the "standard" fill-value for its
             type.
+        compression_kwargs : dict, optional
+            NetCDF data compression keyword arguments.
 
         Returns
         -------
@@ -1732,6 +1776,9 @@ class Saver:
         else:
             cube = None
 
+        if compression_kwargs is None:
+            compression_kwargs = {}
+
         # Work out the var-name to use.
         # N.B. the only part of this routine that may use a mesh _or_ a cube.
         cf_name = self._get_coord_variable_name(cube_or_mesh, element)
@@ -1749,6 +1796,9 @@ class Saver:
         # (e.g. =points if a coord, =data if an ancillary, etc)
         data = element._core_values()
 
+        if cube is None or cube.shape != data.shape:
+            compression_kwargs = {}
+
         if np.issubdtype(data.dtype, np.str_):
             # Deal with string-type variables.
             # Typically CF label variables, but also possibly ancil-vars ?
@@ -1811,6 +1861,7 @@ class Saver:
                 data.dtype.newbyteorder("="),
                 element_dims,
                 fill_value=fill_value,
+                **compression_kwargs,
             )
 
             # Add the axis attribute for spatio-temporal CF-netCDF coordinates.
@@ -1820,7 +1871,9 @@ class Saver:
                     _setncattr(cf_var, "axis", axis.upper())
 
             # Create the associated CF-netCDF bounds variable, if any.
-            self._create_cf_bounds(element, cf_var, cf_name)
+            self._create_cf_bounds(
+                element, cf_var, cf_name, compression_kwargs=compression_kwargs
+            )
 
         # Add the data to the CF-netCDF variable.
         self._lazy_stream_data(data=data, cf_var=cf_var)
@@ -2784,16 +2837,16 @@ def save(
         for cube, packspec, fill_value in zip(cubes, packspecs, fill_values):
             sman.write(
                 cube,
-                local_keys,
-                unlimited_dimensions,
-                zlib,
-                complevel,
-                shuffle,
-                fletcher32,
-                contiguous,
-                chunksizes,
-                endian,
-                least_significant_digit,
+                local_keys=local_keys,
+                unlimited_dimensions=unlimited_dimensions,
+                zlib=zlib,
+                complevel=complevel,
+                shuffle=shuffle,
+                fletcher32=fletcher32,
+                contiguous=contiguous,
+                chunksizes=chunksizes,
+                endian=endian,
+                least_significant_digit=least_significant_digit,
                 packing=packspec,
                 fill_value=fill_value,
             )


Test patch:
diff --git a/docs/src/whatsnew/latest.rst b/docs/src/whatsnew/latest.rst
index c92ebbeb2..dfe45c338 100644
--- a/docs/src/whatsnew/latest.rst
+++ b/docs/src/whatsnew/latest.rst
@@ -55,6 +55,10 @@ This document explains the changes made to Iris for this release
    the :data:`iris.FUTURE` object, to avoid breaking existing code.
    (:issue:`3678`, :pull:`6481`, :pull:`6540`)
 
+#. `@bjlittle`_ extended ``zlib`` compression of :class:`~iris.cube.Cube` data payload when saving to NetCDF
+   to also include any auxiliary coordinates and ancillary variables with the same ``shape``.
+   (:issue:`6539`, :pull:`6552`)
+
 
 🐛 Bugs Fixed
 =============
diff --git a/lib/iris/tests/unit/fileformats/netcdf/saver/test_Saver.py b/lib/iris/tests/unit/fileformats/netcdf/saver/test_Saver.py
index 890e706c0..292a21b18 100644
--- a/lib/iris/tests/unit/fileformats/netcdf/saver/test_Saver.py
+++ b/lib/iris/tests/unit/fileformats/netcdf/saver/test_Saver.py
@@ -32,7 +32,7 @@ from iris.coord_systems import (
     TransverseMercator,
     VerticalPerspective,
 )
-from iris.coords import AuxCoord, DimCoord
+from iris.coords import AncillaryVariable, AuxCoord, DimCoord
 from iris.cube import Cube
 from iris.fileformats.netcdf import Saver, _thread_safe_nc
 import iris.tests.stock as stock
@@ -176,6 +176,18 @@ class Test_write(tests.IrisTest):
                 saver.write(cube)
             self.assertCDL(nc_path)
 
+    @staticmethod
+    def _filter_compression_calls(patch, compression_kwargs, mismatch=False):
+        result = []
+        for call in patch.call_args_list:
+            kwargs = call.kwargs
+            if all(kwargs.get(k) == v for k, v in compression_kwargs.items()):
+                if not mismatch:
+                    result.append(call.args[0])
+            elif mismatch:
+                result.append(call.args[0])
+        return result
+
     def _simple_cube(self, dtype):
         data = self.array_lib.arange(12, dtype=dtype).reshape(3, 4)
         points = np.arange(3, dtype=dtype)
@@ -233,6 +245,111 @@ class Test_write(tests.IrisTest):
         )
         self.assertIn(create_var_call, dataset.createVariable.call_args_list)
 
+    def test_compression(self):
+        cube = self._simple_cube(">f4")
+        data_dims, shape = range(cube.ndim), cube.shape
+
+        # add an auxiliary coordinate to test compression
+        aux_coord = AuxCoord(np.zeros(shape), var_name="compress_aux", units="1")
+        cube.add_aux_coord(aux_coord, data_dims=data_dims)
+
+        # add an ancillary variable to test compression
+        anc_coord = AncillaryVariable(
+            np.zeros(shape), var_name="compress_anc", units="1"
+        )
+        cube.add_ancillary_variable(anc_coord, data_dims=data_dims)
+
+        patch = self.patch(
+            "iris.fileformats.netcdf.saver._thread_safe_nc.DatasetWrapper.createVariable"
+        )
+        compression_kwargs = {
+            "complevel": 9,
+            "fletcher32": True,
+            "shuffle": True,
+            "zlib": True,
+        }
+
+        with self.temp_filename(suffix=".nc") as nc_path:
+            with Saver(nc_path, "NETCDF4", compute=False) as saver:
+                saver.write(cube, **compression_kwargs)
+
+        self.assertEqual(5, patch.call_count)
+        result = self._filter_compression_calls(patch, compression_kwargs)
+        self.assertEqual(3, len(result))
+        self.assertEqual({cube.name(), aux_coord.name(), anc_coord.name()}, set(result))
+
+    def test_non_compression__shape(self):
+        cube = self._simple_cube(">f4")
+        data_dims, shape = (0, 1), cube.shape
+
+        # add an auxiliary coordinate to test non-compression (shape)
+        aux_coord = AuxCoord(np.zeros(shape[0]), var_name="non_compress_aux", units="1")
+        cube.add_aux_coord(aux_coord, data_dims=data_dims[0])
+
+        # add an ancillary variable to test non-compression (shape)
+        anc_coord = AncillaryVariable(
+            np.zeros(shape[1]), var_name="non_compress_anc", units="1"
+        )
+        cube.add_ancillary_variable(anc_coord, data_dims=data_dims[1])
+
+        patch = self.patch(
+            "iris.fileformats.netcdf.saver._thread_safe_nc.DatasetWrapper.createVariable"
+        )
+        compression_kwargs = {
+            "complevel": 9,
+            "fletcher32": True,
+            "shuffle": True,
+            "zlib": True,
+        }
+
+        with self.temp_filename(suffix=".nc") as nc_path:
+            with Saver(nc_path, "NETCDF4", compute=False) as saver:
+                saver.write(cube, **compression_kwargs)
+
+        self.assertEqual(5, patch.call_count)
+        result = self._filter_compression_calls(
+            patch, compression_kwargs, mismatch=True
+        )
+        self.assertEqual(4, len(result))
+        # the aux coord and ancil variable are not compressed due to shape, and
+        # the dim coord and its associated bounds are also not compressed
+        expected = {aux_coord.name(), anc_coord.name(), "dim0", "dim0_bnds"}
+        self.assertEqual(expected, set(result))
+
+    def test_non_compression__dtype(self):
+        cube = self._simple_cube(">f4")
+        data_dims, shape = (0, 1), cube.shape
+
+        # add an auxiliary coordinate to test non-compression (dtype)
+        data = np.array(["."] * np.prod(shape)).reshape(shape)
+        aux_coord = AuxCoord(data, var_name="non_compress_aux", units="1")
+        cube.add_aux_coord(aux_coord, data_dims=data_dims)
+
+        patch = self.patch(
+            "iris.fileformats.netcdf.saver._thread_safe_nc.DatasetWrapper.createVariable"
+        )
+        patch.return_value = mock.MagicMock(dtype=np.dtype("S1"))
+        compression_kwargs = {
+            "complevel": 9,
+            "fletcher32": True,
+            "shuffle": True,
+            "zlib": True,
+        }
+
+        with self.temp_filename(suffix=".nc") as nc_path:
+            with Saver(nc_path, "NETCDF4", compute=False) as saver:
+                saver.write(cube, **compression_kwargs)
+
+        self.assertEqual(4, patch.call_count)
+        result = self._filter_compression_calls(
+            patch, compression_kwargs, mismatch=True
+        )
+        self.assertEqual(3, len(result))
+        # the aux coord is not compressed due to its string dtype, and
+        # the dim coord and its associated bounds are also not compressed
+        expected = {aux_coord.name(), "dim0", "dim0_bnds"}
+        self.assertEqual(expected, set(result))
+
     def test_least_significant_digit(self):
         cube = Cube(
             self.array_lib.array([1.23, 4.56, 7.89]),