Repository: pydata/xarray

Issue:
Add support for netcdf4 enum
### Is your feature request related to a problem?

When a netcdf file contains [netcdf4 enums ](https://unidata.github.io/netcdf4-python/#enum-data-type), xarray ignores the underlying enum type.
The association between the values of the variable and their actual meaning is then lost.

MRE:
```py
import netCDF4 as nc
import xarray as xr

# -- Create dataset with an enum using the netcdf4 lib
ds = nc.Dataset("mre.nc", "w", format="NETCDF4")   
cloud_type_enum = ds.createEnumType(int,"cloud_type",{"clear":0, "cloudy":1})
print(ds.enumtypes)
# {'cloud_type': <class 'netCDF4._netCDF4.EnumType'>: name = 'cloud_type', numpy dtype = int64, fields/values ={'clear': 0, 'cloudy': 1}} 
ds.createVariable("cloud", cloud_type_enum)
ds["cloud"][0] = 1
ds.close()

# -- Open dataset with xarray
xr_ds = xr.open_dataset("./mre.nc")
print(xr_ds.cloud)
# <xarray.DataArray 'cloud' ()> \n [1 values with dtype=int64]   
# --> We get no metadata about the cloud_type enum that we created above 
xr.ds.to_netcdf("mre_xr.nc")

# -- Open xarray outputted dataset with netCDF4 lib
print(nc.Dataset("mre_xr.nc", "r", format="NETCDF4").enumtypes())
# {}
# --> Empty dictionary: the enum we created is lost
```

If you know CF, enums could replace replace `flag_meanings` and `flag_values`, see [CF](http://cfconventions.org/cf-conventions/cf-conventions.html#flags)
Enums are [not yet ](https://github.com/cf-convention/discuss/issues/238) part of CF though.

### Describe the solution you'd like

As far as I understand, to describe the enum we only need a dictionary that map numbers (enum key) to string (enum value) and a way to reference this dictionary in variables that are "typed" to this enum.
Bear in mind that the dtype of the variable would still be a number, the enum type would be a secondary metadata.

### Describe alternatives you've considered

Most people that produce data could get away with using flag_meanings and flag_values to describe their data in a way which is both CF proof and properly managed by xarray.
For me, the only workaround at the moment is to use the netCDF4 library directly.


### Additional context

```py
nc.__version__
# 1.6.2

xr.__version__
# 2023.2.0
```

Discussion:
bzah: According to netCDF4 example [here](https://unidata.github.io/netcdf4-python/#enum-data-type), to deal with missing values in variables based having a meaning explained by an enum, the enum declaration should contain a mapping between fillvalue and "Missing" or equivalent value.

But the netCDF C and python libraries do not enforce this.
So we can end up with clunky variables that contains missing_values but do not 

Related to https://github.com/Unidata/netcdf-c/issues/982, in particular https://github.com/Unidata/netcdf-c/issues/982#issuecomment-389284835

benbovy: Thanks @bzah for working on this! I’m probably not the most qualified to review this PR as I’m not familiar with the netCDF4 Enum data type, though.

> Validate with xarray team if it's ok to add attribute to Variable and DataArray

This seems a bit too invasive to me for such an edge case. Propagating variable metadata (`attrs` and `encoding`) in xarray operations is already complicated and is often source of bugs. Extending the xarray.Variable class with two additional attributes to track will make the situation worse.

Since this is specific to the netCDF4 format, a less invasive approach would be to implement a class `EnumArray(VariableCoder)` (see xarray/coding/variables.py) and plug it into xarray's builtin netcdf backend. I see two options for decoding an enum netcdf variable:

1. return an xarray.Variable with integer (e.g., np.uint8) dtype and extract the enum_dict as an item of the variable `attrs` (or `encoding`?).
2. wrap the netcdf4 variable as a custom, explicitly indexed (duck) array (see xarray/core/indexing.py for a few classes inheriting from `ExplicitlyIndexedNDArrayMixin`) 
bzah: Hi @benbovy and thanks for the comment. I guessed it would be a bit spicy to add attributes to Variable, but I wanted to have a naive working approach to enable discussions. 
Note that enums are also available in HDF5 so we could implement an enum handler for some other backends. I don't think enums are part of zarr specification though.

TBH, I don't really like using encoding and attrs. I can see that it is convenient to put everything into theses two dictionaries, but this is not tidy enough for me. I like when I can look at a class definition and know what can be there and what cannot. With attrs or encoding this is all hidden IMHO.

I like the ExplicitlyIndexedNDArrayMixin idea. I will try to make something out of it.
dcherian: (1) is definitely a lot easier. We'd also want to support specifying the enum type at write, so that we can roundtrip the file.

(2) would be a lot more involved. What kind of operations would you like to see take advantage of the enum dictionary?
jhamman: > (1) is definitely a lot easier. We'd also want to support specifying the enum type at write, so that we can roundtrip the file.

+1 on this. Don't want to push you too hard off (2) but (1) would have been my recommended approach.

> TBH, I don't really like using encoding and attrs.

I'd be interested to get more of your rational on this. We've been discussing making `encoding` a read-only field (outside of the backend constructors). For more control over the content of attrs, you may want to look at [xarray-dataclasses](https://pypi.org/project/xarray-dataclasses/) or [xarray-schema](https://github.com/xarray-contrib/xarray-schema).
kmuehlbauer: > (1) is definitely a lot easier. We'd also want to support specifying the enum type at write, so that we can roundtrip the file.

+1 on that, too. I'm having this on the plate for h5netcdf anyway, so would be good to coordinate. 
bzah: Ok I have a simple working implementation for enums. I still have unit tests to fix and to add though.
However, I **know** that it will not work for datasets that already have enums out there.
There is indeed a hole in the specification and implementation of HDF5 and netcdf-c that make it possible to create clunky datasets.

Basically, you can create datasets with fill_values outside the enum range and they are considered valid by HDF5.
For example when creating a dataset with: 
```py
    import netCDF4 as nc
    import xarray as xr

    clouds_ds = nc.Dataset("clouds_ds__explicit_fill_value.nc", "w")
    cloud_type = clouds_ds.createEnumType("u1","cloud_type", {"clear": 0, "cloudy":1})
    clouds_ds.createDimension("time", size=10)
    clouds_ds.createVariable(
                "clouds",
                cloud_type,
                "time",
                fill_value=255, # or None, same result
            )
    clouds_ds["clouds"][0] = 1
    print(clouds_ds["clouds"][:].data)
    # [out] [ 1 255 255 255 255 255 255 255 255 255 ]
    clouds_ds.close()
```
netCDF4 lets you create a variable with fill_value outside the enum range but
- ncdump will crash if you do `ncdump clouds_ds__explicit_fill_value.nc`
 - And with the current PR's implementation xarray will fail to write it too:
```py
xr_ds = xr.open_dataset("clouds_ds__enumed_fill_value.nc")
xr_ds.to_netcdf("xr_clouds-clouds_ds__enumed_fill_value.nc")
# --> throws an exception because xarray unmasks the values 
#       and try to push 255 (fill_value) in the resulting netCDF file.
```

It's worth noting that data producer may be tempted to avoid specifying a missing values in the enum definition if they believe it will always be filled with something. But I believe it should be discouraged. 

## Possible workarounds
When reading a netCDF with enums, if fill_value (either in attributes or from the mask) is **not** in the enum possible values and there are missing values, then:
 - 1) Add a dedicated enum key value pair for the fill_value. Like `{"_Undefined": <fill_value>}`. Thus modifying the enum declaration.
 - 2) Or, keep track of the mask of missing values and re-apply it at writing time (assuming it's still valid).
 - 3) Or, forbid opening the file as we consider it invalid, and perhaps have a flag to fallback to i.

I don;t like **i.** because we loose by simply opening a file and rewriting a copy, the content would not be identical.
**ii.** seems bad too because if the variable is modified by the user we can't know where to apply the mask.
**iii.** would be the best IMO, in particular it might force data producer that use xarray to think about missing_values when designing enums.

Relevant discussion on netcdf-c: https://github.com/Unidata/netcdf-c/issues/982

Do you have suggestions ?



kmuehlbauer: @bzah Thanks for this first wip implementation. I'll try to review over the next days.
kmuehlbauer: Hi @bzah, yes indeed, the default netcdf fill_value issue is a tricky one. There is a general discussion in #2742 with quite some offspring issues.

In general xarray has a relaxed view when it comes to reading non-standard/broken/mismatched (you name it) files. If it is readable, xarray should be able to import it. As netcdf4-python is able to read those files users will expect xarray to ingest it too.

So I'd add another point to your above list:

-  iv. do nothing on read (just read), then raise on write with a meaningful error

As this affects only existing files

- which have default_fill value activated and are not written completely
- which set a `_FillValue` outside the enum-range

we might get away without too much hassle.

For the overall approach we could think to create `flag_meanings` and `flag_values` attributes on read (established CF convention) in conjunction with a new key `enum` in `encoding` (`encoding["enum"]="my_enum_name"`). On write, those attributes should be removed and the netCDF4.EnumType be created instead.

PROs:

- users current workflows with `flag_meanings`/`flag_values` will automagically work for the new netCDF4.EnumType too
- even if encoding is lost while processing, data will still be written out in a meaningful way (`flag_meanings` and `flag_values`)
- can be meaningful serialized to other backends, which do not have a notion of enum
- users could use the new netCDF4.EnumType as output just by setting `encoding["enum"]="my_enum_name"` for their existing variables (if they have `flag_meanings` and `flag_values` available)
- no need to add `dict` to `valid_types` for the backend API
- no need to invent new naming scheme (`enum_name`, `enum_meaning`)
- backwards and roundtrip compatible (encoding["dtype"] isn't touched)

CONs:

- I'm surely biased, but I can't immediately see any

Also note this comment from @samain-eum https://github.com/cf-convention/discuss/issues/238#issuecomment-1541610179 where EUMETSAT is following a similar path.

WDYT @bzah?











bzah: Hi @kmuehlbauer, thanks for linking the fill_value issue, interesting read. I would be willing to try to open a PR to fix that (fetching the mask, getting the implicit fill_value and making it explicit in attrs) once I'm done with Enums.

Regarding:
>    iv. do nothing on read (just read), then raise on write with a meaningful error

Looks better than raising an error on read, but might be frustrating for user if they do all their modifications and get an error on calling `to_netcdf`.
In particular if the opening, processing and saving is done by a library, then it would be the role of this library to reject or fix the file.
Maybe printing a warning at the opening of the file and having a flag `has_valid_enum_fill_value` would help ?


As for:
> create flag_meanings and flag_values attributes on read [...] [and] encoding["enum"]="my_enum_name". On write, those attributes should be removed and the netCDF4.EnumType be created instead. 

I like the idea, it's simpler than mine and from xarray point of view looks elegant and make it easy to be CF compliant.
In particular, we avoid _some_ of inconsistencies described by @samain-eum because xarray users will not have access to the EnumType, so they can only modify flag_meanings and flag_values to update the enum declaration.

However, one enumType may be used in several variables, possibly in several groups. In my opinion, the biggest isuse with flag_meanings and flag_values is how to synchronize them across variables when one change:
Say I have an enum like: `safety: {"safe": 0, "unsafe": 1}`.
As this is a very generic enum, it can be used in several groups and variables. We would be tempted to declare it once in a parent group, just like you would do with a dimension.
However, if our xarray user want to add a level `"very_unsafe": 2`, it would be tedious to keep the consistency for all the flag_meanings and flag_values in other variables.
Even without groups, in the same Dataset we could have multiple DataArray which rely on the same enum and we would have the same consistency issue.
Whereas, the EnumType in netCDF4 declared once and modifying it would update every variable typed with it.

One solution could be to have these `<enum_name>_flag_values` and `<enum_name>_flag_meanings` at Dataset level but it's no longer CF compliant.

Note that my implementation has the same consistency problem as what you are suggesting.
kmuehlbauer: > Hi @kmuehlbauer, thanks for linking the fill_value issue, interesting read. I would be willing to try to open a PR to fix that (fetching the mask, getting the implicit fill_value and making it explicit in attrs) once I'm done with Enums.

This has been tried before, but there was no conclusion on how to handle this, see https://github.com/pydata/xarray/pull/5680#issuecomment-895455163 and ff.

> 
> Regarding:
> 
> > iv. do nothing on read (just read), then raise on write with a meaningful error
> 
> Looks better than raising an error on read, but might be frustrating for user if they do all their modifications and get an error on calling `to_netcdf`. In particular if the opening, processing and saving is done by a library, then it would be the role of this library to reject or fix the file. Maybe printing a warning at the opening of the file and having a flag `has_valid_enum_fill_value` would help ?

Warnings might be overlooked or disabled by users. They might help a bit, though. The users might be frustrated because their source data is broken, but an explicit error message raised by xarray describing the problem should help them to fix things before another write attempt. And, just to note that, without that the error would be thrown by netCDF4-python (as is now). 

> As for:
> 
> > create flag_meanings and flag_values attributes on read [...] [and] encoding["enum"]="my_enum_name". On write, those attributes should be removed and the netCDF4.EnumType be created instead.
> 
> I like the idea, it's simpler than mine and from xarray point of view looks elegant and make it easy to be CF compliant. In particular, we avoid _some_ of inconsistencies described by @samain-eum because xarray users will not have access to the EnumType, so they can only modify flag_meanings and flag_values to update the enum declaration.

I'm interested in how to modify an existing enum. I could not find anything about that in the netCDF4-python docs.

> However, one enumType may be used in several variables, possibly in several groups. In my opinion, the biggest isuse with flag_meanings and flag_values is how to synchronize them across variables when one change: Say I have an enum like: `safety: {"safe": 0, "unsafe": 1}`. As this is a very generic enum, it can be used in several groups and variables. We would be tempted to declare it once in a parent group, just like you would do with a dimension. However, if our xarray user want to add a level `"very_unsafe": 2`, it would be tedious to keep the consistency for all the flag_meanings and flag_values in other variables. Even without groups, in the same Dataset we could have multiple DataArray which rely on the same enum and we would have the same consistency issue. Whereas, the EnumType in netCDF4 declared once and modifying it would update every variable typed with it.

This is one thing which might be a problem, but the backends can do this kind of discovery (eg. for dimensions IIRC). I have doubt's that the enum type can be updated without touching/rewriting all connected variables. The enum type is directly written to the hdf5 dataset (see h5dump), beside being declared as DATATYPE.

```
DATATYPE "my_enum" H5T_ENUM {
      H5T_STD_I16LE;
      "a"                1;
      "b"                2;
   };
 DATASET "my_var" {
    DATATYPE  H5T_ENUM {
       H5T_STD_I16LE;
       "a"                1;
       "b"                2;
    }
  :
  :
```

Looking at this, I'm also interested how netcdf-c maps the declared DATATYPE to already existing DATASETs with that DATATYPE? And why netcdf decided to have named enum types as obviously every DATASET has the enum type attached to itself? I'll do some experimenting myself also for the implementation in h5netcdf.

Another interesting note is that h5py adds a little metadata to it's numpy dtype to mark it as enum:

```python
import h5py
dt = h5py.enum_dtype({"RED": 0, "GREEN": 1, "BLUE": 42}, basetype='i')
print(dt.type)
print(dt.metadata)
<class 'numpy.int32'>
{'enum': {'RED': 0, 'GREEN': 1, 'BLUE': 42}}
```


bzah: Hi @kmuehlbauer and sorry for the late reply.

I have added a `ValueError` with an informative message for when we try to save a variable which has values outside the Enum range.

> I'm interested in how to modify an existing enum. I could not find anything about that in the netCDF4-python docs.

My bad, it doesn't seemed to be possible with netCDF4. But at least, with this PR changes, it will be possible with xarray. One would simply need to edit flag_values and flag_meanings and save the Dataset to netcdf again.

About keeping the consistency of enums across variables: when we write the variables to netCDF4, if multiple enums have the same name, we now also raise an error so that the user know they have to modify the flag_* attributes or the encoding of the variable.
kmuehlbauer: @bzah No worries. I'll try to have a look over the weekend, but can't promise. Trying to catch up next week otherwise.
bzah: I'm not quite sure how to fix the Doctests issue here: 
```py
../../../micromamba/envs/xarray-tests/lib/python3.11/site-packages/h5py/__init__.py:36: in <module>
    _warn(("h5py is running against HDF5 {0} when it was built against {1}, "
E   UserWarning: h5py is running against HDF5 1.14.3 when it was built against 1.14.2, this may cause problems
Error: Process completed with exit code 4.
```
Does anyone have an idea ?
keewis: We're seeing this on `main`, as well, see #8539 for an attempt to silence this (it is an upstream issue).
kmuehlbauer: I've added my suggestions and also removed the test file which slipped in. Needed to special case h5netcdf for now until upstream has added this feature (named enum types).

To make it more explicit I had to add `NativeEnumCoder` to add the dtype metadata in the encoding step. We still can have dedicated `CFEnumCoder` later (flags* stuff).

With the metadata trick this works also for `h5netcdf` backend (with `invalid_netcdf`=True). It writes a transient enum to the dataset (as netCDF4 does) but does not create the named enum type in the file. This has to be implemented in h5netcdf.

I'm always have trouble with typing, so appreciate any help with this. Beside the typing this is ready. @bzah is this also good to go from your side? We might have to add some note to the io-docs.
bzah: Many thanks @kmuehlbauer for these improvements. I will have a look at mypy issues.
kmuehlbauer: @dcherian I think this is ready for final round of review. Can't get the one windows run to work, error seems unrelated. Thanks @bzah for fixing the typing.
kmuehlbauer: @dcherian I'm about to merge this, but I'm a bit unsure about the failing test. It looks like its not related, but every once in a while all tests succeed. Is this a flaky thing and we can merge away? Thanks!

Patch:
diff --git a/doc/whats-new.rst b/doc/whats-new.rst
index db32de3c..80ea2974 100644
--- a/doc/whats-new.rst
+++ b/doc/whats-new.rst
@@ -224,6 +224,10 @@ New Features
 
 - Use `opt_einsum <https://optimized-einsum.readthedocs.io/en/stable/>`_ for :py:func:`xarray.dot` by default if installed.
   By `Deepak Cherian <https://github.com/dcherian>`_. (:issue:`7764`, :pull:`8373`).
+- Decode/Encode netCDF4 enums and store the enum definition in dataarrays' dtype metadata.
+  If multiple variables share the same enum in netCDF4, each dataarray will have its own
+  enum definition in their respective dtype metadata.
+  By `Abel Aoun <https://github.com/bzah>_`(:issue:`8144`, :pull:`8147`)
 - Add ``DataArray.dt.total_seconds()`` method to match the Pandas API. (:pull:`8435`).
   By `Ben Mares <https://github.com/maresb>`_.
 - Allow passing ``region="auto"`` in  :py:meth:`Dataset.to_zarr` to automatically infer the
diff --git a/xarray/backends/netCDF4_.py b/xarray/backends/netCDF4_.py
index cf753828..d3845568 100644
--- a/xarray/backends/netCDF4_.py
+++ b/xarray/backends/netCDF4_.py
@@ -49,7 +49,6 @@ if TYPE_CHECKING:
 # string used by netCDF4.
 _endian_lookup = {"=": "native", ">": "big", "<": "little", "|": "native"}
 
-
 NETCDF4_PYTHON_LOCK = combine_locks([NETCDFC_LOCK, HDF5_LOCK])
 
 
@@ -141,7 +140,9 @@ def _check_encoding_dtype_is_vlen_string(dtype):
         )
 
 
-def _get_datatype(var, nc_format="NETCDF4", raise_on_invalid_encoding=False):
+def _get_datatype(
+    var, nc_format="NETCDF4", raise_on_invalid_encoding=False
+) -> np.dtype:
     if nc_format == "NETCDF4":
         return _nc4_dtype(var)
     if "dtype" in var.encoding:
@@ -234,13 +235,13 @@ def _force_native_endianness(var):
 
 
 def _extract_nc4_variable_encoding(
-    variable,
+    variable: Variable,
     raise_on_invalid=False,
     lsd_okay=True,
     h5py_okay=False,
     backend="netCDF4",
     unlimited_dims=None,
-):
+) -> dict[str, Any]:
     if unlimited_dims is None:
         unlimited_dims = ()
 
@@ -308,7 +309,7 @@ def _extract_nc4_variable_encoding(
     return encoding
 
 
-def _is_list_of_strings(value):
+def _is_list_of_strings(value) -> bool:
     arr = np.asarray(value)
     return arr.dtype.kind in ["U", "S"] and arr.size > 1
 
@@ -414,13 +415,25 @@ class NetCDF4DataStore(WritableCFDataStore):
     def ds(self):
         return self._acquire()
 
-    def open_store_variable(self, name, var):
+    def open_store_variable(self, name: str, var):
+        import netCDF4
+
         dimensions = var.dimensions
-        data = indexing.LazilyIndexedArray(NetCDF4ArrayWrapper(name, self))
         attributes = {k: var.getncattr(k) for k in var.ncattrs()}
+        data = indexing.LazilyIndexedArray(NetCDF4ArrayWrapper(name, self))
+        encoding: dict[str, Any] = {}
+        if isinstance(var.datatype, netCDF4.EnumType):
+            encoding["dtype"] = np.dtype(
+                data.dtype,
+                metadata={
+                    "enum": var.datatype.enum_dict,
+                    "enum_name": var.datatype.name,
+                },
+            )
+        else:
+            encoding["dtype"] = var.dtype
         _ensure_fill_value_valid(data, attributes)
         # netCDF4 specific encoding; save _FillValue for later
-        encoding = {}
         filters = var.filters()
         if filters is not None:
             encoding.update(filters)
@@ -440,7 +453,6 @@ class NetCDF4DataStore(WritableCFDataStore):
         # save source so __repr__ can detect if it's local or not
         encoding["source"] = self._filename
         encoding["original_shape"] = var.shape
-        encoding["dtype"] = var.dtype
 
         return Variable(dimensions, data, attributes, encoding)
 
@@ -485,21 +497,24 @@ class NetCDF4DataStore(WritableCFDataStore):
         return variable
 
     def prepare_variable(
-        self, name, variable, check_encoding=False, unlimited_dims=None
+        self, name, variable: Variable, check_encoding=False, unlimited_dims=None
     ):
         _ensure_no_forward_slash_in_name(name)
-
+        attrs = variable.attrs.copy()
+        fill_value = attrs.pop("_FillValue", None)
         datatype = _get_datatype(
             variable, self.format, raise_on_invalid_encoding=check_encoding
         )
-        attrs = variable.attrs.copy()
-
-        fill_value = attrs.pop("_FillValue", None)
-
+        # check enum metadata and use netCDF4.EnumType
+        if (
+            (meta := np.dtype(datatype).metadata)
+            and (e_name := meta.get("enum_name"))
+            and (e_dict := meta.get("enum"))
+        ):
+            datatype = self._build_and_get_enum(name, datatype, e_name, e_dict)
         encoding = _extract_nc4_variable_encoding(
             variable, raise_on_invalid=check_encoding, unlimited_dims=unlimited_dims
         )
-
         if name in self.ds.variables:
             nc4_var = self.ds.variables[name]
         else:
@@ -527,6 +542,33 @@ class NetCDF4DataStore(WritableCFDataStore):
 
         return target, variable.data
 
+    def _build_and_get_enum(
+        self, var_name: str, dtype: np.dtype, enum_name: str, enum_dict: dict[str, int]
+    ) -> Any:
+        """
+        Add or get the netCDF4 Enum based on the dtype in encoding.
+        The return type should be ``netCDF4.EnumType``,
+        but we avoid importing netCDF4 globally for performances.
+        """
+        if enum_name not in self.ds.enumtypes:
+            return self.ds.createEnumType(
+                dtype,
+                enum_name,
+                enum_dict,
+            )
+        datatype = self.ds.enumtypes[enum_name]
+        if datatype.enum_dict != enum_dict:
+            error_msg = (
+                f"Cannot save variable `{var_name}` because an enum"
+                f" `{enum_name}` already exists in the Dataset but have"
+                " a different definition. To fix this error, make sure"
+                " each variable have a uniquely named enum in their"
+                " `encoding['dtype'].metadata` or, if they should share"
+                " the same enum type, make sure the enums are identical."
+            )
+            raise ValueError(error_msg)
+        return datatype
+
     def sync(self):
         self.ds.sync()
 
diff --git a/xarray/coding/variables.py b/xarray/coding/variables.py
index 54383394..c3d57ad1 100644
--- a/xarray/coding/variables.py
+++ b/xarray/coding/variables.py
@@ -566,7 +566,7 @@ class NonStringCoder(VariableCoder):
 
 class ObjectVLenStringCoder(VariableCoder):
     def encode(self):
-        return NotImplementedError
+        raise NotImplementedError
 
     def decode(self, variable: Variable, name: T_Name = None) -> Variable:
         if variable.dtype == object and variable.encoding.get("dtype", False) == str:
@@ -574,3 +574,22 @@ class ObjectVLenStringCoder(VariableCoder):
             return variable
         else:
             return variable
+
+
+class NativeEnumCoder(VariableCoder):
+    """Encode Enum into variable dtype metadata."""
+
+    def encode(self, variable: Variable, name: T_Name = None) -> Variable:
+        if (
+            "dtype" in variable.encoding
+            and np.dtype(variable.encoding["dtype"]).metadata
+            and "enum" in variable.encoding["dtype"].metadata
+        ):
+            dims, data, attrs, encoding = unpack_for_encoding(variable)
+            data = data.astype(dtype=variable.encoding.pop("dtype"))
+            return Variable(dims, data, attrs, encoding, fastpath=True)
+        else:
+            return variable
+
+    def decode(self, variable: Variable, name: T_Name = None) -> Variable:
+        raise NotImplementedError()
diff --git a/xarray/conventions.py b/xarray/conventions.py
index 94e285d8..1d8e81e1 100644
--- a/xarray/conventions.py
+++ b/xarray/conventions.py
@@ -48,10 +48,6 @@ if TYPE_CHECKING:
     T_DatasetOrAbstractstore = Union[Dataset, AbstractDataStore]
 
 
-def _var_as_tuple(var: Variable) -> T_VarTuple:
-    return var.dims, var.data, var.attrs.copy(), var.encoding.copy()
-
-
 def _infer_dtype(array, name=None):
     """Given an object array with no missing values, infer its dtype from all elements."""
     if array.dtype.kind != "O":
@@ -111,7 +107,7 @@ def _copy_with_dtype(data, dtype: np.typing.DTypeLike):
 def ensure_dtype_not_object(var: Variable, name: T_Name = None) -> Variable:
     # TODO: move this from conventions to backends? (it's not CF related)
     if var.dtype.kind == "O":
-        dims, data, attrs, encoding = _var_as_tuple(var)
+        dims, data, attrs, encoding = variables.unpack_for_encoding(var)
 
         # leave vlen dtypes unchanged
         if strings.check_vlen_dtype(data.dtype) is not None:
@@ -162,7 +158,7 @@ def encode_cf_variable(
     var: Variable, needs_copy: bool = True, name: T_Name = None
 ) -> Variable:
     """
-    Converts an Variable into an Variable which follows some
+    Converts a Variable into a Variable which follows some
     of the CF conventions:
 
         - Nans are masked using _FillValue (or the deprecated missing_value)
@@ -188,6 +184,7 @@ def encode_cf_variable(
         variables.CFScaleOffsetCoder(),
         variables.CFMaskCoder(),
         variables.UnsignedIntegerCoder(),
+        variables.NativeEnumCoder(),
         variables.NonStringCoder(),
         variables.DefaultFillvalueCoder(),
         variables.BooleanCoder(),
@@ -447,7 +444,7 @@ def decode_cf_variables(
                 decode_timedelta=decode_timedelta,
             )
         except Exception as e:
-            raise type(e)(f"Failed to decode variable {k!r}: {e}")
+            raise type(e)(f"Failed to decode variable {k!r}: {e}") from e
         if decode_coords in [True, "coordinates", "all"]:
             var_attrs = new_vars[k].attrs
             if "coordinates" in var_attrs:
@@ -633,7 +630,11 @@ def cf_decoder(
     decode_cf_variable
     """
     variables, attributes, _ = decode_cf_variables(
-        variables, attributes, concat_characters, mask_and_scale, decode_times
+        variables,
+        attributes,
+        concat_characters,
+        mask_and_scale,
+        decode_times,
     )
     return variables, attributes
 
diff --git a/xarray/core/dataarray.py b/xarray/core/dataarray.py
index d2ea0f8a..e6e65c73 100644
--- a/xarray/core/dataarray.py
+++ b/xarray/core/dataarray.py
@@ -4062,6 +4062,9 @@ class DataArray(
         name is the same as a coordinate name, then it is given the name
         ``"__xarray_dataarray_variable__"``.
 
+        [netCDF4 backend only] netCDF4 enums are decoded into the
+        dataarray dtype metadata.
+
         See Also
         --------
         Dataset.to_netcdf


Test patch:
diff --git a/xarray/tests/test_backends.py b/xarray/tests/test_backends.py
index 104b6d08..d01cfd7f 100644
--- a/xarray/tests/test_backends.py
+++ b/xarray/tests/test_backends.py
@@ -1704,6 +1704,126 @@ class NetCDF4Base(NetCDFBase):
                 with self.roundtrip(ds):
                     pass
 
+    @requires_netCDF4
+    def test_encoding_enum__no_fill_value(self):
+        with create_tmp_file() as tmp_file:
+            cloud_type_dict = {"clear": 0, "cloudy": 1}
+            with nc4.Dataset(tmp_file, mode="w") as nc:
+                nc.createDimension("time", size=2)
+                cloud_type = nc.createEnumType("u1", "cloud_type", cloud_type_dict)
+                v = nc.createVariable(
+                    "clouds",
+                    cloud_type,
+                    "time",
+                    fill_value=None,
+                )
+                v[:] = 1
+            with open_dataset(tmp_file) as original:
+                save_kwargs = {}
+                if self.engine == "h5netcdf":
+                    save_kwargs["invalid_netcdf"] = True
+                with self.roundtrip(original, save_kwargs=save_kwargs) as actual:
+                    assert_equal(original, actual)
+                    assert (
+                        actual.clouds.encoding["dtype"].metadata["enum"]
+                        == cloud_type_dict
+                    )
+                    if self.engine != "h5netcdf":
+                        # not implemented in h5netcdf yet
+                        assert (
+                            actual.clouds.encoding["dtype"].metadata["enum_name"]
+                            == "cloud_type"
+                        )
+
+    @requires_netCDF4
+    def test_encoding_enum__multiple_variable_with_enum(self):
+        with create_tmp_file() as tmp_file:
+            cloud_type_dict = {"clear": 0, "cloudy": 1, "missing": 255}
+            with nc4.Dataset(tmp_file, mode="w") as nc:
+                nc.createDimension("time", size=2)
+                cloud_type = nc.createEnumType("u1", "cloud_type", cloud_type_dict)
+                nc.createVariable(
+                    "clouds",
+                    cloud_type,
+                    "time",
+                    fill_value=255,
+                )
+                nc.createVariable(
+                    "tifa",
+                    cloud_type,
+                    "time",
+                    fill_value=255,
+                )
+            with open_dataset(tmp_file) as original:
+                save_kwargs = {}
+                if self.engine == "h5netcdf":
+                    save_kwargs["invalid_netcdf"] = True
+                with self.roundtrip(original, save_kwargs=save_kwargs) as actual:
+                    assert_equal(original, actual)
+                    assert (
+                        actual.clouds.encoding["dtype"] == actual.tifa.encoding["dtype"]
+                    )
+                    assert (
+                        actual.clouds.encoding["dtype"].metadata
+                        == actual.tifa.encoding["dtype"].metadata
+                    )
+                    assert (
+                        actual.clouds.encoding["dtype"].metadata["enum"]
+                        == cloud_type_dict
+                    )
+                    if self.engine != "h5netcdf":
+                        # not implemented in h5netcdf yet
+                        assert (
+                            actual.clouds.encoding["dtype"].metadata["enum_name"]
+                            == "cloud_type"
+                        )
+
+    @requires_netCDF4
+    def test_encoding_enum__error_multiple_variable_with_changing_enum(self):
+        """
+        Given 2 variables, if they share the same enum type,
+        the 2 enum definition should be identical.
+        """
+        with create_tmp_file() as tmp_file:
+            cloud_type_dict = {"clear": 0, "cloudy": 1, "missing": 255}
+            with nc4.Dataset(tmp_file, mode="w") as nc:
+                nc.createDimension("time", size=2)
+                cloud_type = nc.createEnumType("u1", "cloud_type", cloud_type_dict)
+                nc.createVariable(
+                    "clouds",
+                    cloud_type,
+                    "time",
+                    fill_value=255,
+                )
+                nc.createVariable(
+                    "tifa",
+                    cloud_type,
+                    "time",
+                    fill_value=255,
+                )
+            with open_dataset(tmp_file) as original:
+                assert (
+                    original.clouds.encoding["dtype"].metadata
+                    == original.tifa.encoding["dtype"].metadata
+                )
+                modified_enum = original.clouds.encoding["dtype"].metadata["enum"]
+                modified_enum.update({"neblig": 2})
+                original.clouds.encoding["dtype"] = np.dtype(
+                    "u1",
+                    metadata={"enum": modified_enum, "enum_name": "cloud_type"},
+                )
+                if self.engine != "h5netcdf":
+                    # not implemented yet in h5netcdf
+                    with pytest.raises(
+                        ValueError,
+                        match=(
+                            "Cannot save variable .*"
+                            " because an enum `cloud_type` already exists in the Dataset .*"
+                        ),
+                    ):
+                        with self.roundtrip(original):
+                            pass
+
 
 @requires_netCDF4
 class TestNetCDF4Data(NetCDF4Base):