Repository: yt-project/yt

Issue:
CF radial frontend is causing SIGSEGV for certain files
### Bug report

**Bug summary**

CFRadial uses `xarray.open_dataset` in `_is_valid` and due to https://github.com/pydata/xarray/issues/6728 it causes SIGSEGV in `yt.load` for certain files (e.g. `GasSloshing/sloshing_nomag2_hdf5_plt_cnt_0150`)

**Code for reproduction**

```python

from yt.frontends.cf_radial.data_structures import CFRadialDataset

CFRadialDataset._is_valid("GasSloshing/sloshing_nomag2_hdf5_plt_cnt_0150")

```

**Actual outcome**

```
$ python3 -Wd -X faulthandler bug.py 
Fatal Python error: Segmentation fault

Current thread 0x00007fe5cdb3c740 (most recent call first):
  File "/home/xarth/codes/xarthisius/xarray/xarray/backends/netCDF4_.py", line 106 in _getitem
  File "/home/xarth/codes/xarthisius/xarray/xarray/core/indexing.py", line 816 in explicit_indexing_adapter
  File "/home/xarth/codes/xarthisius/xarray/xarray/backends/netCDF4_.py", line 93 in __getitem__
  File "/home/xarth/codes/xarthisius/xarray/xarray/core/indexing.py", line 527 in __array__
  File "/home/xarth/codes/xarthisius/xarray/xarray/core/variable.py", line 250 in _as_array_or_item
  File "/home/xarth/codes/xarthisius/xarray/xarray/core/variable.py", line 510 in values
  File "/home/xarth/codes/xarthisius/xarray/xarray/core/variable.py", line 337 in data
  File "/home/xarth/codes/xarthisius/xarray/xarray/core/common.py", line 1866 in contains_cftime_datetimes
  File "/home/xarth/codes/xarthisius/xarray/xarray/core/common.py", line 1875 in _contains_datetime_like_objects
  File "/home/xarth/codes/xarthisius/xarray/xarray/conventions.py", line 345 in decode_cf_variable
  File "/home/xarth/codes/xarthisius/xarray/xarray/conventions.py", line 521 in decode_cf_variables
  File "/home/xarth/codes/xarthisius/xarray/xarray/backends/store.py", line 27 in open_dataset
  File "/home/xarth/codes/xarthisius/xarray/xarray/backends/netCDF4_.py", line 567 in open_dataset
  File "/home/xarth/codes/xarthisius/xarray/xarray/backends/api.py", line 495 in open_dataset
  File "/home/xarth/codes/xarthisius/yt/bug.py", line 10 in <module>
Segmentation fault (core dumped)
```

**Expected outcome**

`_is_valid` not to die.

**Notes**
Happens with `xarray > 2022.3.1`



Discussion:
Thanks for catching this @Xarthisius
Do you think there's anything we can do about it on yt's side ?
Yeah, we can try harder to detect if it's netCDF4 or not. Possibly implementing this: https://github.com/pedro-vicente/netcdf-detect
okay then, for information: @zssherman, @chrishavlin 
One option we could do is to have a very coarse check for cf-radial, but not forbid it to be loaded explicitly.  So for instance, if there's a file name convention that we could check for, that could then proceed through `_is_valid`, and if the file doesn't follow that convention then maybe it needs to be loaded with the base class.
As a reminder, yt.load recently got a `hint` argument that should make direct class initialization unnecessary. Otherwise this seems like a good idea !
also, another idea that I almost implemented when I ran into other problems with the `_is_valid` using xarray that it appears have not actually been fully resolved -- we could use yt's existing netcdf handler (that relies on the netcdf package but not xarray) for just an initial check within `_is_valid` and only use xarray for loading once we know it is a cf radial netcdf. Essentially copy over and modify the `nc4_cm1` frontend's `_is_valid`. 

Resolving patch:
diff --git a/conftest.py b/conftest.py
--- a/conftest.py
+++ b/conftest.py
@@ -186,17 +186,6 @@ def pytest_configure(config):
             ),
         )
 
-    if find_spec("xarray") is not None:
-        # this can be removed when the related fix is published
-        # https://github.com/pydata/xarray/issues/6514
-        config.addinivalue_line(
-            "filterwarnings",
-            (
-                "ignore: SelectableGroups dict interface is deprecated. "
-                "Use select.:DeprecationWarning"
-            ),
-        )
-
 
 def pytest_collection_modifyitems(config, items):
     r"""
diff --git a/yt/frontends/cf_radial/data_structures.py b/yt/frontends/cf_radial/data_structures.py
--- a/yt/frontends/cf_radial/data_structures.py
+++ b/yt/frontends/cf_radial/data_structures.py
@@ -16,6 +16,7 @@
 from yt.data_objects.static_output import Dataset
 from yt.funcs import mylog
 from yt.geometry.grid_geometry_handler import GridIndex
+from yt.utilities.file_handler import NetCDF4FileHandler, warn_netcdf
 from yt.utilities.on_demand_imports import _xarray as xr
 
 from .fields import CFRadialFieldInfo
@@ -293,25 +294,22 @@ def _is_valid(cls, filename, *args, **kwargs):
         # This accepts a filename or a set of arguments and returns True or
         # False depending on if the file is of the type requested.
 
+        warn_netcdf(filename)
         is_cfrad = False
         try:
-            with xr.open_dataset(filename, engine="netcdf4") as ds:
-                if hasattr(ds, "attrs") and isinstance(ds.attrs, dict):
-                    con = "Conventions"
-                    cons = ds.attrs.get(con, "") + ds.attrs.get(con.lower(), "")
-                    is_cfrad = "CF/Radial" in cons
-        except (
-            ImportError,
-            OSError,
-            AttributeError,
-            TypeError,
-            ValueError,
-            RuntimeError,
-        ):
-            # catch all these to avoid errors when xarray cant handle a file
-            # note: using engine="netcdf4" avoids warnings that are raised
-            # in newer vesrions of xarray (around v'2022.3.0'). If netcdf4 is not
-            # installed, open_dataset will raise a ValueError when engine="netcdf4"
+            # note that we use the NetCDF4FileHandler here to avoid some
+            # issues with xarray opening datasets it cannot handle. Once
+            # a dataset is as identified as a CFRadialDataset, xarray is used
+            # for opening. See https://github.com/yt-project/yt/issues/3987
+            nc4_file = NetCDF4FileHandler(filename)
+            with nc4_file.open_ds(keepweakref=True) as ds:
+                con = "Conventions"  # the attribute to check for file conventions
+                cons = ""  # the value of the Conventions attribute
+                for c in [con, con.lower()]:
+                    if hasattr(ds, c):
+                        cons += getattr(ds, c)
+                is_cfrad = "CF/Radial" in cons
+        except (OSError, AttributeError, ImportError):
             return False
 
         return is_cfrad


Test patch: