Repository: yt-project/yt

Issue:
DEP: reliably solving import order (h5py VS netcdf4)
`netcdf4` and `h5py` are both (currently) optional dependencies. We use proxy objects defined in `yt.utilities.on_demand_imports` to access their functionalities, which are instances of the `OnDemand` class, whose whole purpose is to delay imports until they are actually needed, and serve a informative error message if packages requested are not installed.

`netcdf4` is however a special case for which we intentionally trigger the import mechanism at yt's startup time:

https://github.com/yt-project/yt/blob/4a3c266cb2216a103069754b828e40971087a988/yt/utilities/on_demand_imports.py#L85-L94

An existing (minor) issue with this workaround is that it doesn't work if `h5py` was imported before `yt`.
A more worrisome one is that, were `h5py` be promoted back to a hard requirement to `yt` (which has been proposed and discussed multiple times recently), this solution would simply not work, because we'd have to give up control over the import order.

Now, a couple naive observations:
- the ticket on `xarray` was closed almost 3 years ago
- travis-ci isn't something we use
- fundamentally, the bug isn't ours to solve

I don't have a deep understanding of the lock issue that motivated this patch, so I may be wrong in my conclusions, but from where I'm standing it looks like we could just stop making `netcdf4` a special case among our optional dependencies.

I believe this issue needs to be resolved if we ever want to promote `h5py` to a hard requirement, but otherwise there's no rush.

ping @chrishavlin @jzuhone


Discussion:
Even though the xarray ticket is closed, I'd say this behavior is still an issue since it comes out of intricacies of how environments are setup so it's possible that in certain combinations of h5py and netcdf4 versions you'd still get this error and I don't think it's as simple as requiring some minimum version of h5py and/or netcdf4. 

Here's a question -- say we make h5py a hard dependency can we not just still use the `on_demand_import` framework for h5py imports within yt code? 

I will say that I do think we need to handle this in some way. Even though it's not a yt bug, it is a bug that affects yt users without a clear fix when it comes up. But I might be OK with handling it on the documentation side: it might be sufficient to add notes to the descriptions of the frontends that use netcdf4 along the lines of "if you are using these frontends and you get mysterious file locking issues, try importing netcdf4 before importing yt".  

Also, I'm not sure exactly how relevant it is, but we do want to make sure we don't regress on this issue https://github.com/yt-project/yt/issues/3987 
> say we make h5py a hard dependency can we not just still use the on_demand_import framework for h5py imports within yt code?

Technically yes, but it sounds like technical debt that might get hard to get rid of down the line.

> I will say that I do think we need to handle this in some way. Even though it's not a yt bug, it is a bug that affects yt users without a clear fix when it comes up. But I might be OK with handling it on the documentation side: it might be sufficient to add notes to the descriptions of the frontends that use netcdf4 along the lines of "if you are using these frontends and you get mysterious file locking issues, try importing netcdf4 before importing yt".

In principle I think I would like that better than the current solution, but I suspect on the rare cases this bug is seen in the wild, no one will reach for yt's doc for a workaround. Best case scenario they might reach for help on slack *and* one of us sees it.

I think we should at least put the responsibility of handling import order on `h5py_imports` instead of `netCDF4_imports` so as to lower the risk `h5py_imports` is replaced by plain `import h5py` by unsuspecting future-us. IMO this may be sufficient to close this issue.


This thread is an interesting and insightful one on h5py-netcdf4 conflicts: https://github.com/Unidata/netcdf4-python/issues/910 

And after more searching and thinking, I actually am coming around to the idea of just removing our forced import order entirely. We only ever saw this manifest in travis-ci, which as you say we're not using. I've not been able to reproduce it locally and I'm not aware of anyone every reporting it. So ya, I think we should remove it and see how it goes! 

But I'm also OK with moving the logic up to h5py_imports. 
I think now is a good time try the bolder approach, I'll give it a try !

Resolving patch:
diff --git a/yt/utilities/on_demand_imports.py b/yt/utilities/on_demand_imports.py
--- a/yt/utilities/on_demand_imports.py
+++ b/yt/utilities/on_demand_imports.py
@@ -83,16 +83,6 @@ def inner(self):
 
 
 class netCDF4_imports(OnDemand):
-    def __init__(self):
-        # this ensures the import ordering between netcdf4 and h5py. If h5py is
-        # imported first, can get file lock errors on some systems (including travis-ci)
-        # so we need to do this before initializing h5py_imports()!
-        # similar to this issue https://github.com/pydata/xarray/issues/2560
-        try:
-            import netCDF4  # noqa F401
-        except ImportError:
-            pass
-
     @safe_import
     def Dataset(self):
         from netCDF4 import Dataset
@@ -299,6 +289,16 @@ def optimize(self):
 
 
 class h5py_imports(OnDemand):
+    def __init__(self):
+        # this ensures the import ordering between netcdf4 and h5py. If h5py is
+        # imported first, can get file lock errors on some systems (including travis-ci)
+        # so we need to do this before initializing h5py_imports()!
+        # similar to this issue https://github.com/pydata/xarray/issues/2560
+        try:
+            import netCDF4  # noqa F401
+        except ImportError:
+            pass
+
     @safe_import
     def File(self):
         from h5py import File


Test patch: