Package: h5netcdf
A Python interface for thenetCDF4file-format that reads and writes local or
remote HDF5 files directly viah5pyorh5pyd, without relying on the Unidata
netCDF library.Why h5netcdf?It has one less binary dependency (netCDF C). If you already have h5py
installed, reading netCDF4 with h5netcdf may be much easier than installing
netCDF4-Python.We’ve seen occasional reports of better performance with h5py than
netCDF4-python, though in many cases performance is identical. Forone workflow, h5netcdf was reported to be almost4x fasterthannetCDF4-python.Anecdotally, HDF5 users seem to be unexcited about switching to netCDF –
hopefully this will convince them that netCDF4 is actually quite sane!Finally, side-stepping the netCDF C library (and Cython bindings to it)
gives us an easier way to identify the source of performance issues and
bugs in the netCDF libraries/specification.InstallEnsure you have a recent version of h5py installed (I recommend usingcondaor
the community effortconda-forge).
At least version 3.0 is required. Then:$ pip install h5netcdfOr if you are already using conda:$ conda install h5netcdfNote:From version 1.2. h5netcdf tries to align with anep29-like support policy with regard
to it’s upstream dependencies.Usageh5netcdf has two APIs, a new API and a legacy API. Both interfaces currently
reproduce most of the features of the netCDF interface, with the notable
exception of support for operations that rename or delete existing objects.
We simply haven’t gotten around to implementing this yet. Patches
would be very welcome.New APIThe new API supports direct hierarchical access of variables and groups. Its
design is an adaptation of h5py to the netCDF data model. For example:importh5netcdfimportnumpyasnpwithh5netcdf.File('mydata.nc','w')asf:# set dimensions with a dictionaryf.dimensions={'x':5}# and update them with a dict-like interface# f.dimensions['x'] = 5# f.dimensions.update({'x': 5})v=f.create_variable('hello',('x',),float)v[:]=np.ones(5)# you don't need to create groups first# you also don't need to create dimensions first if you supply data# with the new variablev=f.create_variable('/grouped/data',('y',),data=np.arange(10))# access and modify attributes with a dict-like interfacev.attrs['foo']='bar'# you can access variables and groups directly using a hierarchical# keys like h5pyprint(f['/grouped/data'])# add an unlimited dimensionf.dimensions['z']=None# explicitly resize a dimension and all variables using itf.resize_dimension('z',3)Notes:Automatic resizing of unlimited dimensions with array indexing is not available.Dimensions need to be manually resized withGroup.resize_dimension(dimension, size).Arrays are returned padded withfillvalue(taken from underlying hdf5 dataset) up to
current size of variable’s dimensions. The behaviour is equivalent to netCDF4-python’sDataset.set_auto_mask(False).Legacy APIThe legacy API is designed for compatibility withnetCDF4-python. To use it, importh5netcdf.legacyapi:importh5netcdf.legacyapiasnetCDF4# everything here would also work with this instead:# import netCDF4importnumpyasnpwithnetCDF4.Dataset('mydata.nc','w')asds:ds.createDimension('x',5)v=ds.createVariable('hello',float,('x',))v[:]=np.ones(5)g=ds.createGroup('grouped')g.createDimension('y',10)g.createVariable('data','i8',('y',))v=g['data']v[:]=np.arange(10)v.foo='bar'print(ds.groups['grouped'].variables['data'])The legacy API is designed to be easy to try-out for netCDF4-python users, but it is not an
exact match. Here is an incomplete list of functionality we don’t include:Utility functionschartostring,num2date, etc., that are not directly necessary
for writing netCDF files.h5netcdf variables do not support automatic masking or scaling (e.g., of values matching
the_FillValueattribute). We prefer to leave this functionality to client libraries
(e.g.,xarray), which can implement their exact desired scaling behavior. Nevertheless
arrays are returned padded withfillvalue(taken from underlying hdf5 dataset) up to
current size of variable’s dimensions. The behaviour is equivalent to netCDF4-python’sDataset.set_auto_mask(False).Invalid netCDF filesh5py implements some features that do not (yet) result in valid netCDF files:Data types:BooleansComplex valuesNon-string variable length typesEnum typesReference typesArbitrary filters:Scale-offset filtersBy default[1], h5netcdf will not allow writing files using any of these features,
as files with such features are not readable by other netCDF tools.However, these are still valid HDF5 files. If you don’t care about netCDF
compatibility, you can use these features by settinginvalid_netcdf=Truewhen creating a file:# avoid the .nc extension for non-netcdf filesf=h5netcdf.File('mydata.h5',invalid_netcdf=True)...# works with the legacy API, too, though compression options are not exposedds=h5netcdf.legacyapi.Dataset('mydata.h5',invalid_netcdf=True)...In such cases the_NCPropertiesattribute will not be saved to the file or be removed
from an existing file. A warning will be issued if the file has.nc-extension.Footnotes[1]h5netcdf we will raiseh5netcdf.CompatibilityError.Decoding variable length stringsh5py 3.0 introducednew behaviorfor handling variable length string.
Instead of being automatically decoded with UTF-8 into NumPy arrays ofstr,
they are required as arrays ofbytes.The legacy API preserves the old behavior of h5py (which matches netCDF4),
and automatically decodes strings.The new API matches h5py behavior. Explicitly setdecode_vlen_strings=Truein theh5netcdf.Fileconstructor to opt-in to automatic decoding.Datasets with missing dimension scalesBy default[2]h5netcdf raises aValueErrorif variables with no dimension
scale associated with one of their axes are accessed.
You can setphony_dims='sort'when opening a file to let h5netcdf invent
phony dimensions according tonetCDFbehaviour.# mimic netCDF-behaviour for non-netcdf filesf=h5netcdf.File('mydata.h5',mode='r',phony_dims='sort')...Note, that this iterates once over the whole group-hierarchy. This has affects
on performance in case you rely on laziness of group access.
You can setphony_dims='access'instead to defer phony dimension creation
to group access time. The created phony dimension naming will differ fromnetCDFbehaviour.f=h5netcdf.File('mydata.h5',mode='r',phony_dims='access')...Footnotes[2]Keyword default settingphony_dims=Nonefor backwards compatibility.Track OrderAs of h5netcdf 1.1.0, if h5py 3.7.0 or greater is detected, thetrack_orderparameter is set toTrueenablingorder trackingfor newly created
netCDF4 files. This helps ensure that files created with the h5netcdf library
can be modified by the netCDF4-c and netCDF4-python implementation used in
other software stacks. Since this change should be transparent to most users,
it was made without deprecation.Since track_order is set at creation time, any dataset that was created withtrack_order=False(h5netcdf version 1.0.2 and older except for 0.13.0) will
continue to opened with order tracker disabled.The following describes the behavior of h5netcdf with respect to order tracking
for a few key versions:Version 0.12.0 and earlier, thetrack_orderparameter`order was missing
and thus order tracking was implicitely set toFalse.Version 0.13.0 enabled order tracking by setting the parametertrack_ordertoTrueby default without deprecation.Versions 0.13.1 to 1.0.2 settrack_ordertoFalsedue to a bug in a
core dependency of h5netcdf, h5pyupstream bugwhich was resolved in h5py
3.7.0 with the help of the h5netcdf team.In version 1.1.0, if h5py 3.7.0 or above is detected, thetrack_orderparameter is set toTrueby default.ChangelogChangelogLicense3-clause BSD