Package: netcdf-flattener
netcdf-flattenerFlatten netCDF files while preserving references as described in the CF Conventions 1.8, chapter 2.7.UsageThe flattener takes as input and output NetCDFDatasetobjects, which the user can create or open from".nc"files
using the netCDF4 API. To flatten the Dataset named"input_dataset"into a Dataset named"output_dataset", use the
following command. In most cases,"output_dataset"will be an empty Dataset.import netcdf_flattener
netcdf_flattener.flatten(input_dataset, output_dataset)By default, the flattener is in strict mode and returns an exception if a an internal reference from a variable
attribute to a dimension or variable could not be resolved. To use the lax mode that continues the flattening process
with warning, specify thelax_modeparameter:netcdf_flattener.flatten(input_dataset, output_dataset, lax_mode=True)For copying variables that would otherwise be larger than the available memory, thecopy_slicesparameter allows to
specify slices to be used when copying the variable. They are specified per variable in a dictionary. The slicing shape
is eitherNonefor using a default slice value, or a custom slicing shape in the form of a tuple of the same dimension
as the variable. If a variable from the Dataset is not contained in the dict, it will not be sliced and copied normally.Slice shapes should be small enough to fit in memory, but not too small larges loops on small slice can degrade
performances drastically. Typically, slices of size in the order of 10^6 to 10^8 are suitable.netcdf_flattener.flatten(input_dataset, output_dataset, copy_slices={"/grp1/var1": (1000,1000,500,), "/grp1/var3": None})LimitationsWhen a CF coordinate variable in the input dataset is in a different
group to its corresponding dimension, the same variable in the output
flattened dataset willno longerbe a CF coordinate variable, as its
name will be prefixed with a different group identifier than its
dimension. In such cases, it is up to the user to apply theproximal
and lateral search alogrithms,
in conjunction with the mappings defined in theflattener_name_mapping_variablesandflattener_name_mapping_dimensionsglobal attributes, to find which
netCDF variables areacting asCF coordinate variables in the
flattened dataset.For example, if an input dataset has dimensionlatin the root
group and coordinate variablelat(lat)in group/grp1, then
the flattened dataset will contain dimensionlatand variablegrp1__lat(lat), both in its root group. In this case, theflattener_name_mapping_variablesglobal attribute of the flattened
dataset will contain the mapping"grp1__lat: /grp1/lat"and theflattener_name_mapping_dimensionsglobal attribute will contain
the mapping"lat: /lat".DeploymentFrom PyPinetCDF-flatteneris in installable withpip, for example:pip install netcdf-flattenerFrom sourceInstall the build dependencies:python3 -m pip install --upgrade pip setuptools wheelDownload the source code fromhttps://gitlab.eumetsat.int/open-source/netcdf-flattenerand compile
the wheel file, by running the following command from the repository root:python3 setup.py bdist_wheelInstall the wheel file usingpip:python3 -m pip install dist/netcdf_flattener-*.whlSupportQuestions and issues should be raised at the issue tracker in the
canonical source code repository:https://gitlab.eumetsat.int/open-source/netcdf-flattenerAutomated testingDependenciesRunning the tests requires having the NetCDF4 libraries installed (ncdump and ncgen applications are required). You can
install them either using your package manager, orbuild them from the source.On CentOS:sudo yum install netcdfInstall Pytest:python3 -m pip install pytestAll other dependencies are managed by pip and use OSI-approved licenses.Run the testsRun Pytest from the root of the repository:python3 -m pytestDocumentationA Sphinx project is provided to generate the HTML documentation from the code.Install Sphinx:python3 -m pip install sphinxFrom the "doc" folder, build the documentation:cd doc
sphinx-build -b html . buildThe entry point to the documentation is the doc/build/index.html file.LicenseThis code is under Apache 2.0 License. SeeLICENSEfor the full license text.AuthorsSeeAUTHORSfor details.