Package: ncdata
ncdataGeneric NetCDF data in Python.Provides fast data exchange between analysis packages, and full control of storage
formatting.Especially : Ncdataexchanges data between Xarray and Irisas efficently as possible"lossless, copy-free and lazy-preserving".This enables the user to freely mix+match operations from both projects, getting the
"best of both worlds".import xarrayimport ncdata.iris_xarray as nciimport iris.quickplot as qpltds = xarray.open_dataset(filepath)ds_resample = ds.rolling(time=3).mean()cubes = nci.cubes_from_xarray(ds_resample)temp_cube = cubes.extract_cube("air_temperature")qplt.contourf(temp_cube[0])ContentsMotivationPrimary UseSecondary UsesPrinciplesWorking Usage ExamplesAPI documentationInstallationProject StatusCode stabilityIris and Xarray version compatibilityCurrent LimitationsKnown ProblemsReferencesDeveloper NotesMotivationPrimary UseFast and efficient translation of data between Xarray and Iris objects.This allows the user to mix+match features from either package in code.For example:fromncdata.iris_xarrayimportcubes_to_xarray,cubes_from_xarray# Apply Iris regridder to xarray datadataset=xarray.open_dataset("file1.nc",chunks="auto")(cube,)=cubes_from_xarray(dataset)cube2=cube.regrid(grid_cube,iris.analysis.PointInCell)dataset2=cubes_to_xarray(cube2)# Apply Xarray statistic to Iris datacubes=iris.load("file1.nc")dataset=cubes_to_xarray(cubes)dataset2=dataset.group_by("time.dayofyear").argmin()cubes2=cubes_from_xarray(dataset2)data conversion is equivalent to writing to a file with one library, and reading it
back with the other .... except that no actual files are writtenboth real (numpy) and lazy (dask) variable data arrays are transferred directly,
without copying or computingSecondary UsesExact control of file formattingNcdata can also be used as a transfer layer between Iris or Xarray file i/o and the
exact format of data stored in files.I.E. adjustments can be made to file data before loading it into Iris/Xarray; or
Iris/Xarray saved output can be adjusted before writing to a file.This allows the user to workaround any package limitations in controlling storage
aspects such as : data chunking; reserved attributes; missing-value processing; or
dimension control.For example:fromncdata.xarrayimportfrom_xarrayfromncdata.irisimportto_irisfromncdata.netcdf4importto_nc4,from_nc4# Rename a dimension in xarray outputdataset=xr.open_dataset("file1.nc")xr_ncdata=from_xarray(dataset)dim=xr_ncdata.dimensions.pop("dim0")dim.name="newdim"xr_ncdata.dimensions["newdim"]=dimforvarinxr_ncdata.variables.values():var.dimensions=["newdim"ifdim=="dim0"elsedimfordiminvar.dimensions]to_nc4(ncdata,"file_2a.nc")# Fix chunking in Iris inputncdata=from_nc4("file1.nc")forvarinncdata.variables:# custom chunking() mimics the file chunks we wantvar.chunking=lambda:(100.0e6ifdim=="dim0"else-1fordiminvar.dimensions)cubes=to_iris(ncdata)Manipulation of datancdata can also be used for data extraction and modification, similar to the scope of
CDO and NCO command-line operators but without file operations.However, this type of usage is as yet still undeveloped :  There is no inbuilt support
for data consistency checking, or obviously useful operations such as indexing by
dimension.
This could be added in future, but it is also true that many such operations (like
indexing) may be better done using Iris/Xarray.Principlesncdata represents NetCDF data as Python objectsncdata objects can be freely manipulated, independent of any data filencdata variables can contain either real (numpy) or lazy (Dask) arraysncdata can be losslessly converted to and from actual NetCDF filesIris or Xarray objects can be converted to and from ncdata, in the same way that
they are read from and saved to NetCDF filestranslationbetween Xarray and Iris is based on conversion to ncdata, which
is in turn equivalent to file i/othus, Iris/Xarray translation is equivalent tosavingfrom one
package into a file, thenloadingthe file in the other packagencdata exchanges variable data directly with Iris/Xarray, with no copying of real
data or computing of lazy datancdata exchanges lazy arrays with files using Dask 'streaming', thus allowing
transfer of arrays larger than memoryCode Examplesmostly TBDproof-of-concept script fornetCDF4 file i/oproof-of-concept script foriris-xarray conversionsAPI documentationsee theReadTheDocs buildInstallationInstall from conda-forge with condaconda install -c conda-forge ncdataOr from PyPI with pippip install ncdataProject StatusCode StabilityWe intend to followPEP 440or (older)SemVerversioning principles.Release version is at"v0.1".This is a first complete implementation, with functional operational of all public APIs.The code is however still experimental, and APIs are not stable (hence no major version yet).Iris and Xarray CompatibilityC.I. tests GitHub PRs and merges, against latest releases of Iris and Xarraycompatible with iris >= v3.7.0see :support added in v3.7.0Known limitationsUnsupported features :not planneduser-defined datatypes are not supportedthis includes compound and variable-length typesUnsupported features :planned for future releasegroups (not yet fully supported ?)file output chunking controlKnown problemsAs-of v0.1in conversion from iris cubes withfrom_iris,
use of anunlimited_dimskey currently causes an exceptionhttps://github.com/pp-mo/ncdata/issues/43ReferencesIris issue :https://github.com/SciTools/iris/issues/4994planning presentation :https://github.com/SciTools/iris/files/10499677/Xarray-Iris.bridge.proposal.--.NcData.pdfin-Iris code workings :https://github.com/pp-mo/iris/pull/75Developer NotesDocumentation buildFor a full docs-build, a simplemake htmlwill do for now.Thedocs/Makefilewipes the API docs and invokes sphinx-apidoc for a full rebuildResults are then available atdocs/_build/html/index.htmlThe above is just forlocal testingif required :
We have automatic builds for releases and PRs viaReadTheDocsRelease actionsCut a release on GitHub : this triggers a new docs version onReadTheDocsBuild the distributionif needed, getbuildrunpython -m buildPush to PyPIif needed, gettwinerunpython -m twine --repository testpypi upload dist/*this uploads to TestPyPIif that checks OK,remove--repository testpypiand repeat--> uploads to "real" PyPIcheck thatpip install ncdatacan now find the new versionUpdate conda to source the new version from PyPIcreate a PR on thencdata feedstockupdate :version numberSHANote : thePyPI referencewill normally look after itselfAlso : make any required changes todependencies-- normallyno change requiredget PR merged ; wait a few hours ; check the new version appears inconda search ncdata