Package: tiledb-netcdf
tiledb_netcdfAn adapter to convertNetCDFfiles toTileDBorZarrarrays.Usage ExamplesHere we'll demonstrate using this library to convert NetCDF files to TileDB and Zarr, and read the results using Iris and Xarray.Converting to TileDBTileDB supports direct interaction with blob storage as provided by public cloud platforms.
Currently this library only supports Azure Blob Storage containers, but we will add AWS support
in a future release. If you'd like that support right now, do raise an issue 👆!1. Create a common data modelUse the data model classNCDataModelto create a data model:fromnctotdbimportNCDataModeldata_model=NCDataModel('/path/to/my/file.nc')data_model.populate()Manually modifying classificationClassifying NetCDF variables is a long way from a precise science, and occasionally the
process may fail to correctly classify a variable. In such a case you can manually modify
the classification processes by using the following instead of callingdata_model.populate():data_model=NCDataModel('/path/to/my/file.nc')my_bespoke_data_var_name='foobarbaz'withdata_model.open_netcdf():data_model.classify_variables()data_model.data_var_names=[my_bespoke_data_var_name]data_model.get_metadata()2. Write the data modelWith a data model created, we can convert the contents of the NetCDF file to a supported
cloud-ready fileformat (one ofTileDBorZarr). We'll demonstrate writing to TileDB here,
with Zarr covered in the next section.As mentioned earlier, we can write either to posix-like filepaths or directly to
an Azure Storage account blob container. Let's cover each of these in turn.2a. FilepathsfromnctotdbimportTileDBWriter# TileDB.tiledb_save_path='/path/to/my_tdb'tiledb_name='my_tiledb'unlimited_dims='z'# Useful if you know you're going to need to append to the `z` dimensionwriter=TileDBWriter(data_model,array_filepath=tiledb_save_path,array_name=tiledb_name,unlimited_dims=unlimited_dims)writer.create_domains()2b. Blob containerSome more setup is needed to interface with an Azure Storage account Blob container.
We need to provide authentication to the Azure Storage account and configure TileDB operations
to work with the Blob container we wish to write to:importtiledb# Azure blob storage definitions.storage_account_name='my_azure_storage_account_name'container='my_blob_container_name'uri=f'azure://{container}'access_key='my_azure_access_key'# TileDB configuration for Azure Blob.cfg=tiledb.Config()cfg['vfs.azure.storage_account_name']=storage_account_namecfg['vfs.azure.storage_account_key']=access_keycfg["vfs.s3.use_multipart_upload"]='false'ctx=tiledb.Ctx(config=cfg)Important!Do not share or publish your Azure Storage account key! You can also
set an environment variable that TileDB will use instead of pasting your account key
into your code.Now we can write to our TileDB array. This is much the same as with posix-like
paths, other than that we must also pass the TileDBCtx(context) object and specify
a container rather than a filepath to save to:writer=TileDBWriter(data_model,container=container,array_name=tiledb_name,unlimited_dims=unlimited_dims,ctx=ctx)writer.create_domains()3. AppendWe can also append the contents of one or more extra NetCDF files along a named dimension.
The extra NetCDF files can be specified either as a list of filepaths or as a list of data model
objects. If filepaths are specified they will be automatically converted to data model objects.append_files=['file1.nc','file2.nc','file3.nc']data_array_name='data'# The name of the data arrays in the TileDB array, typically `data`.writer.append(append_files,unlimited_dims,data_array_name)You can track the progress of append operations by enabling verbose mode:writer.append(append_files,unlimited_dims,data_array_name,verbose=True)If you have a large number of files to append or you simply want the append to complete
faster,tiledb_netcdfcan use dask to parallelise the append operation on a per-file basis.
Assuming you have already set up a dask cluster,my_cluster:client=dask.distributed.Client(my_cluster)logfile="append.log"writer.append(append_files,unlimited_dims,data_array_name,parallel=True,logfile=logfile)Note:it is recommended you also log parallel appends for error tracking, should
anything go wrong during the append process.3a. Scalar AppendOne case of appending needs to be handled differently. This is the case where the datasets to
be appended are scalar along the append dimension. For example, you may wish to append along thetimedimension, but the base dataset and all files to be appended only contain a single
(that is, scalar) time point. In this case a scalar append needs to be carried out.Typically the append algorithm uses the separation between points along the append dimension
to calculate the offsets of all datasets to be appended. With only a single point along the
append dimension this is not possible, so instead you need to also supply a file to the append
call that allows the offset between files to be calculated. To ensure the correct offset is
calculated, this filemustdescribe the next step in the append dimension from the file
originally used to create the TileDB array.The file used to calculate the offset is passed into the append operation using thebaselinekeyword argument. For example:append_files=['file1.nc','file2.nc','file3.nc','file4.nc','file5.nc']data_array_name='data'writer.append(append_files,unlimited_dims,data_array_name,baselines={unlimited_dims:append_files[0]})Note:The file used to calculate the offsets is not appended as well as being used to calculate
the offset. You will need to include the offset file in the append files as well!Note:All appends with a scalar append dimension must be supplied with abaselinefile to calculate the offset, even if an append has already successfully been carried out. You must also specify one baseline file per scalar append dimension, in a dictionary of{"append_dim": baseline_file}.If you try and perform an append along a scalar dimension without providing abaselinefile to calculate the offset, you will encounter an error message:ValueError: Cannot determine scalar step without a baseline dataset.3b. Custom offsets between files being appendedYou may occasionally need to override the offset between successive files being appended, for example to introduce some padding between files, or to handle unexpected short files. This can be done using
theoverride_offsetskwarg toappend. As with specifyingbaselines, you need to pass a dictionary linking the named append dimension to the offset override you wish to apply to that dimension. For example:append_files=['file1.nc','file2.nc','file3_short.nc','file4.nc','file5.nc']expected_dim_len=10data_array_name='data'writer.append(append_files,unlimited_dims,data_array_name,override_offsets={unlimited_dims:expected_dim_len})In this case, the third file is shorter than expected (as helpfully indicated in its filename), and the override offset allows us to pad the append dimension with missing data where the file runs short. We can use thefill_missing_pointsmethod to fill in the gap in the associated dimension coordinate after the append has completed:writer.fill_missing_points(unlimited_dims)Note:you do not have to provide an override offset for every append dimension.4. Read Converted ArraysWe can use theReaderclasses to read our TileDB array with Iris or Xarray:fromnctotdbimportTileDBReader# From a posix-like filepath:tiledb_reader=TileDBReader(tiledb_name,array_filepath=tiledb_save_path)# Or directly from Azure Blob:tiledb_reader=TileDBReader(tiledb_name,container=container,ctx=ctx)# TileDB to Iris.cubes=tiledb_reader.to_iris()# Convert all TileDB arrays to Iris Cubes.cube=tiledb_reader.to_iris('array_name')# Convert a named variable to an Iris Cube.# TileDB to Xarray.dss=tiledb_reader.to_xarray()# Convert all TileDB arrays to Xarray.ds=tiledb_reader.to_xarray('array_name')# Convert a named variable to an Xarray dataset.Converting to ZarrWe can also convert NetCDF files to Zarr using this library, and read these Zarrs
back into Iris and Xarray. A similar set of APIs is provided for Zarr as was provided
for TileDB.1. Create a common data modelThis is exactly the same as for TileDB. The differentiation comes at the next step
when we choose the data representation format we want to use to store the contents of
the NetCDF file represented by the data model.fromnctotdbimportNCDataModelmy_nc_filepath='/path/to/my/file.nc'data_model=NCDataModel(my_nc_file)data_model.populate()2. Write to ZarrWith a data model created we can write the contents of the NetCDF file as exposed via
the data model. Here we write the contents to Zarr:fromnctotdbimportZarrWriterzarr_writer=ZarrWriter(data_model,'/path/to/my_zarr',array_name='my_zarr')zarr_writer.create_zarr()3. AppendWe can also add the contents of other NetCDF files to the Zarr we created, and
extend one of the Zarr's dimensions:my_other_nc_filepath='/path/to/my/other_file.nc'other_data_model=NCDataModel(my_other_nc_file)append_var_name='array_name'append_dim='dimension_name'zarr_writer.append(other_data_model,append_var_name,append_dim)4. Read ZarrFinally we can read the Zarr we created into Iris and Xarray:fromnctotdbimportZarrReaderzarr_reader=ZarrReader('/path/to/my_zarr')cubes=zarr_reader.to_iris()ds=zarr_reader.to_xarray()