How to Create Xarray Datasets. Defining a dataset from scratch | by Luke Gloege, Ph.D. | Towards Data Science

Xarray is an open-source Python package for working with labeled multi-dimensional datasets. It is indispensable when working with NetCDF formatted data, which is common in the Earth science community.
Opening a NetCDF file with xarray is straightforward.
ds = xr.open_dataset("/path/to/file.nc")
However, creating a xarray object that you can save to a NetCDF file requires a little more work. In this post, I will go over how to do just that. I will first go over how to convert a pandas DataFrame to a xarray Dataset and then go over how to create a dataset from scratch.
· 1. Pandas DataFrame to Xarray Dataset ∘ 1.1 How do I add metadata and change variable names? ∘ 1.2 Create a multi-dimensional dataset ∘ 1.3 How do I save my data?· 2. Create Xarray dataset from scratch ∘ 2.1 Dataset with one coordinate ∘ 2.2 Dataset with multiple coordinates ∘ 2.3 Dataset with coordinates matrices ∘ 2.4 Dataset with coordinate vectors and matrices· 3. Final thoughts
Starting with a DataFrame, you can directly convert it to a Dataset.
ds = df.to_xarray()
This can be an excellent starting point since it creates a xarray object for you. In the example below, I create a dataFrame with one variable, y, and one index, x. I then use to_xarray() to make it into a xarray object.
This dataset isn’t formatted very well yet. Here is the ds.info() output:
Ideally, info() should tell you some metadata about the variables and the dataset as a whole. For example, variable attributes could includeunits , which would be important for something like temperature, and global attributes could include the original data source or the creator of the dataset. The goal is to provide the user with some context to the data.
In the next subsection, we will add some metadata to this dataset.
For climate and forecasting (cf), there is a standard convention for metadata. A discussion of this is beyond the scope of this post, but here are two resources for those that are interested:
cfconventions.org
overview of cf conventions
Please note that this post is meant to illustrate a concept and thus I may deviate from conventional standards.
With the object created we can begin to add metadata and change variable/coordinate names.
Dictionary key:value pairs are used to rename variables/coordinates and for adding attributes. Here is the basic syntax:
rename variable: ds.rename({“old_name” : “new_name”})
var attribute:ds[“var”].attrs = {“attribute_name”:”attribute_value”}
global attribute: ds.attrs = {“attribute_name”:”attribute_value”}
The code snippet below first renames the coordinates and variables and then creates variable and global attributes.
(Note: xarray does not have an inplace=True option like pandas.)
Here is the ds.info() output after adding some metadata
This is now a nicely formatted dataset. The user can quickly understand what this dataset represents and who created it. In section 2 we will create this same dataset from scratch.
For multiple dimensions, simply include a list of dimensions in set_index(). Then you can add metadata and change variables the same way as before
Once you are satisfied with your changes, you can save them to a NetCDF file:
ds.to_netcdf('/save/to/this/path/file.nc')
In this next section, I will go over creating a dataset from scratch. This is useful if you have a large NumPy array or list you want to save as a NetCDF.
The following syntax is used to create a dataset with xarray:
ds = xr.Dataset(data_vars, coords, attrs)
A complete dataset consists of three dictionaries:
data_vars : The key is the variable name and value is a tuple consisting of(dimensions, data, variable_attributes)- dimensions → list of names- data → the data can be in any format (Pandas, Numpy, list, etc.)- variable_attributes → optional dictionary of attributes
coords : this defines the coordinates and their attributes
attrs: optional dictionary of global attributes
Attributes are optional but strongly encouraged.
The following uses xr.Dataset() to create the same dataset as the previous example.
data_vars: defines the variable velocity with one dimension, time, and includes two attributes, units an long_name.
coords: defines the time coordinate, whose dimension is also named time.
attrs: defines the global attributes, creation_data, author, and email
If your data is multi-dimensional it will typically have multiple coordinates.
Here, I create a dataset where the variable has two dimensions, x and y , and two coordiantes, lat and lon.
In the previous example, the coordinate and dimension were the same name. This example illustrates that they do not need to be the same.
In the previous example, each coordinate was represented by a vector. However, there are instances where it makes sense to represent the coordinate as a matrix. This is common with Earth system models on a non-regular grid.
Let’s take the previous example and turn each coordinate into a matrix using np.meshgrid()
np.meshgrid creates coordinate matrices from coordinate vectors. The row mesh grid repeats the rows across the length of cols , while col mesh grid repeat cols across the length of rows. Here is each mesh grid.
Now let’s create a dataset using the coordinates represented by matrices.
This represents the same data as before, the only difference is each coordinate is represented as a matrix instead of a vector. This is meant to illustrate a concept. In most cases representing the coordinate as a vector makes more sense.
Coordinates can be a mix of matrices and vectors. You just need to make sure that all the dimensions in your variables are accounted for in your coordinates.
Let’s create a dataset with three dimensions, time, lat, and lon. The spatial dimensions (lat, lon) will be each be represented by matrices and time will be a vector.
Hopefully, this helps you create your own datasets and augment existing ones.
If your data is small or already in a Pandas dataframe. Then using to_xarray() is the easiest way to create the initial object. In any other instance, it may be easier to create the dataset from scratch.
I covered two ways to represent coordinates: vectors or matrices. My advice is to keep the coordinates as a vector whenever possible. Using matrices to store coordinate information is common if the data is not on a regular grid.
If you get an error creating a dataset, the first thing to check is the dimensions. It is easy to put the dimensions out of order.
My other piece of advice is that even though attributes are optional when creating a dataset, it is good practice to always include them so you and other people can easily interpret the dataset. I prefer to add attributes after creating the dataset object. This is a personal preference that I think improves readability.
Thanks for reading and I am happy to help troubleshoot any issues