Read ERA5 Directly into Memory with Python | by Luke Gloege, Ph.D. | Towards Data Science

ERA5 is the fifth generation climate reanalysis produced by the European Centre for Medium-Range Weather Forecasts (ECMWF). The product provides hourly output for many atmospheric, land-surface, and ocean-state parameters together with estimates of uncertainty. However, unless you are working in the cloud, storing raw hourly output can take up a lot of space.
This post outlines how to install the climate data store (CDS) API to read and analyze ERA5 output in Python.
The CDS API allows access to the ERA5 family of datasets, which includes ERA5 and, the land-only complement, ERA5-land. There are four main sub-sets to ERA5: hourly and monthly products each on pressure levels (including upper-air fields) and single levels (atmospheric, ocean, and land surface quantities). ERA5-land only includes single levels.
The four ERA5 subsets
ERA5 hourly data on single levels
ERA5 monthly averaged data on single levels
ERA5 hourly data on pressure levels
ERA5 monthly averaged data on pressure levels
The two ERA5-land subsets
ERA5-land hourly data
ERA5-land monthly data
The ERA5 reanalysis is described at length in this manuscript.
To begin using the CDS API the first thing you need to do is create an account with Copernicus and store your unique API key in a dot file. Here are the steps to do this:
Create an account with Copernicus by signing up here.Once you have an account, sign in to your Copercius account here and note the UID and API key at the bottom of the page.Paste the code snippet below into your terminal, replacing <UID> and <API key> with those from step 2:
Create an account with Copernicus by signing up here.
Once you have an account, sign in to your Copercius account here and note the UID and API key at the bottom of the page.
Paste the code snippet below into your terminal, replacing <UID> and <API key> with those from step 2:
{  echo 'url: https://cds.climate.copernicus.eu/api/v2'  echo 'key: UID:API_KEY'  echo 'verify: 0'} > ~/.cdsapirc
The above command creates the file ~/.cdsapirc with your API key, which is necessary to use the CDS API. As a sanity check, use more ~/.cdsapirc to ensure everything appears correct. A correct file will look similar to this:
url: https://cds.climate.copernicus.eu/api/v2key: 12345:a99b9c9d-9e99-9999-9999-fg99h9i99j9kverify: 0
Do not copy the above into your ~/.cdsapirc file, this is just for illustration.
Once you have your API key in ~/.cdsapirc you can install the cdsapi client via conda or pip:
pip install cdsapiorconda install cdsapi
Once installed, make sure you can import and start the client in a Python script without any errors.
import cdsapicds = cdsapi.Client()
The code below shows how to request monthly mean temperatures on a specified pressure level over a range of dates. Output is stored in memory as an xarray dataset, ds. The code below also allows the output to be saved to disk, this action can be toggled on or off using the download_flag:
import cdsapiimport xarray as xrfrom urllib.request import urlopen# start the clientcds = cdsapi.Client()# dataset you want to readdataset = "reanalysis-era5-pressure-levels-monthly-means"# flag to download datadownload_flag = False# api parameters params = {    "format": "netcdf",    "product_type": "monthly_averaged_reanalysis",    "variable": "temperature",    "pressure_level": '1000',    'year':['2019','2020'],    'month':['01','02','03'],    "time": "00:00",     "grid": [1.0, 1.0],    "area": [90, -180, -90, 180],    }# retrieves the path to the filefl = cds.retrieve(dataset, params)# download the file if download_flag:    fl.download("./output.nc")# load into memorywith urlopen(fl.location) as f:    ds = xr.open_dataset(f.read())
Here is an example of a more involved request where I am requesting January, February, and March for 2020. In addition, I am requesting the first day for each month at noon, 12:00. I am requesting data on a quarter degree grid and just area around the continental United States: [49.38, -124.67, 25.84, -66.95. Note that the area parameter specifies a bounding box with coordinates: [north, west, south, east].
import cdsapiimport xarray as xrfrom urllib.request import urlopenc = cdsapi.Client()# dataset to readdataset = 'reanalysis-era5-pressure-levels'# flag to download datadownload_flag = False# api parameters params = {    'format': 'netcdf',    'product_type': 'reanalysis',    'variable': 'temperature',    'pressure_level':'1000',    'year':['2020'],    'month':['01','02','03'],    'day': ['01'],    'time': ['12:00'],    'grid': [0.25, 0.25],    'area': [49.38, -124.67, 25.84, -66.95],    }# retrieves the path to the filefl = c.retrieve(dataset, params)# download the file if download_flag:    fl.download("./output.nc")# load into memorywith urlopen(fl.location) as f:    ds = xr.open_dataset(f.read())
In place of the year, month, day, and time parameters, you could alternatively specify a list of dates:
'date':['2020-01-01 12:00', '2020-02-01 12:00', '2020-03-01 12:00'],
or you could specify a date and time like this:
'date':['2020-01-01', '2020-02-01', '2020-03-01'],'time: '12:00',
The above snippets each result in the same request, illustrating how flexible the API is. These requests are very specific and most of the time you will want to request a continuous range.
There are a few ways to do this. The easiest is to separate the dates you want by a slash, which specifies the start and end date for the requested range:
'date':'2020-01-01 12:00/2020-03-01 12:00'
Another option is to use pandas date_range. However, note that the input to the API has to be a list of strings. Here is an example using pandas:
‘date’: list(pd.date_range(‘2020–01–01 12:00',’2020–03–01 12:00', freq=’M’).strftime(‘%Y-%m-%d %H:%M’)),
Unfortunately, you can not use the date parameter when requesting monthly averages, it only works for the high temporal resolution data. For monthly averaged data you will have to use the year and month parameters, which also accept the slash notation:start/end.
The cdsapi is great for complex requests and is very flexible, but sometimes you get an error message that can be hard to diagnose.
no data is available within your requested subset. Request returned no data is a common error message, and it isn’t very helpful. This usually indicates that a keyword value is outside the available range. For example, only a specific set of pressure levels can be requested. Also, the time parameter has to be to the nearest hour using a 24-hour clock. Here are the common errors I see and how to fix them:
Pressure level must be contained in: [1, 2, 3, 5, 7, 10, 20, 30, 50, 70, range(100, 1000, 25)], where range(100,1000,25) indicates 100 to 1000 by increments of 25. If you get an error, check that the pressure you are requesting is contained in this list.
Time must be to the nearest hour on a 24-hour clock: 00:00, 01:00, ..., 23:00. If you get an error, check that the time is to the nearest hour. If you are requesting monthly means, make sure the time parameter is set to ‘00:00’.
I am unaware of any documentation for the CDS API, which can make it difficult to use since the allowable keywords and variable names vary depending on the dataset. The following links will be helpful:
Allowable keyword parameters for each dataset are here.
Variables available are listed in tables here. You supply what is listed in thevariable name in CDS column.
You could also look at the example script in the table here.
Another option is to use the web interface to make a selection and look at “show API request” at the bottom. Here is the form for “monthly averaged on pressure levels.” Other datasets can be found here. Click on the dataset you want to access and then the “Download data” tab to make your selection.
Once you understand how it works, you realize the CDS API is not very intimidating and is very powerful. I wrote a function to abstract away some of the nuances I described. This function is open source, so feel free to copy and modify it as you see fit.
Here is an example using the get_era5() function to create the image below:
Hats off to the ECMWF team for releasing the cdsapi and making the ERA5 reanalysis products easily accessible. The API has clear syntax and the ability to re-grid the output on the server-side is a godsend. ECMWF is also working on a package that aims to simplify access to these datasets even more so we can focus more on science and less on coding. At the time of writing it is still under development, but I encourage you to keep an eye on climetlab. However, climetlab does download the data behind the scenes. Hopefully, the ability to read output into memory without downloading it will be a feature in climetlab. Nevertheless, we can always revert to using the CDS API to achieve this.
Hopefully, this post helps you analyze ERA5 output and streamlines your data pipelines. Happy coding!
medium.com
medium.com
medium.com
Thank you for taking the time to read my post. This function started from conversations with my colleague, Jashvina Devadoss, who was working on a similar function.
Please comment below if you have questions, suggestions, or found this post useful.
Thank you for reading and supporting Medium writers