Package: ecml-tools
ecml-toolsA package to hold various functions to support training of ML models on ECMWF data.InstallationThis package will be a collection of tools, with their own dependencies. In order to not
install unnecessary dependencies, the package is split in parts.For handling datasets, you will need to install thedataextra:pipinstallecml-tools[data]For provenance tracking, you will need to install theprovenanceextra:pipinstallecml-tools[provenance]To install everything:pipinstallecml-tools[all]DatasetsAdatasetwraps azarrfile that follows the format used by ECMWF to train its machine learning models.fromecml_tools.dataimportopen_datasetds=open_dataset("aifs-ea-an-oper-0001-mars-o96-1979-2022-1h-v2")The dataset can be passed as a path or URL to azarrfile, or as a name. In the later case, the package will use the entryzarr_rootof~/.ecml-toolfile to create the full path or URL:zarr_root:/path_or_url/to/the/zarrsAttributes of a datasetAs the underlyingzarr, thedatasetis an iterable:fromecml_tools.dataimportopen_datasetds=open_dataset("aifs-ea-an-oper-0001-mars-o96-1979-2022-1h-v2")# Print the number of rows (i.e. dates):print(len(ds))# Iterate throw the rows,forrowinds:print(row)# or access a item directly.print(row[10])# You can retrieve the shape of the dataset,print(ds.shape)# the list of variables,print(ds.variables)# the mapping between variable names and columns indextwo_t_index=ds.name_to_index["2t"]row=ds[10]print("2t",row[two_t_index])# Get the list of dates (as NumPy datetime64)print(ds.dates)# The number of hours between consecutive datesprint(ds.frequency)# The resolution of the underlying gridprint(ds.resolution)# The list of latitudes of the data values (NumPy array)print(ds.latitudes)# The same for longitudesprint(ds.longitudes)# And the statiticsprint(ds.statistics)The statistics is a dictionary of NumPy vectors following the order of the variables:{"mean":...,"stdev":...,"minimum":...,"maximum":...,}To get the statistics for2t:two_t_index=ds.name_to_index["2t"]stats=ds.statisticsprint("Average 2t",stats["mean"][two_t_index])Subsetting datasetsYou can create a view on thezarrfile that selects a subset of dates.Changing the frequencyfromecml_tools.dataimportopen_datasetds=open_dataset("aifs-ea-an-oper-0001-mars-o96-1979-2022-1h-v2",freqency="12h")Thefrequencyparameter can be a integer (in hours) or a string following with the suffixh(hours) ord(days).Selecting yearsYou can select ranges of years using thestartandendkeywords:fromecml_tools.dataimportopen_datasettraining=open_dataset("aifs-ea-an-oper-0001-mars-o96-1979-2022-1h-v2",start=1979,end=2020)test=open_dataset("aifs-ea-an-oper-0001-mars-o96-1979-2022-1h-v2"start=2021,end=2022)The selection includes all the dates of theendyears.Selecting more precise rangesYou can select a few months, or even a few days:fromecml_tools.dataimportopen_datasettraining=open_dataset("aifs-ea-an-oper-0001-mars-o96-1979-2022-1h-v2",start=202306,end=202308)test=open_dataset("aifs-ea-an-oper-0001-mars-o96-1979-2022-1h-v2"start=20200301,end=20200410)The following are equivalent way of describingstartorend:2020and"2020"202306,"202306"and"2023-06"20200301,"20200301"and"2020-03-01"You can omit eitherstartorend. In that case the first and last date of the dataset will be used respectively.Combining bothYou can combine both subsetting methods:fromecml_tools.dataimportopen_datasettraining=open_dataset("aifs-ea-an-oper-0001-mars-o96-1979-2022-1h-v2",start=1979,end=2020,frequency="6h")Combining datasetsYou can create a virtual dataset by combining two or morezarrfiles.fromecml_tools.dataimportopen_datasetds=open_dataset("dataset-1","dataset-2","dataset-3",...)When given a list ofzarrfiles, the package will automatically work out if the files can beconcatenatedorjoinedby looking at the range of dates covered by each files.If the dates are different, the files are concatenated. If the dates are the same, the files are joined. See below for more information.Concatenating datasetsYou can concatenate two or more datasets along the dates dimension. The package will check that all datasets are compatible (same resolution, same variables, etc.). Currently, the datasets must be given in chronological order with no gaps between them.fromecml_tools.dataimportopen_datasetds=open_dataset("aifs-ea-an-oper-0001-mars-o96-1940-1978-1h-v2","aifs-ea-an-oper-0001-mars-o96-1979-2022-1h-v2")Please note that you can pass more than twozarrfiles to the function.NOTE:When concatenating file, the statistics are not recomputed; it is the statistics of first file that are returned to the user.Joining datasetsYou can join two datasets that have the same dates, combining their variables.fromecml_tools.dataimportopen_datasetds=open_dataset("aifs-ea-an-oper-0001-mars-o96-1979-2022-1h-v2","some-extra-parameters-from-another-source-o96-1979-2022-1h-v2",)If a variable is present in more that one file, that last occurrence of that variable will be used, and will be at the position of the first occurrence of that name.Please note that you can join more than twozarrfiles.Selection, ordering and renaming of variablesYou can select a subset of variables when opening azarrfile. If you pass alist, the variables are ordered according the that list. If you pass aset, the order of the file is preserved.fromecml_tools.dataimportopen_dataset# Select '2t' and 'tp' in that orderds=open_dataset("aifs-ea-an-oper-0001-mars-o96-1979-2022-1h-v2",select=["2t","tp"],)# Select '2t' and 'tp', but preserve the order in which they are in the fileds=open_dataset("aifs-ea-an-oper-0001-mars-o96-1979-2022-1h-v2",select={"2t","tp"},)You can also drop some variables:fromecml_tools.dataimportopen_datasetds=open_dataset("aifs-ea-an-oper-0001-mars-o96-1979-2022-1h-v2",drop=["10u","10v"],)and reorder them:fromecml_tools.dataimportopen_dataset# ... using a listds=open_dataset("aifs-ea-an-oper-0001-mars-o96-1979-2022-1h-v2",reorder=["2t","msl","sp","10u","10v"],)# ... or using a dictionnaryds=open_dataset("aifs-ea-an-oper-0001-mars-o96-1979-2022-1h-v2",reorder={"2t":0,"msl":1,"sp":2,"10u":3,"10v":4},)You can also rename variables:fromecml_tools.dataimportopen_datasetds=open_dataset("aifs-ea-an-oper-0001-mars-o96-1979-2022-1h-v2",rename={"2t":"t2m"},)This will be useful when your join datasets and do not want variables from one dataset to override the ones from the other.Using all optionsYou can combine all of the above:fromecml_tools.dataimportopen_datasetds=open_dataset("aifs-ea-an-oper-0001-mars-o96-1979-2022-1h-v2","some-extra-parameters-from-another-source-o96-1979-2022-1h-v2",start=2000,end=2001,frequency="12h",select={"2t","2d"},...)Building a dataset from a configurationIn practice, you will be building datasets from a configuration file, such as a YAML file:importyamlfromecml_tools.dataimportopen_datasetwithopen("config.yaml")asf:config=yaml.safe_load(f)training=open_dataset(config["training"])test=open_dataset(config["test"])This is possible becauseopen_datasetcan be build from simple lists and dictionaries:# From a stringds=open_dataset("aifs-ea-an-oper-0001-mars-o96-1979-2022-1h-v2")# From a list of stringsds=open_dataset(["aifs-ea-an-oper-0001-mars-o96-1979-2022-1h-v2","aifs-ea-an-oper-0001-mars-o96-2023-2023-1h-v2",])# From a dictionnaryds=open_dataset({"dataset":"aifs-ea-an-oper-0001-mars-o96-1979-2022-1h-v2","frequency":"6h",})# From a list of dictionnaryds=open_dataset([{"dataset":"aifs-ea-an-oper-0001-mars-o96-1979-2022-1h-v2","frequency":"6h",},{"dataset":"some-extra-parameters-from-another-source-o96-1979-2022-1h-v2","frequency":"6h","select":["sst","cape"],},])# And even deeper constructsds=open_dataset([{"dataset":"aifs-ea-an-oper-0001-mars-o96-1979-2022-1h-v2","frequency":"6h",},{"dataset":[{"dataset":"aifs-od-an-oper-8888-mars-o96-1979-2022-6h-v2","drop":["ws"],},{"dataset":"aifs-od-an-oper-9999-mars-o96-1979-2022-6h-v2","select":["ws"],},],"frequency":"6h","select":["sst","cape"],},])