Package: climetlab-s2s-ai-challenge
S2S AI challenge DatasetsThis is aclimetlabplugin for Sub-seasonal to Seasonal (S2S) Artificial Intelligence Challenge:https://s2s-ai-challenge.github.io/.In this README is a description of how to get the data for the S2S AI challenge. Here is a more generaldescription of the S2S data. The data used for the S2S AI challenge is a subset of the S2S data library. More detail can be found athttps://confluence.ecmwf.int/display/S2Sandhttps://confluence.ecmwf.int/display/S2S/Parameters.There are several ways to use the datasets. Either by direct download (wget,curl,browser) forGRIBandNetCDFformats; or using theclimetlabpython package with this addon, forGRIBandNetCDFandzarrformats.zarris a cloud-friendly experimental data format and supports dowloading only the part of the data that is required. It has been designed to work better than classical format on a cloud environment (experimental).Installationpip install -U climetlab climetlab_s2s_ai_challengeAPIimportclimetlabUseclimetlab.load_dataset('s2s-ai-challenge-{datasetname}')with the following keywords:datasetname: name of the dataset, seedataset descriptionparameter:variable, seehindcast input for the different modelsorigin: name of the model [ecmwf,eccc,ncep] or modelling center [ecmf,cwao,kwbc]. Provideoriginonly fortraining/test-input/hindcast/forecast-input.date:YYYYMMDDis the date of the 2020 forecast fortest-input/forecast-input. The same dates are required for the on-the-flytraining-input/hindcast-inputbut returns the multi-year hindcast for the givenMMDDdate. Please provideint,str,np.datetimeor list of the former in formatYYYYMMDDD. Providing nodatekeyword downloads all dates.format: data format, choose from [netcdf(always available),grb(only forinput-*),zarr(experimental)]CoordinatesOverview of the time-related coordinates.nameCF conventionstandard_namedescriptioncommentforecast_timeforecast_reference_timeThe forecast reference time in NWP is the "data time", the time of the analysis from which the forecast was made. It is not the time for which the forecast is valid.lead_timeforecast_periodForecast period is the time interval between the forecast reference time and the validity time.valid_timetimetime for which the forecast is validforecast_time+lead_timeAll datasets are on a global 1.5 degree grid.Parameterparameterdescribes the variable to download. The most important two variables for thes2s-ai-challengeare the two target variablest2mandtp:parameterlong_namestandard_nameunitdescription & aggregation typeweek 3-4 aggregationweek 5-6 aggregationlink to sourcet2m2m temperatureair_temperatureKTemperature at 2m height averaged for the date givenaverage [day 14, day 27]average [day 28, day 41]model,observationstptotal precipitationprecipitation_amountkg m-2Total precipitation accumulated fromforecast_timeuntil includingvalid_time, e.g.lead_time = 1 daysaccumulates precipitation_fluxprfrom 6-hourly steps 0,6,12,18 at dateforecast_timeday 28 minus day 14day 42 minus day 28modelprprecipitation fluxprecipitation_fluxkg m-2Precipitation accumulated for the date givenusetpusetpobservationsGiven the different nature of the parameters, withforecast_timeJan 2nd 2020,tplead_time=1 daysandt2mlead_time=0 daysdescribe both the weather of Jan 2nd 2020, astpis aggregatedprfrom Jan 2nd 00:00 to Jan 3rd 00:00 andt2mas the average of Jan 2nd. Furthermoretpis aggregated sinceforecast_time, i.e.tplead_time=5 daysispraggregated from Jan 2nd 00:00 to Jan 7th 00:00.For the remaining variable descriptions, seeECWMF S2S description.Datasets descriptionThere are four datasets provided for this challenge. As we are aiming at bringing together the two communities of Machine Learning and Weather Prediction, they have been aliases to use both two points of views:MLNWPtraining-inputhindcast-inputTraining  dataset (input for training the ML models)training-output-referencehindcast-like-observationsTraining dataset (output for training the ML models)training-output-benchmarkhindcast-benchmarkBenchmark output (on the training dataset)test-inputforecast-inputTest dataset (DO NOT use for training)test-output-referenceforecast-like-observationsTest dataset (DO NOT use)test-output-benchmarkforecast-benchmarkBenchmark output (on the test dataset)observationsobservationsObservations withtimedimensionOverfittingis always an potential issue when using ML algorithms. To address this, the data is usually split into three datasets :training,validationandtest.
This terminology has lead tosome confusion in the past.
Splitting thehindcast-input(training-input) dataset between training and validation is standard way and should be decided carefully.Theforecast-input(test-input) must not be used as a validation dataset: it must not be used to tune the hyperparameters or make decision about the ML model.Hindcast inputThese data are hindcast data. This is used as the input for training the ML models.Thehindcast-input(training-input) dataset consists of data from three differentmodels/centers:center namemodel nameecmwfecmfeccccwaoncepkwbcUse eitherorigin="ecmf"(model name) ororigin="ecmwf"(center name).This dataset is available asformat:grib,netcdf.ECMWF hindcast dataforecast_time: from 2000/01/02 to 2019/12/31, corresponding to the weekly Thurdays in 2020.lead_time: 0 to 46 daysvalid_time(forecast_time+lead_time): from 2000/01/02 to 2020/02/13availables parameters :t2m(2t)/siconc(ci)/gh/lsm/msl/q/rsn/sm100/sm20/sp/sst/st100/st20/t/tcc/tcw/tp/ttr/u/v(differing name inMARS database)ECCC hindcast dataforecast_time: from 2000/01/02 to 2019/12/31, corresponding to the weekly Thurdays in 2020.lead_time: 1 to 32 daysvalid_time(forecast_time + lead_time): from  2000/01/03 to 2020/02/01availables parameters:t2m(2t)/siconc(ci)/gh/lsm/msl/q/rsn/sp/sst/t/tcc/tcw/tp/ttr/u/v(differing name inMARS database)parameters not available:sm20/sm100/st20/st100NCEP hindcast dataforecast_time: from 1999/01/07 to 2010/12/30, corresponding to the weekly Thurdays in2010, see Vitart et al. 2017, NCEP hindcast dates differ from ECMWF and ECCC hindcastslead_time: 1 to 44 daysvalid_time(forecast_time+lead_time): from 1999/01/07 to 2011/02/11availables parameters:t2m(2t)/siconc(ci)/gh/lsm/msl/q/sm100/sm20/sp/sst/st100/st20/t/tcc/tcw/tp/ttr/u/v(differing name inMARS database)parameter not available:rsnhindcast=climetlab.load_dataset('s2s-ai-challenge-training-input',date=[20200102],origin='ecwmf',parameter='tp',format='netcdf').to_xarray()hindcast.coordsCoordinates:*realization(realization)int64012345678910*forecast_time(forecast_time)datetime64[ns]2000-01-02...2019-01-02*lead_time(lead_time)timedelta64[ns]0days1days...45days46days*latitude(latitude)float6490.088.587.085.5...-87.0-88.5-90.0*longitude(longitude)float640.01.53.04.5...355.5357.0358.5valid_time(forecast_time,lead_time)datetime64[ns]2000-01-02...2...# for ncep hindcast provide 2010 date stringshindcast_ncep=climetlab.load_dataset('s2s-ai-challenge-training-input',date=[20100107],origin='ncep',parameter='tp',format='netcdf').to_xarray()hindcast_ncep.coordsCoordinates:*realization(realization)int640123*forecast_time(forecast_time)datetime64[ns]1999-01-07...2010-01-07*lead_time(lead_time)timedelta64[ns]1days2days...43days44days*latitude(latitude)float6490.088.587.085.5...-87.0-88.5-90.0*longitude(longitude)float640.01.53.04.5...355.5357.0358.5valid_time(forecast_time,lead_time)datetime64[ns]1999-01-08...2...List of files :grib,netcdf,
zarr (not available)Forecast inputTheforecast-input(test-input) dataset consists also in data from three different models: ECMWF (ecmf), ECCC (cwao), NCEP (eccc), for different dates.
These data are forecast data.
This could be used the input for applying the ML models in order to generate the output which is submitted for the challenge.For all 3 models:forecast_time: from 2020/01/02 to 2020/12/31, weekly every Thurday in 2020.valid_time(forecast_time+lead_time): from 2020/01/02 to 2021/02/xxavailable parameters (same as forhindcast input (training input)ECMWF forecastlead_time: 0 to 46 daysECCC forecastlead_time: 1 to 32 daysNCEP forecastlead_time: 1 to 44 daysforecast=climetlab.load_dataset('s2s-ai-challenge-test-input',date=[20200102],origin='ecmwf',parameter='tp',format='netcdf').to_xarray()forecast.coordsCoordinates:*realization(realization)int6401234567...44454647484950*forecast_time(forecast_time)datetime64[ns]2020-01-02*lead_time(lead_time)timedelta64[ns]0days1days...45days46days*latitude(latitude)float6490.088.587.085.5...-87.0-88.5-90.0*longitude(longitude)float640.01.53.04.5...355.5357.0358.5valid_time(forecast_time,lead_time)datetime64[ns]2020-01-02...2...List of files :grib,netcdf,
zarr (missing)Observations (Reference Output)Thehindcast-like-observations(training-output-reference) dataset matches thetraining-inputof the ECMWF and ECCC model.
Theforecast-like-observations(test-output-reference) dataset matches thetest-inputof all three models.The observations are the ground truth to compare with the ML model output and evaluate them. It consists in observations from instruments oftemperatureand accumulated totalprecipitation. TheNOAA CPCdatasets were downloaded fromIRIDL. We provide observations in the same dimensions as the forecasts/hindcasts to have an easy match of forecasts/hindcast and ground truth.See the script for technical details.
These observations are the ground truth and do not correspond to a model. The format is alwaysnetcdf.hindcast_like_obs=climetlab.load_dataset('s2s-ai-challenge-training-output-reference',date=[20200102],parameter='tp').to_xarray()# origin and format not acceptedhindcast_like_obs.coordsCoordinates:*latitude(latitude)float6490.088.587.085.5...-87.0-88.5-90.0*longitude(longitude)float640.01.53.04.5...355.5357.0358.5*forecast_time(forecast_time)datetime64[ns]2000-01-02...2019-01-02*lead_time(lead_time)timedelta64[ns]0days1days...45days46daysvalid_time(lead_time,forecast_time)datetime64[ns]2000-01-02...2...forecast_like_obs=climetlab.load_dataset('s2s-ai-challenge-test-output-reference',date=[20200102],parameter='tp').to_xarray()# origin and format not acceptedforecast_like_obs.coordsCoordinates:*latitude(latitude)float6490.088.587.085.5...-87.0-88.5-90.0*longitude(longitude)float640.01.53.04.5...355.5357.0358.5*forecast_time(forecast_time)datetime64[ns]2020-01-02*lead_time(lead_time)timedelta64[ns]0days1days...45days46daysvalid_time(lead_time,forecast_time)datetime64[ns]2020-01-02...2...In case you want to train on the NCEP hindcast, which hasforecast_times from 1999 to 2010, please use download observations with a time dimensions2s-ai-challenge-observationsand useclimetlab_s2s_ai_challenge.extra.forecast_like_observationsto match observations to the correspondingvalid_times of the forecast/hindcast.obs_time=climetlab.load_dataset('s2s-ai-challenge-observations',parameter=['pr','t2m']).to_xarray()# equivalentobs_lead_time_forecast_time=climetlab.load_dataset('s2s-ai-challenge-observations',parameter=['pr','t2m']).to_xarray(like=hindcast_ncep)obs_lead_time_forecast_time=climetlab_s2s_ai_challenge.extra.forecast_like_observations(hindcast_ncep,obs_time)obs_lead_time_forecast_time.coords<xarray.Dataset>Dimensions:(forecast_time:12,latitude:121,lead_time:44,longitude:240)Coordinates:valid_time(forecast_time,lead_time)datetime64[ns]1999-01-08...2...*latitude(latitude)float6490.088.587.085.5...-87.0-88.5-90.0*longitude(longitude)float640.01.53.04.5...355.5357.0358.5*forecast_time(forecast_time)datetime64[ns]1999-01-07...2010-01-07*lead_time(lead_time)timedelta64[ns]1days2days...43days44daysThis function can be used for all initialized hindcasts and forecasts from the SubX and S2S projects. Beware of the different starting dates of 2020 forecasts when using them for thes2s-ai-challenge. Observations from NOAA CPC areregridded conservativelywithxesmfto the S2S 1.5 deg grid. The original 0.5 degree spatial resolution raw observations can be obtained viagrid='720x360'.Generally speaking, only data available when the forecast is issued can be used by the ML models to perform their forecast:Rule: Observed data beyond the forecast date should not be used for prediction, for instance a forecast starting on 2020/07/01 should not use observed data beyond 2020/07/01).For all rules, see thechallenge website.See also the general rules of the challengehere.Dates in the observation dataset are from 2000/01/01 to 2021/02/15.The observations dataset have been build from real instrument observations.Thehindcast-like-observations(training-output-reference) dataset :Available from 2000/01/01 to 2019/12/31, weekly every 7 days (every Thurday)Observation data before 2019/12/31 can be used for training (as the truth to evaluate and optimize the ML models or tweak hyper parameters using train/valid split or cross-validation).Theforecast-like-observations(test-output-reference) dataset.Available from 2020/01/01 to 2021/02/20 , weekly every 7 days (every Thurday)The test data mustnotbe used during training. In theory, these data should not be disclosed during the challenge, but the nature of the data make is possible to access it from other sources. That is the reason why the code used for training model must be submitted along with the prediction (as a jupyter notebook) and the top ranked proposition will be reviewed by the organizing board.During forecast phase (i.e. the evaluation phase using the forecast-input dataset), 2020 observation data is used. Rule 1 still stands: Observed data beyond the forecast start date should not be used for prediction.Forecast Benchmark (Benchmark output)Theforecast-benchmark(test-output-benchmark) dataset is a probabilistic re-calibrated ECMWF forecast with categoriesbelow normal,near normal,above normal. The calibration has been performed by using the tercile boundaries from the model climatology rather than from observations.The benchmark data is available as follows:forecast_time: from 2020/01/02 to 2020/12/31, weekly every 7 days (every Thurday).lead_time: 14 days and 28 days, where this day represents the first day of the biweekly aggregatevalid_time(forecast_time+lead_time): from 2020/01/01 to 2021/01/29category:'below normal','near normal','above normal'bench=climetlab.load_dataset('s2s-ai-challenge-training-output-benchmark',parameter='tp').to_xarray()# origin, date and format not acceptedbench.coordsCoordinates:*category(category)object'below normal''near normal''above normal'*forecast_time(forecast_time)datetime64[ns]2000-01-02...2019-12-31*lead_time(lead_time)timedelta64[ns]14days28days*latitude(latitude)float6490.088.587.085.5...-87.0-88.5-90.0*longitude(longitude)float640.01.53.04.5...355.5357.0358.5valid_time(forecast_time,lead_time)datetime64[ns]2000-01-16...2...bench=climetlab.load_dataset('s2s-ai-challenge-test-output-benchmark',parameter='tp').to_xarray()# origin, date and format not acceptedbench.coordsCoordinates:*category(category)object'below normal''near normal''above normal'*forecast_time(forecast_time)datetime64[ns]2020-01-02...2020-12-31*lead_time(lead_time)timedelta64[ns]14days28days*latitude(latitude)float6490.088.587.085.5...-87.0-88.5-90.0*longitude(longitude)float640.01.53.04.5...355.5357.0358.5valid_time(forecast_time,lead_time)datetime64[ns]2020-01-16...2...ObservationsFor other initialized forecasts, you can downloadobservationsfor parameterst2mandprwith atimedimension corresponding tovalid_time. These are then used to create observations formatted like initialized forecasts/hindcasts locally. Observations are available from 1999 to 2021. Seeparameterfor description.climetlab_s2s_ai_challenge.extra.forecast_like_observationsmatches observations to the correspondingvalid_times of the forecast/hindcast.forecast=climetlab.load_dataset('s2s-ai-challenge-training-input',date=20100107,origin='ncep',parameter='tp',format='netcdf').to_xarray()obs_lead_time_forecast_time=climetlab.load_dataset('s2s-ai-challenge-observations',parameter=['pr','t2m']).to_xarray(like=forecast)# equivalentobs_time=climetlab.load_dataset('s2s-ai-challenge-observations',parameter=['pr','t2m']).to_xarray()obs_time.coordsCoordinates:*time(time)datetime64[ns]1999-01-011999-01-02...2021-04-29*latitude(latitude)float6490.088.587.085.5...-87.0-88.5-90.0*longitude(longitude)float640.01.53.04.5...354.0355.5357.0358.5obs_lead_time_forecast_time=climetlab_s2s_ai_challenge.extra.forecast_like_observations(forecast,obs_time)obs_lead_time_forecast_time<xarray.Dataset>Dimensions:(forecast_time:12,latitude:121,lead_time:44,longitude:240)Coordinates:valid_time(forecast_time,lead_time)datetime64[ns]1999-01-08...2...*latitude(latitude)float6490.088.587.085.5...-87.0-88.5-90.0*longitude(longitude)float640.01.53.04.5...355.5357.0358.5*forecast_time(forecast_time)datetime64[ns]1999-01-07...2010-01-07*lead_time(lead_time)timedelta64[ns]1days2days...43days44daysDatavariables:t2m(forecast_time,lead_time,latitude,longitude)float32...tp(forecast_time,lead_time,latitude,longitude)float32na...Attributes:script:climetlab_s2s_ai_challenge.extra.forecast_like_observationsData download (GRIB or NetCDF)The URLs to download the data are constructed according to the following patterns:For input datasets, the pattern ishttps://object-store.os-api.cci1.ecmwf.int/s2s-ai-challenge/data/{datasetname}/0.3.0/{format}/{origin}-{fctype}-{parameter}-YYYYMMDD.ncFor observations datasets (reference output), the pattern ishttps://object-store.os-api.cci1.ecmwf.int/s2s-ai-challenge/data/{datasetname}/{parameter}-YYYYMMDD.ncFor benchmark datasets, the pattern will be similar tohttps://object-store.os-api.cci1.ecmwf.int/s2s-ai-challenge/data/training-output-benchmark/{parameter}.nc{datasetname} : In the URLs the dataset name must follow the ML naming (training-inputortraining-output-referenceortraining-output-benchmark).{format} isnetcdf. Training output is also available as GRIB file,  usingformat='grib'and replacing".nc"by".grib"{fctype}:hindcastfor training orforecastfor test{parameter} ist2mforsurface temperature at 2m,tpfortotal precipitation{origin} :ecmwforecccorncep{weeks} from ["34","56",["34", "56"]] only forbenchmark{grid} from ["240x121"(default),"720x360"] only forobservationsYYYYMMDDis the date of the 2020 forecast fortest-input/forecast-input. The same dates are required for the on-the-flytraining-input/hindcast-inputbut return the multi-year hindcast for thatMMDD.The list of files for thetraining-inputdataset can be found atGRIB:https://object-store.os-api.cci1.ecmwf.int/s2s-ai-challenge/data/training-input/0.3.0/grib/index.html,NetCDF:https://object-store.os-api.cci1.ecmwf.int/s2s-ai-challenge/data/training-input/0.3.0/netcdf/index.html,The list of files for thetraining-output-benchmarkdataset can be found athttps://object-store.os-api.cci1.ecmwf.int/s2s-ai-challenge/data/training-output-benchmark/index.html(NetCDF only)The list of files for thetraining-output-referencedataset can be found athttps://object-store.os-api.cci1.ecmwf.int/s2s-ai-challenge/data/training-output-reference/index.html(NetCDF only)Example to retrieve the file with wget :wget https://object-store.os-api.cci1.ecmwf.int/s2s-ai-challenge/data/training-input/0.3.0/grib/ncep-hindcast-q-20101014.grib(132.8M )Zarr format (experimental).The zarr storage location include all the reference data. The zarr urls arenotdesigned to be open in a browser (seezarr):
While accessing the zarr storage without climetlab may be possible, we recommend using climetlab with the appropriate plugin (climetlab-s2s-ai-challenge)Zarr urls are not available.Using climetlab to access the data (supports grib, netcdf and zarr)The climetlab python package allows easy access to the data with a few lines of code such as:!pip install climetlab climetlab_s2s_ai_challenge
import climetlab as cml
cml.load_dataset("s2s-ai-challenge-training-input",
                         origin='ecmwf',
                         date=[20200102,20200109],
                         # optional : format='grib'
                         parameter='tp').to_xarray()
cml.load_dataset("s2s-ai-challenge-training-output-reference",
                         date=[20200102,20200109],
                         parameter='tp').to_xarray()See the demo notebooks here:https://github.com/ecmwf-lab/climetlab-s2s-ai-challenge/notebooks.Accessing the training data :NetcdfnbviewercolabGribnbviewercolabGetting the observation (reference output) dataset see thedemo_observations notebook.Getting the benchmark dataset see thedemo_benchmark notebook.