Package: agera5tools
AgERA5toolsTools for mirroring, manipulating (exporting, extracting) and
servingAgERA5data.The agera5tools consist of a set of commandline scripts as well as theagera5toolspython package
which can be used toset up a mirror for AgERA5 that can automatically build a
local copy and keep it up to date with the latest AgERA5 data.Allow operations on the downloaded NetCDF files directly such as dumping, point extraction and clippingServe AgERA5 data on web API through the HTTP protocol. By providing the latitude and longitude in the
URL, agera5tools can retrieve the corresponding data and return it as JSON.Commandline toolsThe agera5 commandline tools currently have 8 options. The first four are for setting up and managing
the local AgERA5 database:initto generate a configuration file and initialize the set up.buildto download the relevant AgERA5 data from Copernicus Climate Data Store (CDS) and build the local database.mirrorto update the current database with new days from the CDS.serveto serve AgERA5 data through a web API and return as JSON encoded data.The other four tools operate directly on the NetCDF files downloaded from the CDS.extract_point: this can be used to extract a time-series of variables for a given locationdumpwhich can be used to dump one day of AgERA5 data to CSV, JSON or SQLiteclipwhich can be used to extract a subset of AgERA5 data which will be written to a new NetCDF file.dump_gridwhich dumps the AgERA5 grid definition to CSV, JSON or SQLite.Init$agera5toolsinit--help
usingconfigfrom/data/agera5/agera5tools.yaml
Usage:agera5toolsinit[OPTIONS]InitializesAgERA5toolsThisimpliesthefollowing:-Creatingatemplateconfigurationfileinthecurrentdirectory-Creatinga$HOME/.cdsapircfileforaccesstotheCDS-Creatingthedatabasetables-Fillingthegridtablewiththereferencegrid.

Options:--helpShowthismessageandexit.Build$agera5toolsbuild--help
usingconfigfrom/data/agera5/agera5tools.yaml
Usage:agera5toolsbuild[OPTIONS]BuildstheAgERA5databasebybulkdownloadfromCDS

Options:-d,--to_databaseLoadAgERA5dataintothedatabase-c,--to_csvWriteAgERA5datatocompressedCSVfiles.--helpShowthismessageandexit.Mirror$agera5toolsmirror--help
usingconfigfrom/data/agera5/agera5tools.yaml
Usage:agera5toolsmirror[OPTIONS]IncrementallyupdatestheAgERA5databasebydailydownloadsfromtheCDS.

Options:-c,--to_csvWriteAgERA5datatocompressedCSVfiles.--helpShowthismessageandexit.Serve$agera5toolsserve--help
usingconfigfrom/data/agera5/agera5tools.yaml
Usage:agera5toolsserve[OPTIONS]StartsthehttpservertoserveAgERA5datathroughHTTP

Options:-p,--portINTEGERPorttonumbertostartlistening,default=8080.--helpShowthismessageandexit.Extract point$agera5toolsextract_point--help
Usage:agera5toolsextract_point[OPTIONS]AGERA5_PATHLONGITUDELATITUDESTARTDATEENDDATEExtractsAgERA5dataforgivenlocationanddaterange.AGERA5_PATH:pathtotheAgERA5datasetLONGITUDE:thelongitudeforwhichtoextract[dd,-180:180]LATITUDE:thelatitudeforwhichtoextract[dd,-90:90]STARTDATE:thestartdate(yyyy-mm-dd,>=1979-01-01)ENDDATE:thelastdate(yyyy-mm-dd,<=1weekago)Options:-o,--outputPATHoutputfiletowriteto:.csv,.jsonand.db3(SQLite)aresupported.GivingnooutputwillwritetostdoutinCSVformat--tocelsiusConverttemperaturevaluesfromdegreesKelvintoCelsius--helpShowthismessageandexit.Dump$agera5toolsdump--help
Usage:agera5toolsdump[OPTIONS]AGERA5_PATHDAYDumpAgERA5dataforagivendaytoCSV,JSONorSQLiteAGERA5_PATH:PathtotheAgERA5datasetDAY:specifiesthedaytobedumped(yyyy-mm-dd)Options:-o,--outputPATHoutputfiletowriteto:.csv,.jsonand.db3(SQLite)aresupported.GivingnooutputwillwritetostdoutinCSVformat--tocelsiusConverttemperaturevaluesfromdegreesKelvintoCelsius--add_grididAddsagridIDinsteadoflatitude/longitudecolumns.--bboxFLOAT...Boundingbox:<lon_min><lon_max><lat_min<<latmax>--helpShowthismessageandexit.Clip$agera5toolsclip--help
Usage:agera5toolsclip[OPTIONS]AGERA5_PATHDAYExtractsaportionofagERA5forthegivenboundingboxandsavestoNetCDF.AGERA5_PATH:PathtotheAgERA5datasetDAY:specifiesthedaytobedumped(yyyy-mm-dd)Options:--base_fnameTEXTBasefilenametouse,otherwisewilluse'agera5_clipped'-o,--output_dirPATHDirectorytowriteoutputto.Ifnotprovided,willusecurrentdirectory.--boxFLOAT...Boundingbox:<lon_min><lon_max><lat_min<<latmax>--helpShowthismessageandexit.dump_gridUsage:agera5toolsdump_grid[OPTIONS]DumptheagERA5gridtoaCSV,JSONorSQLiteDB.

Options:-o,--outputPATHoutputfiletowriteto:.csv,.jsonand.db3(SQLite)aresupported.GivingnooutputwillwritetostdoutinCSVformat--helpShowthismessageandexit.Python packageThe shell commands described above can also be used from python directly by importing the agera5tools package.
Their working is nearly identical as the shell commands. The major difference is that the python functions
return either datasets (clip) or dataframes (extract_point, dump, dump_grid). An example for theclipfunction:In[1]:importdatetimeasdt...:importagera5tools...:fromagera5tools.utilimportBoundingBox...:day=dt.date(2018,1,1)...:bbox=BoundingBox(lon_min=87,lon_max=90,lat_min=24,lat_max=27)...:ds=agera5tools.clip(day,bbox)...:In[2]:dsOut[2]:<xarray.Dataset>Dimensions:(time:1,lon:30,lat:30)Coordinates:*time(time)datetime64[ns]2018-01-01*lon(lon)float6487.187.2...89.990.0*lat(lat)float6426.926.8...24.124.0Datavariables:Precipitation_Flux(time,lat,lon)float32dask.array<chunksize=(1,30,30),meta=np.ndarray>Solar_Radiation_Flux(time,lat,lon)float32dask.array<chunksize=(1,30,30),meta=np.ndarray>Temperature_Air_2m_Max_Day_Time(time,lat,lon)float32dask.array<chunksize=(1,30,30),meta=np.ndarray>Temperature_Air_2m_Mean_24h(time,lat,lon)float32dask.array<chunksize=(1,30,30),meta=np.ndarray>Temperature_Air_2m_Min_Night_Time(time,lat,lon)float32dask.array<chunksize=(1,30,30),meta=np.ndarray>Vapour_Pressure_Mean(time,lat,lon)float32dask.array<chunksize=(1,30,30),meta=np.ndarray>Wind_Speed_10m_Mean(time,lat,lon)float32dask.array<chunksize=(1,30,30),meta=np.ndarray>Attributes:CDI:ClimateDataInterfaceversion1.9.2(http://mpimet.mpg.de/...history:FriMar1215:04:432021:cdosplitday/archive/ESG/wit015/...Conventions:CF-1.7CDO:ClimateDataOperatorsversion1.9.2(http://mpimet.mpg.de/...It works in a very similar way for theextract_pointfunction:In[6]:fromagera5tools.utilimportPointIn[7]:pnt=Point(latitude=26,longitude=89)In[8]:df=agera5tools.extract_point(pnt,startday=dt.date(2018,1,1),endday=dt.date(2018,1,31)),In[7]:df.head(5)Out[7]:dayprecipitation_fluxsolar_radiation_flux...temperature_air_2m_min_night_timevapour_pressure_meanwind_speed_10m_mean02018-01-010.3113282992...12.15679911.8097311.31758912018-01-021.9113646220...12.34204111.7118601.41607522018-01-030.1414817991...11.06451411.1988711.52426832018-01-040.0314131904...10.86187711.4132781.56640542018-01-050.0714315206...12.29296910.9841811.597181[5rowsx8columns]Note that extracting point data for a long timeseries can be time-consuming because all netCDF files have to be opened, decompressed and the point extracted.Installing agera5toolsRequirementsThe agera5tools package requires python >=3.8 and has a number of dependencies:pandas == 1.4.1PyYAML >= 6.0Pandas >= 1.5SQLAlchemy >= 1.4PyYAML >= 6.0xarray >= 2022.12.0dask >= 2022.7.0click >= 8.1flask >= 2.2cdsapi >= 0.5.1dotmap >= 1.3netCDF4 >= 1.6requests >= 2.28wsgiserver >= 1.3Lower versions of dependencies may work, but have not been tested.InstallingInstallingagera5toolscan be done through the github repository to get the latest version:pipinstallhttps://github.com/ajwdewit/agera5tools/archive/refs/heads/main.zipor directory from PyPI:pipinstallagera5tools