Package: edam
EnvironmentalDataAcquisitionModuleEDAM (Environmental Data Acquisition Module), is a template framework that facilitates
timeseries data acquisition and integration. EDAM templates are written using programming
language-agnostic semantics, and can be reused both for input and output, thus enabling
data reuse via transformations across different formats.InstallationThis project runs onpython3. Thus, python3 should be installed.It is highly recommended installing the module via Python virtual environment
You can install and run this version. Issue the followings:pip install edamAbove command installsedam, which will be available to the virtual environment.
It also creates an.edamfolder in your system's home directory.
All EDAM related documents are located in this folder.
User=draftedtemplateandmetadata, andinputfiles should be placed
under these corresponding directories.More information about the proposed workflow can be found
in theproposed-workflow.Table of contentsInstallationEDAM template languageProposed workflowTest casesAgMIP and APSIM weather data filesAgMIP data filesAPSIM stationUK Meteorological OfficeEDAM template languageMetadata fileIn the metadata file you can defineobservables,stationandsensors.Anobservablecan have the following attributes:name (e.g. Wind)unit (e.g. direction km/h)observable_id (e.g. wind). This corresponds to the name one can use in the .tmplTemplateIn a template one can use in aplaceholder ({{}})only thetemplate_id's, which were defined in the metadata files.Atemplate_idcan have certain attributes.
These are:value (e.g.{{wind.value}})timestamp. In case the table (csv) has a dedicated column which denotes the timestamp of aspecific observable, this will be defined as{{wind.timestamp.time}}(if it's in the hh:mm format).In case the one column-timestamp refers to more than one observable (e.g. BoM wind_direction_max and wind_speed_max), this should be defined as{{same_timestamp(windm_spd.timestamp.time, windm_dir.timestamp.time)}}In case a column has the timestamp which corresponds toall observables in each row, users will use the{{timestamp.}}.Proposed workflowEditsetting.yamlto correspond to your database systemDraft a metadata file and put it into metadata folder (~/.edam/metadata)Draft a template file (~/.edam/templates)Add your input csv file into inputs folder (~/.edam/inputs)Test casesAfter parsing datasets usingedamcommand, you can issueviewercommand
and a webservice starts on you computer. If no changes tosettings.yaml(checkProposed workflow) have been made, you can visithttp://127.0.0.1:5000and view the data.For more details about theedamcommand usage,
issue aedam --helpin your terminal.AgMIP and APSIM weather data filesThe Agricultural Model Intercomparison and Improvement Project (AgMIP)
brought into the spotlight agricultural modelling data sharing.
Within AgMIP, various agricultural models (such as the APSIM) were transformed
into the AgMIP data scheme.AgMIP and APSIM data files use differenttimestamp components.
The challenge here is to compose these into one universal timestamp.
APSIM usesjulian datesandyears, while AgMIP timestamp is
represented throughyear,month,datecomponents.Another challenge was related with metadata encoded in the preamble
of APSIM data files.
Specifically, prior to the timeseries there are station metadata
such as station name, location and others.
The APSIM data files share the same structure, so these preamble-encoded
metadata distinguish the different stations.AgMIP data filesAgMIP dataset@DATE    YYYY  MM  DD  SRAD  TMAX  TMIN  RAIN  WIND  DEWP  VPRS  RHUM
1980001  1980   1   1  15.0  26.0  12.2   0.0   1.4   4.8   8.6    25
1980002  1980   1   2   6.9  21.2   9.5   0.0   1.6   8.1  10.8    42
1980003  1980   1   3  10.7  22.2  14.7   8.0   1.5  11.9  14.0    52
1980004  1980   1   4  15.6  24.0  10.0   0.0   2.9   9.2  11.7    39
1980005  1980   1   5  16.3  23.9  12.0   0.0   2.7   1.7   6.9    23
1980006  1980   1   6  15.0  23.0  11.8   0.0   2.7   2.8   7.5    26
1980007  1980   1   7  16.5  22.2  11.4   0.0   2.5  -0.5   5.9    22AgMIP template@DATE    YYYY  MM  DD  SRAD  TMAX  TMIN  RAIN  WIND  DEWP  VPRS  RHUM
{%for timestamp, srad, tmax, tmin, rain, wind, dewp, vprs, rhum in chunk%}
1980001  {{timestamp.year}}   {{timestamp.month}}   {{timestamp.day}}  {{srad.value}}  {{tmax.value}}  {{tmin.value}}   {{rain.value}}   {{wind.value}}   {{dewp.value}}   {{vprs.value}}    {{rhum.value}}
{%endfor%}AgMIP metadataStation:name:Agmipmobile:Falselatitude:23.200longitude:89.330license:Attributiontags:key1:value1key2:value2Observables:-observable_id:tminname:Temperature Minontology:https://edam.org#Temperaturequalifiers:https://edam.org#min-observable_id:tmaxname:Temperature Maxontology:https://edam.org#Temperaturequalifiers:https://edam.org#max-observable_id:rainname:Rainontology:https://edam.org#Rain-observable_id:sradname:Solar radiationontology:https://edam.org#SolarRadiation-observable_id:windname:Windontology:https://edam.org#WindSpeed-observable_id:rhumname:Humidityontology:https://edam.org#RelativeHumidity-observable_id:vprsname:Vprsontology:https://edam.org#VaporPressure-observable_id:dewpname:Dewing pointontology:https://edam.org#DewingPointUnits of Measurement:-name:Megajoule per square metresymbol:MJ/m^2relevant_observables:srad-name:Kilopascalsymbol:kParelevant_observables:vprs-name:Percentsymbol:\%relevant_observables:rhum-name:Celciussymbol:degCrelevant_observables:tmin, tmax, dewp-name:Millimeterssymbol:mmrelevant_observables:rain-name:kilometers per hoursymbol:km/hrrelevant_observables:windParse AgMIPFollowing command is executed in approximately 2 seconds:edam --input Agmip.csv --template Agmip.tmpl --metadata Agmip.yamlAPSIM stationAPSIM dataset[weather.met.weather]

Latitude   =   36.68
longitude   =   116.98
tav   =   14.6   (oC)   !   annual   average   ambient   temperature
amp   =   28.2   (oC)   !   annual   amplitude   in   mean   monthly   temperature
!!!!   1/01/1961   to   31/12/2005
   day   year   radn   maxt   mint   rain   wind   RH
   273   2002   17.5   27.2   14.6   0   3.5   54
  274   2002   13.6   23.1   14.7   0   5.3   40
  275   2002   15.8   27.1   11.1   0   5.5   29
  276   2002   15.5   25.8   16.5   0   3.8   39
  277   2002   14.9   25.5   14.6   0   2.5   63
  278   2002   15.2   23.1   15.2   0   3   47
  279   2002   13.4   19.9   10.9   0   3   38
  280   2002   15.7   19.3   8.2   0   2.5   47APSIM template[weather.met.weather]

Latitude   =   {{station.latitude}}
longitude   =   {{station.longitude}}
tav   =   {{station.tags.tav}}
amp   =   {{station.tags.amp}}
!!!!   1/01/1961   to   31/12/2005
   day   year   radn   maxt   mint   rain   wind   RH
{%for timestamp, radn,maxt,mint,rain,wind,RH in chunk%}
{{timestamp.dayofyear}}  {{timestamp.year}}  {{radn.value}}  {{maxt.value}}  {{mint.value}}  {{rain.value}}  {{wind.value}}  {{RH.value}}
{%endfor%}APSIM metadataStation:name:Yuchenglicense:AttributionObservables:-observable_id:mintname:Temperatureontology:https://edam.org#Temperaturequalifiers:https://edam.org#min-observable_id:maxtname:Max Temperatureontology:https://edam.org#Temperaturequalifiers:https://edam.org#max-observable_id:rainname:Rainontology:https://edam.org#Rain-observable_id:radnname:Solar radiationontology:https://edam.org#SolarRadiation-observable_id:windname:Windontology:https://edam.org#WindSpeed-observable_id:RHname:Relative humidityontology:https://edam.org#RelativeHumidityUnits of Measurement:-name:Millijoule per square meterssymbol:mJ/m^2relevant_observables:radn-name:Percentsymbol:\%relevant_observables:RH-name:Celciussymbol:degCrelevant_observables:mint, maxt-name:Millimeterssymbol:mmrelevant_observables:rain-name:Meters per secondsymbol:m/srelevant_observables:windParse APSIMFollowing command is executed in less than 1 second:edam --input Yucheng.met --template Yucheng.tmpl --metadata Yucheng.yamlUK Meteorological OfficeIn the context of Open Data, the UK Meteorological Office, reports historical
observations of 27 weather stations.
For every station, monthly observations are stored in one text document.
New observations are appended every month and each weather station can be
found on a certain URI.
URI follows the pattern:http://www.metoffice.gov.uk/pub/data/weather/uk/climate/stationdata/{station name}data.txt,
where {station name} is replaced with an actual station name.Data points reported in Met Office weather stations' documents have
special markers.
The challenge here is concerned with establishing a relation
among markers and attributes.
This relation, expressed using natural language, is defined
in each document's preamble.
For example, "estimated data is marked with a * after the value".
These types of metadata are essential, so they have to be parsed and stored.UK Met datasetCwmystwyth
Location: 277300E 274900N, Lat 52.358 Lon -3.802, 301 metres amsl
Estimated data is marked with a * after the value.
Missing data (more than 2 days missing in month) is marked by  ---.
Sunshine data taken from an automatic Kipp & Zonen sensor marked with a #, otherwise sunshine data taken from a Campbell Stokes recorder.
   yyyy  mm   tmax    tmin      af    rain     sun
              degC    degC    days      mm   hours
   1959   1    4.5    -1.9      20    ---     57.2
  1959    2    7.3     0.9      15    ---     87.2
  1959    3    8.4     3.1       3    ---     81.6
  1959    4   10.8     3.7       1    ---    107.4
  1959    5   15.8     5.8       1    ---    213.5
  1959    6   16.9     8.2       0    ---    209.4
  1959    7   18.5     9.5       0    ---    167.8
  1959    8   19.0    10.5       0    ---    164.8
  1959    9   18.3     5.9       0    ---    196.5
  1959   10   14.8     7.9       1    ---    101.1
  1959   11    8.8     3.9       3    ---     38.9
  1959   12    7.2     2.5       3    ---     19.2UK Met template{{station.name}}
Location:{{station.location}}, Lat {{station.latitude}} Lon {{station.longitude}}, {{station.tags.altitude}}
Estimated data is marked with a * after the value.
Missing data (more than 2 days missing in month) is marked by  ---.
Sunshine data taken from an automatic Kipp & Zonen sensor marked with a #, otherwise sunshine data taken from a Campbell Stokes recorder.
   yyyy  mm   tmax    tmin      af    rain     sun
              degC    degC    days      mm   hours
{%for timestamp, tmax,tmin,af,rain,sun in chunk%}
{{timestamp.year}}  {{timestamp.month}}  {{tmax.value}}  {{tmin.value}}  {{af.value}}  {{rain.value}}  {{sun.value}}
{%endfor%}UK Met metadataStation:license:Attributionregion:United Kingdomurl:http://www.metoffice.gov.uk/tags:key1:value1key2:value2Observables:-observable_id:tminname:Temperature minimumontology:https://edam.org#Temperaturequalifiers:https://edam.org#min-observable_id:tmaxname:Temperature Maxontology:https://edam.org#Temperaturequalifiers:https://edam.org#max-observable_id:rainname:Rainontology:https://edam.org#Rain-observable_id:afname:Days of air frostontology:https://edam.org#AirFrostDays-observable_id:sunname:Sunshine durationontology:https://edam.org#SunShineDurationUnits of Measurement:-name:Dayssymbol:Drelevant_observables:af, sun-name:Celciussymbol:degCrelevant_observables:tmin, tmax-name:Millimeterssymbol:mmrelevant_observables:rainParse all UK Met weather stationsFollowing command downloads and stores data from 27 weather stations.
It is executed in approximately 9 seconds.edam --input "http://www.metoffice.gov.uk/pub/data/weather/uk/climate/stationdata/{_var_}data.txt" --template uk.tmpl --metadata uk.yaml --var "aberporth,armagh, ballypatrick, camborne, cambridge, cardiff, chivenor, cwmystwyth, dunstaffnage, durham, eastbourne, eskdalemuir"