# XQuery/World Temperature records
## Introduction

The Met office recently released the temperature records for about 1600 stations world-wide. Each station record is available online as a text file, for example Stornoway.
This case study describes a project to make this data available as XML. The home page is http://www.cems.uwe.ac.uk/xmlwiki/Climate/index.html

## Parsing temperature record to XML

The first task is to convert the plain text to XML. The main page explains the format of this text file. The code 030260 is the station code defined by the World Meteorological Organisation. It appears that the files are stored in country code directories. (actually these are Blocks in WMO parlance)

### Remote data files

The task of using HTTP to GET a remote data file is a common task for which functions already exist in an XQuery module.
This module declares a constant used in the parsing:
And the basic function to get text, which may be plain text or base64encoded:

### Parsing Function

We will create an XQuery module containing functions to carry out the parsing:
The csv module needs to be imported:
Now the function to parse the MET climate data:

### Main Script

The main script uses these functions to convert a given station's record:
Stornoway

## WMO stations

The station ids are based on those defined by the World Meteorological Organisation. There is a full list of all stations available online as a text file with supporting documentation.
A typical record is
The format of these record is
* Block Number 2 digits representing the WMO-assigned block.
* Station Number 3 digits representing the WMO-assigned station.
* ICAO Location Indicator 4 alphanumeric characters, not all stations in this file have an assigned location indicator. The value "----" is used for stations that do not have an assigned location indicator.
* Place Name Common name of station location.
* State 2 character abbreviation (included for stations located in the United States only).
* Country Name Country name is ISO short English form.
* WMO Region digits 1 through 6 representing the corresponding WMO region, 7 stands for the WMO Antarctic region.
* Station Latitude DD-MM-SSH where DD is degrees, MM is minutes, SS is seconds and H is N for northern hemisphere or S for southern hemisphere. The seconds value is omitted for those stations where the seconds value is unknown.
* Station Longitude DDD-MM-SSH where DDD is degrees, MM is minutes, SS is seconds and H is E for eastern hemisphere or W for western hemisphere. The seconds value is omitted for those stations where the seconds value is unknown.
* Upper Air Latitude DD-MM-SSH where DD is degrees, MM is minutes, SS is seconds and H is N for northern hemisphere or S for southern hemisphere. The seconds value is omitted for those stations where the seconds value is unknown.
* Upper Air Longitude DDD-MM-SSH where DDD is degrees, MM is minutes, SS is seconds and H is E for eastern hemisphere or W for western hemisphere. The seconds value is omitted for those stations where the seconds value is unknown.
* Station Elevation (Ha) The station elevation in meters. Value is omitted if unknown.
* Upper Air Elevation (Hp) The upper air elevation in meters. Value is omitted if unknown.
* RBSN indicator P if station is defined by the WMO as belonging to the Regional Basic Synoptic Network, omitted otherwise.

### Conversion to XML

A function is needed to convert from the DD-MM-SSH format of latitudes and longitudes. This is complicated by the variations in this format. These variations all appear in the data:
* DD-MMH
* DD-MH
* DD-MM-SH
* DD-MM-SSH
Because this format occurs in other data, it has been added to a general module of geographic functions.
The geo module has to be imported:
Parsing the station data.

### Generating the WMO XML file

The XQuery script GETs the text file and converts each line to an XML station element. The elements are then inserted into an empty XML file one by one.

### Indexing

There are 11000 odd stations in total. These need to be indexed for efficient access. In eXist indexes are defined in a configuration file, one per collection (directory). For the collection in which the station XML document is to be written, the configuration file is:
This means that all XML documents in the collection will be indexed on the qnames id and country wherever these appear in the XML structure. Indexing will be performed when a document is added to the collection or an existing document is updated. A re-index can be forced if required.
If the station data is stored in the collection /db/Wiki/Climate/Stations, this configuration file will be stored in /db/system/config/db/Wiki/Climate/Stations as configuration.xconf

### WMO Station set binding

Since the code will reference this collection in a number of places, we add a constant to reference the set of stations to the library module:

## Temperature Station list

A full listing of stations is needed to provide an index. This data is not provided as a simple file, but they are encoded on the HTML page as a JavaScript array.
However there is no location data here, so we will get that from the WMO station list:
The approach taken to converting this to XML was:
* View source on the HTML page
* Locate the station list
* Copy the text
* Save as a text file in the eXIst data base
* A script reads this file and parses it to XML
* The resultant XML is augmented with latitude and longitude from the WMO station data.
* The final XML document is stored in the database in the same Station directory
Storing this file in the same Stations collection means that it will be indexed on the same element names, id and country,as the full WMO station data.
Temperature station list

### Climate station set binding

This set of stations will also be referenced in several places so we define a variable:

## Visualizing the data

We will use XSLT to transform this XML to a presentation of the location of the station and charts of the temperatures. The initial stylesheet was developed by Dave Challender.
( explanation to be added )

### Multiple formats

We would like to present either the original XML or the HTML visualisation page. We could use two scripts, or combine them into one script with a parameter to indicate how the output is to be rendered. eXist functions allow the serialization of the output and the mime-type to be set dynamically.
Stornoway HTML Stornoway XML

## Simple HTML index

We can use the stored station list to create a simple HTML index.
Temperature Station list

## Station Map

We can also generate a (large) KML overlay, with links to each station's page.
We need a function transform a station into a PlaceMark with a link to the HTML station page:
Then the main script iterates over all the temperature stations to generate the full KML file.
Full KML
KML rendered via GoogleMaps

## Work in progress

* Resource URIs
* RDF