# Presentation: 686788

## Development of metadata standards and discovery catalogues within the WMO community

- Jeremy Tandy 19th June 2006

**Notes:**
Top slide

## Outline of the talk:-

- Introduction to WIS
- Metadata profiles
- Discovery catalogues
*Planned* reference implementation

**Notes:**
Outline

## An introduction to WIS

- The status quo: 50 years of ‘interoperability’
- WMO ‘Global Telecommunication System’ (GTS)
- WMO facilitates the free and unrestricted exchange of data and information amongst its membership of 187 member states and territories
- World Weather Watch programme
- Internally focused; ‘expert’ usage
- Point-to-point ‘private’ network
- But things have changed ...
- Advent of the internet
- Need to provide value *beyond* WMO
- Integrate into a wider community

- Image source: http://gridcafe.web.cern.ch/gridcafe

**Notes:**
Outline

## WIS objectives

- WIS: WMO Information System
- A response to the changing requirements of WMO
- Aiming to meet information exchange requirements of *all* WMO programmes and those of external communities
- Standards-based
- Exploiting ‘contemporary’ internet technologies
- Development via ‘expert teams’ coordinated by the *Inter-Commission Coordination Group on WIS (ICG on WIS)*
- Objectives
- Provide an on-line metadata & data catalogue
- Harmonize data formats & transmission standards to support inter-disciplinary use of data and information
- Automatically disseminate information to **subscribers**
- Enable ad-hoc request of data & information
- Reliable, cost-effective, modular, scalable, secure, flexible ...

**Notes:**
Outline

## WIS topology

***Global Information System Centre (GISC)***
***Data Collection and Production Centre (DCPC)***
***National Centre (NC)***

**Notes:**
GISC mission
Receive observations intended for global exchange from NCs and DCPCs within their area of responsibility
Exchange information intended for global exchange with other GISCs
Disseminate, within its area of responsibility, the data and products agreed by WMO for global exchange 
Hold the data and products agreed for global exchange for at least 24 hours and make it available via WMO request/reply (”Pull”) mechanisms
Maintain, in accordance to the WMO standards, a catalogue of all data and products for global exchange and provide access to this catalogue

DCPC mission
Receive information intended for dissemination to NCs within its area of responsibility (i.e. regional collection)
Collect program-specific data and products
Produce regional or specialized data and products “Push” information intended for global exchange to their associated GISC
Disseminate information intended for regional exchange Support access to their products via request/reply (”Pull”) mechanism
Maintain data and product catalogues in a WMO-agreed standard format and facilitate access to this catalogue

NC mission
Collect national observational data
“Push” data intended for global dissemination to the associated GISC
“Push” data intended for regional or specialized distribution to the associated DCPC
Collect, generate and disseminate products for national use
Participate in monitoring the performance of the system
Authorize their national users to access FWIS, as appropriate

## Standards-based metadata

- ‘Target’ metadata standard: ISO 19115
- Previous WMO Core Metadata profile of ISO 19115 was not sufficiently compliant with standard
- ‘bespoke’ XML encoding of ISO ‘*abstract*’ specification
- lack of conformity between ISO 19115 model and XML encoding
- no UML models for proposed extensions
- ISO 19115 Annex C ‘rules for extensions’ not followed
- WMO Core Metadata profile has been re-implemented ...
- ‘base-line’ XML encoding: ISO 19139 ‘*implementation*’ specification ... based on the GML ‘toolkit’, compliant with ISO 19118 ‘Geographic Information – Encoding’
- small number of extensions required to support WMO community ... *4 or 5 in total!*

**Notes:**
Outline

## Creating an ISO-compliant metadata extension

- How do we create an ISO 19115 compliant metadata profile?
- Guidance:
**ISO 19115 Annex C: Metadata extensions and profiles**
**ISO 19115 Annex F: Metadata extension methodology**
- In order to ensure interoperability beyond the community where the extensions are implemented:
- You must document the extension via the ‘extension metadata’ described in ISO 19115, and
- Add an *isoType* attribute to the class indicating which ISO 19115 class was sub-classed

**Notes:**
ISO 19115 Clause C.2: Types of extensions
adding a new metadata section
creating a new metadata code list to replace the domain of an existing metadata element that has “free text” listed as its domain value
creating new metadata code list elements (expanding a code list)
adding a new metadata element
adding a new metadata entity
imposing a more stringent obligation on an existing metadata element
imposing a more restrictive domain on an existing metadata element

ISO 19115 Clause C.4: Rules for creating an extension
the name, definition or data type of an existing element can not be changed
a new element may include extended and existing metadata elements as components
metadata elements can be more stringent 
domains can be more restrictive
the use of domain values can be restricted
code lists of type «CodeList» can be expanded
an extension shall not permit anything not allowed by the standard

## Rules for creating a metadata profile

- check registered profiles
- adhere to the rules for defining an extension
- a profile shall include:
- the core metadata
- all mandatory metadata elements
- all conditional metadata elements, if the dataset meets the condition
- use UML diagrams to describe a profile
- use your own namespace
- publish the profile

**Notes:**
Outline

## Metadata implementation decisions (1)

- Stick with ISO 19139 *unless* it feels ‘unnatural’
_*Discovery*_ metadata only ...
- Clear separation of concerns between providing enough information to *differentiate* between datasets and the *encoding *of the data itself *(e.g. station metadata)*
- Namespaces
- http://wis.wmo.int/2006/metadata
- http://wis.wmo.int/2006/catalogues
- Non-geometric vertical co-ordinates
- New work item with ISO/TC211 to modify the definition of vertical co-ordinates to allow “parametric” co-ordinate systems

**Notes:**
Outline

## Metadata implementation decisions (2)

- Time representation
- ‘time’ is key attribute allowing a user to differentiate between datasets ... modifications required:
- Issue time *(frequency of update for operational meteo.)*
- Numerical simulation related time issues *(analysis time)*
- Usage constraints *(aviation forecasts)*
- Multilingual support
- ISO 19139 multilingual support adequate
- Agreement to use UTF-8 or UTF-16 character encodings
- Keywords and codelists
- Implementation of controlled vocabularies
- ISO 19139 implementation of codelist catalogues
- Hierarchical keyword lists ... similar to NASA’s GCMD earth-science keyword lists

**Notes:**
Outline

## WMO Core Metadata profile – extensions to ISO

- Simple UML models

- Image source: http://gridcafe.web.cern.ch/gridcafe

- Encode using
- ISO 19118 rules

- Export as ISO 19139
- compliant W3C schema

**Notes:**
Outline

## ISO 19139 – Encoding summary (1)

- From ISO 19139 Clause 8 ...
- Each UML class is encoded into 3 XML constructs:
- XML Class Type (XCT)
*describing the content (attributes) of the UML class*
- XML Class Global Element (XCGE)
*ensuring the class has global visibility within the XML schema for ‘import’ etc.*
- XML Class Property Type
*containment of a class is managed through the XML Class Property Type of it’s data type, enabling both “by Value” and “by Ref” implementations of content*
*Note: when an XCT is of xs:simpleType, “by Ref” is not permitted*

## ISO 19139 – Encoding summary (2)

- Naming conventions:
*UML class **	**» **Class1*
*XCT**	** ** **	**» **Class1_Type*
*XCGE**	**» **Class1*
*XCPT**	**» **Class1_PropertyType*
- Special case encodings:
*Abstract classes*
*Inheritance and sub-classes** *
*Note: to achieve interoperability only ‘extension’ sub-classing (i.e. adding attributes) is permitted – restriction and multiple inheritance are not allowed*
*Enumerations*
*Codelists*
*Unions*

## Polymorphism and substitution groups (1)

- Polymorphism: *the ability to assume different forms*
- Example:
*[CI_ResponsibleParty] **» individualName [CharacterString]** *could be specialized such that ‘name’ is compartmentalized into first and last names
- Polymorphism provides communities with a mechanism to better refine metadata to meet organizational need
*individualName* is extended within a ‘community’ namespace ...
- whilst still utilizing ISO 19139 schemas (*gmd* namespace), and
- still providing usable & understandable instance documents
- In OO, the specialized class “is a” type of the general class; e.g. a _dog_ “is a” type of _animal_

## Polymorphism and substitution groups (2)

- The “is a” relationship implies semantic consistency & substitutability
- ISO 19118 *only* permits simple (extension-only) sub-classing / specialization
- ISO 19139 allows polymorphism primarily though the *_PropertyType* encodings; e.g.
*gmd:PT_FreeText_PropertyType* is substituted for the more general *gco:CharacterString_PropertyType* for multi-lingual support
- Need to inform the XML parser of the substitution:
- via a *substitutionGroup* directive in the schema, OR
- via a *xsi:type* directive in the instance document

## Polymorphism and substitution groups - example

*MD_Metadata*

*identificationInfo*

## The future?

- Whilst WMO are adopting ISO 19139 as the ‘default’ encoding for WMO Core metadata ...
- WMO must ensure that the metadata profile is fit for purpose within the WMO community –
- where the existing ISO standard information models and encodings are not appropriate we should *adapt* them to our needs (e.g. ISO 19111 and parametric spatial referencing systems)
- The ISO standards are still in flux; so long as we *engage* with ISO/TC 211 we can adapt the standards to suit WMO
- This ‘profile’ is a *big* step forward, but it should not be considered a **final** version ... it will continue to evolve

**Notes:**
Outline

## Integration with other ISO metadata profiles?

**Numerical Simulation Discovery Metadata (aka NumSim)**
- The DIF describes datasets at the discovery level, but where simulations are involved, discovery metadata needs more information than is available in existing schema.
- A new schema which is being developed and trialled at the BADC should be accessible to both DIF and ISO19115 parent discovery schema, although at the moment it is rather standalone.
**WMO has proposed to adopt NumSim as a formal mechanism for describing simulation metadata within WMO Core metadata profile. **Prior to adoption, WMO will liaise with BADC to ensure that the profile is fit for purpose and has been rigorously tested.
*http://proj.badc.rl.ac.uk/ndg/wiki/NumSim*

**Notes:**
Outline

## Discovery catalogues 

- Discovery catalogue exposes ISO 19139 metadata
- Search criteria:
**Time, Location, Originator, etc**
- Unclassified search is hard work ...
- Need to implement browse functionality ...

## Relationships for discovery

*organisation*

- ISO 19139

*data instance*
*[location, time]*

*feature type*

*service*

*pre-defined query*

*phenomena*
*definition*

*controlled *
*vocabulary*

- external
- ontology

- Discover by browsing the **relationships** between things

## Supporting metadata

- For a functional discovery catalogue, we require *more* than just ‘dataset’ metadata ...
- a number of resources are *external* to the metadata instances ...
- Coordinate Reference System definitions
- Unit of Measure definitions
- Phenomena (parameter) definitions
- Codelists
- Keyword thesauri
- Feature catalogues
- Gazetteers
- Station lists

**Notes:**
Outline

## Granularity: what is a dataset?

- Getting the granularity of datasets correct is a key challenge!
- What is a dataset? It is up to the data provider to decide *how* they want to package the data they have collected before publishing to the GISC

- coarse grained,
- few records

- fine grained,
- millions of records

*«increasing flexibility»*

*«increasing performance»*
*(query, search, synchronization etc.)*

## Service binding ... accessing the data 

## WIS prototype demonstration – Nov 2006

- Metadata harvest
- via OAI protocol

**Notes:**
JCOMM E2EDM participants:
RIHMI-WDC (Russian NODC), Russia 
VLIZ, Belgium
IFREMER, France
MetOffice, UK
MEDS, Canada
NODC, USA

- Questions & Answers