# Presentation: 400196

## World Data Center Climate:

- Livermore, June 19th – 21st, 2006

- Status and Portal Integration

- WDCC Home: www.wdcc-climate.de / WDCC Contact: data@dkrz.de

- Michael Lautenschlager, Hannes Thiemann and Frank Toussaint ICSU World Data Center Climate
- Model and Data / Max-Planck-Institute for Meteorology
- Hamburg, Germany

## Content:

- WDCC Status
- CERA Concept
- Portal Integration

## WDCC Content

- ERA40

- Data from
- Earth System
- Modelling and
- Related
- Observations

- Start: Approved in January 2003
- Maintenance:  Model and Data (M&D/MPI-M) and German Climate Computing Centre (DKRZ)

- June 2006: 590 Experiments / 79.000 Data Sets

## Data Export from WDC Climate

- Corresponds to 2 – 10 TB/month

## Geographical Distribution of WDCC Users

- Total number of registered users: 750 (Mai 2006)

## Data Import into WDC Climate

- ECHAM5/MPI-OM IPCC AR4 Scenarios (ca. 110 TB)

## (I) Data catalogue and Pointer to Unix files
Enable search and identification of data 
Allow for data access as they are (coarse granularity raw data files)

(II) Application-oriented data storage in BLOB tables
Time series of individual variables are stored as BLOB entries in DB Tables (fine granularity data products)
Allow for fast and selective data access 
Storage in standard data format (GRIB, NetCDF/CF)
Allow for application of standard data processing routines (PINGOs, CDOs)

- Enable search and identification of data
- Allow for data access as they are (coarse granularity raw data files)
- (II) Application-oriented data storage in BLOB tables
- Time series of individual variables are stored as BLOB entries in DB Tables (fine granularity data products)
- Allow for fast and selective data access
- Storage in standard data format (GRIB, NetCDF/CF)
- Allow for application of standard data processing routines (PINGOs, CDOs)

**CERA****1)**** Concept:**
- Semantic Data Management

**1)**   Climate and Environmental data Retrieval and Archiving

## WDCC Data Topology

- Metadata entries
- (XML, ASCII)
- + Data Files

- Level 2 – Interf.:
- Separate files
- containing BLOB
- table data in
- application
- adapted structure
- (time series of
- single variables)

**WDCC Data Topology**

- BLOB DB Table corresponds to scalable, virtual file at the operating system level.

## CERA Data Model

## Data matrix of model experiment

- Model variables

- Model Run Time

**2 D:** small BLOBS (180 KB)
**3 D:** large BLOBS (3 MB)
**Raw data file:** direct model output (1.3 – 16.2 GB)

- Each columm is one BLOB Table in CERA-DB

- Raw data file in DKRZ Archive

## Climate Model Data Structures

- 	single variable
- 	single level
- 	time series of 2D gridded data records
- 	Formats: GRIB-1 – NetCDF/CF (- GRIB-2)

**Climate Model Data Structures**

- Application related data structure (2-D)

- original data structure (4-D)

## DKRZ Architecture

- TX7: Intel Itanium-2 with Linux

## Portal Integration

- Two strategies:
- One way integration: discovery and use metadata are integrated in a central data portal in one step
- Example:
- C3Grid data catalogue (refer to presentation from Heinrich Widmann)
- Two way integration: discovery metadata are integrated in central data portal, use metadata are extracted from remote archive when they are needed for data download and processing
- Example:
- Primary data publication in TIB library catalogue (STD-DOI)
- WDCC integration in NDG (NERC Data Grid)

## Primary data publication (STD-DOI)

- URL: http://www.std-doi.de/

- Data
- Review

- Primary Data
- Publication
- Process

- ISO 690-2:
- Metadata for citation of electronic media

## Example: Publ.-DOI from WDCC

## DOI

- URN

## Publ.-DOI

## 830 GB

## Data retrieval procudure is given at the end (user identification is required)

- Data retrieval procudure is given at the end (user identification is required)

- Ident.-DOI

## WDCC Metadaten und OAI-PMH

**O** p e n  **A** r c h i v e s  **I** n i t i a t i v e
**P**rotocol for **M**etadata **H**arvesting

## WDCC support ofOAI-PMH requests

- Ü

**Identify**
- 	get information about a repository
**2.****	****ListMetadataFormats**
- 	list of available metadata formats
**3.****	****ListSets**
- 	list the structure of a repository (sets,...)
**4.****	****ListIdentifiers**
- 	list of all identifiers of a set
**5.****	****GetRecord**
- 	retrieve one individual metadata record
**6.****	****ListRecords**
- 	list records of a set (used for harvesting)

## OAI-PMH http

- Ü

**http request:**
-  	base URL
-  	list of keyword arguments
- 	Form: key=value pairs
-  	Request type GET or POST (URI syntax)
**http response:**
-  	responseDate (format: UTCdatetime)
-  	request (request that generated a response)
-  	error (incl. request that generated the error)
- http://www.openarchives.org/OAI/openarchivesprotocol.html

## 	WDCC OAI server at:

**	****WDCC OAI server at:**
- (Software: dlese (www.dlese.org) + apache-tomcat 5.5.12 + Java 1.5)
-  	http://uranus.dkrz.de:8080/oai/provider
- 	-	35 IPCC experiments with more than 11000 datasets
- 		Metadata Format: ISO 19115
- 		C3Grid 	(http://gsphere.awi.de:8080/gridsphere/gridsphere)
- 	-	40 STD-DOI experiments with more than 1700 datasets
- 		Metadata Format: DIF
- 		GO-ESSP (NDG, http://ndg.badc.rl.ac.uk/)

## NDG

- Ü

- DIF XMLs
- WDCC

- OAI Server
- WDCC
- (Software: dlese)

- OAI Client
- (dlese)

- DIF XMLs
- Provider 2

- OAI Server 2

- OAI Server n

- Discovery Portal

**OAI Harvesting (Pull or Notification)**

- Catalog
- record 1...n

- Process

- Delivery

## URL: http://glue.badc.rl.ac.uk/discovery/

- Keyword: ECHAM4