# Presentation: 324903

## Embedding Live Access Server into GFDL Data Portal Infrastructure 

- K.O’Brien (PMEL),  S.Nikonov (GFDL),
- R.Schweitzer (PMEL),  S.Hankin  (PMEL),  V.Balaji (GFDL)

*The 6**th** GO-ESSP Workshop June 11-13 2007 *
*Jussieu, Paris*

## Outlines

- Curator metadata DB, part of GFDL FMS Runtime Environment (FRE) is a centralized metadata storage for entire modeling process.
- Live Access Server (LAS) is important component of GFDL Data Portal.
- FRE –> Curator DB –> LAS metadata stream.
- Benefits of symbiosis.

*The 6**th** GO-ESSP Workshop June 11-13 2007 *
*Jussieu, Paris*

##     Live Access Server  +  Curator  =

**LASurator **(not LACerater !!!)

*The 6**th** GO-ESSP Workshop June 11-13 2007 *
*Jussieu, Paris*

- Symbiotic self configuring system combining LAS and Curator metadata DB.
- Part of FRE
- Essential part of GFDL Data Portal

## 
Technical part of modeling process consists of three important parts: assembling model, configuring it and running simulation. 
Two ways of conducting these stages: 
write once and forever sophisticated universal scripts with a lot of input parameters needed for running them; 
write tools which generate disposable scripts with configurations’ parameters needed for simulation hardcoded in these scripts. 
FRE was design using the 2nd way avoiding scientists the long road of configuring model experiment every time. 1st version offers XML metadata file as user interface for configuring model and experiment.
The 2nd version will give convenience and automation to scientists in controlling the process of model building through more user friendly interface then XML file based on centralized storage – Curator DB.

- Technical part of modeling process consists of three important parts: assembling model, configuring it and running simulation.
- Two ways of conducting these stages:
- write once and forever sophisticated universal scripts with a lot of input parameters needed for running them;
- write tools which generate disposable scripts with configurations’ parameters needed for simulation hardcoded in these scripts.
- FRE was design using the 2nd way avoiding scientists the long road of configuring model experiment every time. 1st version offers XML metadata file as user interface for configuring model and experiment.
- The 2nd version will give convenience and automation to scientists in controlling the process of model building through more user friendly interface then XML file based on centralized storage – Curator DB.

- Curator DB is important part of FRE

*The 6**th** GO-ESSP Workshop June 11-13 2007 *
*Jussieu, Paris*

## Modes of working Curator DB within FRE

**Research mode** (component oriented) - modeler introduces new physical processes / parameterization / algorithmizations / components from newly developed modules. New entities have to be described in database.
**Production mode** (simulation oriented) – experimenter composes coupled model from available components described in database, builds scenario, postprocessing plan and runs experiment. All this activity is recorded in database.
-      Thoroughly elaborated very  friendly GUI is the critical thing for these modes   otherwise users will avoid the database load stage => DB will be empty.
**Automatic mode** – applications write metadata into database harvesting it from model output data files or queries it from DB during data searching.
**Publishing mode** – metadata is extracted from DB by publishing tools for describing public data.

*The 6**th** GO-ESSP Workshop June 11-13 2007 *
*Jussieu, Paris*

## Curator DB on different stages of modeling process

- Curator DB on different stages of modeling process

- Metadata Curator DB

- Component Building

- Model Assembling

- Experiment Preparation

- Postprocessing

- Data Portal Service

*The 6**th** GO-ESSP Workshop June 11-13 2007 *
*Jussieu, Paris*

**FMS Runtime Environment**
- (currently developed version)

**in development**
**currently**

## Curator DB Design Compartments 

**Process Domains & Physical Process**:   decomposition of physical reality into homogeneous domains with descriptions of theoretical approaches for processes there considered for modeling.
**Algorithmization**:   describes program modules of elementary physical processes
**Composition**:   components, couplers; drivers; technical environment needed for assembling model as computer application
**Simulation**:   metadata on simulations and model output data
**Publishing**:   all metadata on data available for public; includes descriptions needed for Data Portal software (OPeNDAP, LAS).

*The 6**th** GO-ESSP Workshop June 11-13 2007 *
*Jussieu, Paris*

## Curator database

- Contains a lot of metadata information needed for entire modeling process
- Well ordered “information architecture”
- Live Access Server
- Configurable metadata access
- Configurable data browse/access

*The 6**th** GO-ESSP Workshop June 11-13 2007 *
*Jussieu, Paris*

## The GFDL IPCC hierarchy, for example:

- IPCC  (Project)
- GFDL    (Institution)
- CM2.0    (Model number)
- Climate of the 20th Century   (Scenario)
- Realization 1    (Which realization)
- Run 1 (Which run)

- 3 hourly data (Temporal domain)

- Jan 1991 – Dec 2000  (Dates of on-line data)

*The 6**th** GO-ESSP Workshop June 11-13 2007 *
*Jussieu, Paris*

- Atmosphere   (Domain)

## Metadata Tree

- CM2.0

- CM2.1

- Climate of the 20th Century

- 1%/year CO2 increase scenario (to quadrupling)

- 1%/year CO2 increase scenario (to quadrupling)

- Climate of the 20th Century

- Run 2

- Run 1

- Run 2

- Run 1

- annual

- annual

- Jan 2046 – Dec 2050

- Jan 2096 – Dec 2100

- Jan 2196 – Dec 2200

- Jan 2146 – Dec 2150

- Precipitation

- Surface Air Temperature

- Surface Air Temperature

- Surface Latent Heat Flux

- Sea Level Pressure

*The 6**th** GO-ESSP Workshop June 11-13 2007 *
*Jussieu, Paris*

## Challenge: 

- How to configure LAS to allow efficient user interaction and reflecting all variety of available data
- How to minimize work for LAS installer

- Solution:
- Use Curator DB to pull together pertinent metadata on experiments
- Modify LAS utility addXML to read metadata from mySQL database rather than individual or aggregated files
- Create generic velocity templates to present in friendly HTML for users

*The 6**th** GO-ESSP Workshop June 11-13 2007 *
*Jussieu, Paris*

## Simple but effective architecture

- Aggregations’ URLs

- metadata

- metadata

- metadata

**addXML**

- Categories

*The 6**th** GO-ESSP Workshop June 11-13 2007 *
*Jussieu, Paris*

**Curator DB**

**metadata**
**Extractor,**
**XMLGenerator  **

**metadata**
**Extractor/**
**XMLGenerator  **

- metadata

## Behind the scenes

*The 6**th** GO-ESSP Workshop June 11-13 2007 *
*Jussieu, Paris*

## Tools populating Curator 

- Populating Curator DB is automatic.
- One piece of software scans data storage following the list of public experiments from Curator DB. Traversing data files it extracts metadata and write it in Curator. In next version of FRE metadata will populate DB automatically in course of all stages from assembling model through experiment running.
- Another tool analyses metadata from Curator and creates aggregations records in accordance to the aggregation criterion. Currently, Experiment is used as a sorting criterion for upper level category.
- Last stage is generating THREDDS configuration XML files based on prepared metadata in Curator.

*The 6**th** GO-ESSP Workshop June 11-13 2007 *
*Jussieu, Paris*

## Interaction FRE ↔ Curator → LAS(final goal)

*The 6**th** GO-ESSP Workshop June 11-13 2007 *
*Jussieu, Paris*

**metadata**

**FMS**

**Config data**

**metadata**

**LAS**

- in development
- currently

- GUI

## LAS oriented Curator DB design features 

- Created specific tables for
- Inventorying metadata about time spans and variables of available data;
- Describing projections in details;
- Storing THREDDS aggregations descriptions including such fields as type of averaging, domain of variable (atmos, land,...), time limits, URL, aggregation criterion.

- Dynamic hierarchy
- Implemented flexibility of criteria for categories hierarchy giving to user freedom in choosing his own interface layout preferences.

*The 6**th** GO-ESSP Workshop June 11-13 2007 *
*Jussieu, Paris*

## Curator DB table samples

*The 6**th** GO-ESSP Workshop June 11-13 2007 *
*Jussieu, Paris*

## Future development –Discovery/Navigation Interface

- Imagine a search interface with Ajax talking to Curator DB or machine oriented auxiliary DB automatically designed based on adopted ontology and populated by RDF triples from main metadata DB - Curator (like it’s proposed and explored in ESG project) ....

*The 6**th** GO-ESSP Workshop June 11-13 2007 *
*Jussieu, Paris*

- As the user constrains the search, output and menu lists adjust immediately.

## The 6th GO-ESSP Workshop June 11-13 2007 

*The 6**th** GO-ESSP Workshop June 11-13 2007 *
*Jussieu, Paris*

## The 6th GO-ESSP Workshop June 11-13 2007 

*The 6**th** GO-ESSP Workshop June 11-13 2007 *
*Jussieu, Paris*

##  Information Products

- Desktop:
- Matlab,
- IDL, IDV,
- Ferret,
- GrADS, ...

-  Information Products

- Files:
- netCDF,
- binary,
- spreadsheet,
- GIS layer, ...

## Benefits

- Curator DB contains extensive metadata (aggregations, gridspec descriptions, time spans of data availability).
- LAS configuration can be pulled directly from Curator DB.
- Simplifies the configuration of the complicated information hierarchy for the LAS installer.   (using LAS “categories”)
- Simplifies the complex mosaic of datasets which are presented to the LAS user.
- LAS addXML tool works faster.

*The 5**th** GO-ESSP Workshop*
*June 19-21 2006,  LLNL*

## 			Thanks!

				Questions?

- 				Questions?

*The 6**th** GO-ESSP Workshop June 11-13 2007 *
*Jussieu, Paris*

## Compartment Structure of Curator Database

| Domains |
| --- |
| Atmosphere |
| Ocean |
| Ice |
| Land |
| Surf_BoundaryLayer |
| Rivers |
| Lakes |

| Dynamics |
| --- |

| Radiation |
| --- |

| IceProc |
| --- |

| BiotaProc |
| --- |

| Hydrology |
| --- |

| CloudProc |
| --- |

| Chemistry |
| --- |

| Convection |
| --- |

| ProcCodeBase |
| --- |

| NumArtificies |
| --- |

| GridSpecs |
| --- |

| BoundCond |
| --- |

| TracerModels |
| --- |

| NameLists |
| --- |

| DataSets |
| --- |

| Components |
| --- |

| CmpPMIOD |
| --- |

| CmpDrivers |
| --- |

| Services |
| --- |

| Versioning |
| --- |

| Compiling |
| --- |

| PlatformEnv |
| --- |

| Others |
| --- |

**.**
**.**
**.**

| Projects |
| --- |

| Experiments |
| --- |

| Scenarios |
| --- |

| PostProc |
| --- |

| Variables |
| --- |

| Fields/Files |
| --- |

| Aggregations |
| --- |

| Realizations |
| --- |

| DomConstituents |
| --- |

*The 5**th** GO-ESSP Workshop*
*June 19-21 2006,  LLNL*

| InitCond |
| --- |

| CouplModels |
| --- |

**Notes:**
And finally we have the next nested structures of Curator DB