# Presentation: 653014

## PCMDI Data Management and the Earth System Grid

- PCMDI Software Team

**PCMDI Advisory Committee Meeting**
**Livermore, California**
**April 8, 2008**

- LLNL-PRES-402704

## Outline

**PCMDI data management**
**ESG-CET**
**ESG-CET from user and publisher perspectives**
**Standards**
**Collaborations**

## PCMDI leadership in climate data management

**PCMDI has led the climate community in organizing and providing access to model intercomparison data**
**The CMIP3/AR4 has had a significant impact**
**Currently archiving 35TB from 25 models**
**300+ peer-reviewed publications**
**Demand for data has increased since release of the IPCC WG1 SPM (Feb. ‘07)**
**Data downloads of 400-600GB/day**

***“******A major advance of this assessment of climate change projections compared with the TAR is the large number of simulations available from a broader range of models. Taken together with additional information from observations, these provide a quantitative basis for estimating likelihoods for many aspects of future climate change.”***
**IPCC WG1 Summary for Policymakers**

**New archives are emerging:**
**Cloud Feedback Model Intercomparison Project (CFMIP)**
**North American Regional Climate Change Assessment Program (NARCCAP)**
**CCSM Carbon-Land Model Intercomparison Project (C-LAMP)**
**Current software based on Earth System Grid-II**
** ****	****	**

## Experience from CMIP3, ESG-II

**Standardization is key****:**
**‘****Homogenization of data’: transposition of data from raw history format to one variable per file;**
**Consistent format with standard APIs (netCDF)**
**Consistent metadata: CF-1**
**Targeting a specific community (WG1) helped to bound the problem.**
**Provide ****multiple paths to the data****.**
**ESG Web portal for search and discovery**
**FTP  for bulk downloads, scriptable with wget, ease of client access, familiarity**
**OPeNDAP allows subsetting, richer client access**
**Registering users gives important information on data usage.**
**Helps demonstrate the value of the service to our sponsors.**
**Model data is not static****.**
**>100 entries on the CMIP3 errata pages**
**Goal: Retain what worked in the next iteration: ESG-CET**

## Earth System Grid Center for Enabling Technologies (ESG-CET)

**Petabyte-scale data volumes**
**Federation of sites**
**“****Virtual Datasets” created through **
**   ****subsetting and aggregation**
**Metadata-based search and discovery**
**Bulk data access**
**Web-based and analysis tool access**
**Increased flexibility and robustness**
**PI: Dean Williams (PCMDI)**

- ESG Goals

- Current ESG Sites

- http://www-pcmdi.llnl.gov

## CMIP5/AR5 requirements are driving ESG-CET

**Data Volume**
**10s of terabytes (10****12**** bytes)**
**Downloads: ~500GB/day**
**Models**
**25 models**
**Metadata**
**CF-1 + IPCC-specific**
**User Community**
**Thousands of users**
**WG1, domain knowledge**

**Data Volume**
**1-10 petabytes (10****15**** bytes)**
**Downloads: 10s of TB/day**
**Models**
**~35 models**
**Increased resolution**
**More experiments**
**Increased complexity (ex: biogeochemistry)**
**Metadata**
**CF-1 + IPCC-specific**
**Richer set of search criteria**
**Model configuration**
**Grid specification from CF (support for native grids)**
**User Community**
**10s of thousands of users**
**Wider range of user groups will require better descriptions of data, attention to ease-of-use**

**AR5**

**AR4**

## Improvements over ESG-II

**‘****Faceted’ search**** capability guides the user toward datasets of interest**
**At a given point in the search, only those options which produce non-empty result sets are shown**
**Avoids ‘deadend’ searches**
**Flexible browsing hierarchy**
**Automated, GUI-based publication tools**
**Single sign-on**
**Full support for ****data aggregations**
**A collection of files, usually ordered by simulation time, that can be treated as a single file for purposes of data access, computation, and visualization.**
**Client access**** to subsetting, visualization services.**
**Server-generated visualization products**
**Fine-grained access to datasets based on user groups and roles.**
**User notification service**
**Users can choose to be notified when a dataset has been modified.**
**Pre-computed products**** (e.g., global averages)**

## ESG-CET timeline

**2008: Design and implement core functionality:**
**Browse and search**
**Registration**
**Single-sign on / security**
**Publication**
**Distributed metadata**
**Server-side processing**
**Testbed in early 2009**
**Plan to include at least seven centers in the US, Europe, and Japan:****PCMDI, NCAR, GFDL, ORNL, BADC, MPI, CCSR**
**PCMDI will archive a core set of runs, as in AR4**
**Will accept data from AR5 centers that choose not to join ESG**
**2009: Deal with system integration issues, develop production system.**
**2010: Modeling centers publish data.**
**2011: Research completed and journal articles submitted.**

## Gateways and nodes

**Federated architecture**

**Gateways**
**Portals, search capability, distributed metadata, registration and user management**
**More complex architecture than nodes, fewer sites**
**Initially PCMDI, NCAR, ORNL, eventually GFDL**
**May be customized to an institution’s requirements**
**Nodes**
**Where data is stored and published**
**Data may be on disk or tertiary mass store.**
**Each node has a trust relationship with a specific gateway, for publication.**
**Less complex architecture**
**A site can be both a gateway and a node.**

**Federation is **_**a virtual trust relationship among independent management domains that have their own set of services**_**.** **Users authenticate once to gain access to data across multiple systems and organizations.**

## Architecture of the next generation of ESG

**AR5 ESG Gateway (PCMDI)**

**user****registration**

**security****services**

**monitoring****services**

**metadata**
**services**

**notification****services**

**services ****startup/shutdown**

**OPeNDAP/OLFS****(aggregation)**

**product server**

**publishing****(harvester)**

**storage****management**

**backend analysis****and vis engine**

**workflow**

**metrics****services**

**replica location**** services**

**replica****management**

**ESG Node (GFDL)**

**access****control**

**HTTP/FTP/GFTP****servers**

**metrics****services**

**publishing****(extraction)**

**OPeNDAP/OLFS**

**OPeNDAP/BS**

**backend analysis****and vis engine**

**monitoring****info provider**

**storage****management**

**disk****cache**

**online****data**

**ESG****node**

**ESG****node**

**ESG****node**

**ESG Gateway (CCES)**

**centralized****metrics services**

**ESG****node**

**ESG****node**

**ESG****node**

**ESG Gateway (CCSM)**

**centralized****security services**

**Analysis Tool**

**browser**

**service API**
**service****implementation**

**client**

**Global sevices**

**ARS mandatory****component**

- ESG-CETarchitecture

**Publication GUI**

## ESG-CET portal: a walkthrough from the user perspective

**Single sign-on login**

**Portal may be customized****for a particular institution.**

**AR5 datasets will be visible ****from all portals.**

## Power search

**User selects search categories ****of interest and *****orders***** them.**

## Search is ‘context sensitive’

**At each step, only relevant ****options are displayed.**

**Search returns:**
**- datasets****- available products for each****  dataset**

## File download

**To download files:**
** ****direct download for online datasets**
** ****scripts generated for wget, bulk download**

## Server-generated products

## Dataset publishing on an ESG node: Metadata specification

**Expansion buttons in left pane ****correspond to publication ****steps.**

**Dataset pane**
** ****shows metadata in a file, allows modification**
** ****is project-specific**

**Output pane**
** ****shows logged results**
** ****log level is configurable**

**Status bar**
** ****shows scan progress**

## Data scan

**Selecting an extraction option ****starts the dataset scan**

**1. Dataset is created or updated ****based on input metadata.**

**2. Files are scanned and internal ****database tables populated.**

**3. If an ‘aggregation dimension’ ****is specified, variables are aggregated.**

## Data aggregation and publication

**Publication step: **
** ****Generate THREDDS catalog ****for harvesting, server-side ****configuration**
** ****Release data for harvesting**

## Intercomparison example

## Future application: workflow

- VisTrails is a new scientific workflow management system.  While originally (and solely) developed by researchers at the University of Utah to provide support for data exploration and visualization, VisTrails now is being applied to climate data analysis and visualization as part of the SciDAC-2 Visualization and Analytics Center for Enabling Technology (VACET) collaboration. The image below shows the use of the visual workflow interface to connect CDAT module boxes to perform calculations and a related plot.

## Client access

**ESG-CET will expose application programming interfaces (APIs) that allow (non-browser) clients to access data in read-only mode.**
**Key design challenges:**
**Authorization and authentication in a distributed environment**
**Client access to files or data aggregations, or both?**
**Eventually want to move the computation closer to the data.**
**Minimize the amount of data movement**
**Workflow, computational grids**
**Clients such as CDAT (PCMDI Climate Data Analysis Tools) will have ESG access built in.**
**Analysis and visalization applications will include CDAT, NCL, Ferret, OPeNDAP clients (e.g., Matlab, NCO)**

## PCMDI is leading climate community data standards development.

**CF: The Climate and Forecast Metadata Convention **
**Designed to **_**enhance utility of data files**_** created with the NetCDF application programmer's interface. **
**Standard name table: enables users of data from different sources to decide which quantities are comparable**
_**Facilitates building applications**_** with powerful extraction, regridding, and display capabilities.**
**PCMDI hosts the CF web site and mailing lists**
**CMOR: Climate Model Output Rewriter**** **
**Used to **_**produce CF-compliant netCDF files**_** that fulfill the requirements of many of the climate community's standard model experiments (such as CMIP, CFMIP, NARCCAP, etc.).**
**CMOR-2 is under development for CMIP5 use.**
**GO-ESSP: Global Organization for Earth System Science Portal**** **
**A collaboration of developers and institutions from the climate and NWP**** ****communities. Focus is on development, information sharing, standards, and interoperability of portals and climate data management software.**

## Collaborations and publications

**NOAA GFDL**** is an active contributor to AR5 and ESG-CET, CF and GO-ESSP**
**SciDac Scientific Data Management Center (LBNL)**
**DataMover Lite**** - efficient bulk transfer of data in a secure grid environment.**
**http://sdm.lbl.gov/sdmcenter/index.html**
**SciDac Visualization and Analytics  Center (VACET)**
**University of Utah, LLNL, LBNL, ORNL; http://www.vacet.org**
**Integration of ****VisTrails**** visual analysis tool with CDAT.**
**Earth System Curator**
**Developing database schemas and interfaces for model configuration.**
_**[http://www.earthsystemcurator.org/](http://www.earthsystemcurator.org/)**_
**Cyberinfrastructure Technology Watch (****CTWatch****) Quarterly**
**D. N. Williams, D. E. Bernholdt, I. T. Foster, and D. E. Middleton, 2007: The Earth System Grid Center for Enabling Technologies: Enabling community access to petascale climate datasets. CTWatch, Vol. 3, number 4.**
**Bulletin of the American Meteorological Society (****BAMS****)**
**D. N. Williams et. al.: The Earth System Grid: Enabling access to multi-model climate simulation data. BAMS (in review).**

## Recap

**PCMDI is continuing to play a lead role in standardization and dissemination of model intercomparison datasets.**
**CMIP5/AR5 is the driving application for ESG-CET**
**ESG-CET is leveraging the success of ESG-II, to:**
**support 2-3 orders of magnitude more data,**
**with richer functionality,**
**for a larger group of users.**
**CDAT is an integral part of an emerging global climate data and analysis infrastructure.**