# Presentation: 077973

## Data Management and Analysis for the Earth System Grid

**SciDAC 2008 Conference**
**Seattle, Washington**

- Climate change is not only a scientific challenge of the first order but also a major technological challenge. The international climate community is expected to generate hundreds of petabytes of simulation data within the next three to five years.

## Data management and analysis for the Earth System Grid

**Data management and analysis for the Earth System Grid**

- ESG’s mission is to provide climate researchers worldwide with access to:
- data,
- information,
- models,
- analysis tools, and
- computational capabilities required to make sense of enormous climate simulation datasets.
- ESG’s goals
- make data more useful to climate researchers by developing Grid technology that enhances data usability,
- meet specific distributed database, data access, and data movement needs of national and international climate projects,
- provide a universal and secure web-based data access portal for broad multi-model data collections, and
- provide a wide-range of Grid-enabled climate data analysis tools and diagnostic methods to international climate centers and U.S. government agencies.
- develop key ideas and concepts that are important contributions to other domain areas.

## What is climate change and why is it important?

**What is climate change and why is it important?**

- Climate change is the warming of the Earth’s climate due to increased greenhouse gases.
- Carbon dioxide (CO2),
- Methane (CH4),
- Nitrous oxide (N20)
- Hydorflurocarbons (HFCs),
- Perfluorocarbons (PFCs), and
- Sulphur hexafluoride (SF6)
- Making up ~1 per cent of the atmosphere, greenhouse gases trap radiant heat from the Earth and help warm the planet to sustain life.
- Is this too much of a good thing?
- Current climate models predict global warming of 1.4 to 5.8 **O**C between 1990 and 2100. Based on:
- Future emissions (population growth, technology change, etc.)
- Does not assume climate change policies for reducing emissions
- These projections take into effect aerosols and delaying effects of the oceans
- Earth's surface and lower atmosphere would continue to warm for hundreds of years even if greenhouse gas concentration stopped rising in 2100
- The average sea levels are predicted to rise 9 to 88 cm between 1990 and 2100

## The progression of climate models

**The progression of climate models**

- To help understand climatic changes and potential impacts, scientist rely on powerful supercomputers and complex computer codes to simulate the Earth’s climate.
- By running exhaustive computational experiments aimed at studying the behaviors of individual chemical, biological, and geophysical processes, as well as relationships among these individual processes.
- Climate models are also used to study regional climate changes as well as high-impact characteristics of climate such as typhoons, droughts, and floods.
- Accurately resolving these and other processes, and discovering long-term patterns such as temperature distribution or the melting of land and sea ice, requires the application of intricate mathematical formulas to data at points on ever finer grid resolutions.

## Data integration challenges facing climate science

- Modeling groups will _generate more data_ in the near future than exist today
- Large part of research consists of _writing programs_ to analyze data
- How best to collect, distribute, and find data on a much _larger scale_?
- At each stage tools could be developed to _improve efficiency_
- Substantially more ambitious community modeling projects (Petabyte (PB 1015) and Exabyte (EB 1018)) will require a _distributed database_
- Metadata describing _extended modeling simulations_ (e.g.,  atmospheric aerosols and chemistry, carbon cycle, dynamic vegetation, etc.) (But wait there’s more: economy, public health, energy, etc. )
- How to make information understandable to end-users so that they can _interpret the data_ correctly
- _More users_ than just Working Group (WG) I-science. (WGII-impacts and WGIII-mitigation) (Policy makers, economists, health officials, etc.)
- Integration of _multiple_ analysis tools, formats, data from unknown sources
- _Trust and security_ on a global scale (not just an agency or country, but worldwide )

**Earth System Grid Center for Enabling Technologies: (ESG-CET)**

## Earth System Grid Center for Enabling Technologies (ESG-CET)

**Petabyte-scale**** data volumes**
**Globally**** federated sites**
**“****Virtual Datasets****” created through **
**   ****subsetting and aggregation**
**Metadata-based**** search and discovery**
**Bulk ****data access**
**Web-based and analysis tool**** access**
**Increased ****flexibility and robustness**

**ESG Goals**

**Current ESG Sites**

**http://www.earthsystemgrid.org**
**http://www-pcmdi.llnl.gov**

**Earth System Grid Center for Enabling Technologies (ESG-CET)**

## Data
Different formats not standardized 
Different sites require knowledge of different methods of access
Log onto multiple sites to hopefully find and retrieve data
Gigabyte (GB 109) - Terabyte (TB 1012) data volume
Metadata
Painful to produce
Most kept in files separate from data
Data lost or reproduced numerous times
Locating data
Manual
Unreachable unless one is “in the know” (location kept in someone’s brain)
Not formalized
Data requests/analysis
Beginnings of a formal process 
Far too much done by hand
Logging nearly non-existent

- Different formats not standardized
- Different sites require knowledge of different methods of access
- Log onto multiple sites to hopefully find and retrieve data
- Gigabyte (GB 109) - Terabyte (TB 1012) data volume
- Metadata
- Painful to produce
- Most kept in files separate from data
- Data lost or reproduced numerous times
- Locating data
- Manual
- Unreachable unless one is “in the know” (location kept in someone’s brain)
- Not formalized
- Data requests/analysis
- Beginnings of a formal process
- Far too much done by hand
- Logging nearly non-existent

- Data
- Standard output
- Model compliance tools to facilitate standard output
- Quality assurance
- Different sites but standardized access protocol
- One stop shop
- Terabyte (TB 1012) - Petabyte (PB 1015) data volume
- Metadata
- Exhaustive detail
- Created via semi-automated processes
- Put data in databases and make it visible to others
- Locating data
- Formalized process
- Highly granular – down to per-file, per-model level, per-variable
- Readily searchable - sophisticated search tools
- Data requests/analysis/visualization
- Completely automated
- All logging done automatically
- Secure

**Before ESG**

**ESG Present, Future**

**Computers do the more complicated and repetitive tasks  and scientists **_**focus on research**_

**Tremendous manual intervention, **_**inefficient**_** by any measure**

**Climate model data management and analysis issues**

**Earth System Grid Center for Enabling Technologies: (ESG-CET)**

## Current ESG architecture and underlying technologies

- Climate Data
- Metadata Catalog
- NcML (metadata schema)
- OPeNDAP-g (aggregation and subsetting)
- Data Management
- _S_torage _R_esource _M_gr
- Data Transfer
- _G_lobus _S_ecurity _I_nfrastructure
- _D_ata _M_over _L_ite
- GridFTP
- Monitoring and Discovery Services
- _R_eplica _L_ocation _S_ervice
- Security
- Access Control
- MyProxy
- User Registration
- Long-term Storage
-  Tertiary data storage systems

**Earth System Grid Center for Enabling Technologies: (ESG-CET)**

## Current ESG data archives

- Several archives holding over 200 TB
- 9,000+ registered users
- 60 TB downloaded

- Currently archiving 35 TB from 25 models
- 2,000+ registered projects
- 400+ scientific papers
- ~1/2 PB downloaded -- averaging 400 to 600 GB/day

**ESG CMIP3 (AR4) Archive Facts**

**ESG CCSM Archive Facts**

**ESG Usage Internationally**

**Over 500 sites worldwide**

**Current ESG data archives**

**ESG Data Download Volume per Month**

## Use Case 1: scientific metadata search

- “Find surface temperature data across all models for a specific IPCC experiment that has volcanic forcing.”

**Earth System Grid Center for Enabling Technologies: (ESG-CET)**

- Capture Scientific Metadata in detailed object model
- Faceted Search to slice through data via user-selected categories
- Link to Data Access Points (files or products)
- Broker application for one-click request of data products?

## Use Case 2: large number of files

**Use Case 2: large number of files**

- “Download 1000 files from deep storage to my desktop while I am sleeping.”

**Earth System Grid Center for Enabling Technologies: (ESG-CET)**

- Gateway interaction via Web Services:
- User finds, requests data through Gateway
- User passes request identifier to DML
- DML downloads file as they become available
- DML “releases” files already downloaded
- User checks request status via DML (or Gateway)

## Use Case 3: high-end product, federation

- “Show me sea surface temperature plots for 3 different datasets (output of different models, same forcing) that are stored at 3 different locations.”

**Earth System Grid Center for Enabling Technologies: (ESG-CET)**

- More powerful data selection algorithm
- Integration of LAS product server on Gateway, Data Nodes
- Single Sign-On authentication via OpenID
- Common authorization model

## Use Case 4: intercomparison example

**Earth System Grid Center for Enabling Technologies: (ESG-CET)**

## Much broader model for scientific metadata
‘Faceted’ search capability guides the user toward datasets of interest
At a given point in the search, only those options which produce non-empty result sets are shown
Avoids ‘deadend’ searches
Flexible browsing hierarchy
Automated, GUI-based publication tools
Single sign-on
Full support for data aggregations
A collection of files, usually ordered by simulation time, that can be treated as a single file for purposes of data access, computation, and visualization
File-streaming and release capabilities for data access to deep storage
Client access to subsetting, visualization services
Server-generated visualization products
Fine-grained access to datasets based on user groups and roles.
User notification service
Users can choose to be notified when a dataset has been modified
Pre-computed products (e.g., global averages)
User workspace (storing of favorite products, search criteria, etc.)

- _Much broader _model for scientific metadata
- _‘__Faceted’ search_ capability guides the user toward datasets of interest
- At a given point in the search, only those options which produce non-empty result sets are shown
- Avoids ‘deadend’ searches
- Flexible browsing hierarchy
- Automated, GUI-based _publication tools_
- _Single sign-on_
- Full support for _data aggregations_
- A collection of files, usually ordered by simulation time, that can be treated as a single file for purposes of data access, computation, and visualization
- _File-streaming_ and release capabilities for data access to deep storage
- _Client access_ to subsetting, visualization services
- _Server-generated visualization_ products
- _Fine-grained access to datasets_ based on user groups and roles.
- User _notification service_
- Users can choose to be notified when a dataset has been modified
- Pre-computed products (e.g., global averages)
- _User workspace_ (storing of favorite products, search criteria, etc.)

**ESG-CET improvements over ESG II**

## Architecture of the next generation of ESG-CET

**Architecture of the next generation of ESG-CET**

## The next generation ESG-CET system

**Earth System Grid Center for Enabling Technologies: (ESG-CET)**

**• ****Distributed and federated architecture **
**• ****Support discipline-specific Gateways **
**• ****Support browser-based + direct client access**

## Immediate and future challenges of software development

**Immediate and future challenges of software development**

- _Sustain and build_ upon the very successful ESG archives (e.g., CCSM, CMIP3, CFMIP, PCM, POP, etc.)
- Address _future scientific needs_ for data management and analysis by extending support for sharing and diagnosing climate simulation data
- SciDAC II: *A Scalable and Extensible Earth System Model for Climate Change Science*
- Coupled Model Intercomparison Project, Phase 5 (CMIP5) for scientists contributing to the IPCC Fifth Assessment Report (AR5) in 2010,
- The Climate Science Computational End Station (CCES),
- The North American Regional Climate Change Assessment Program (NARCCAP), and
- Other wide-ranging climate model evaluation activities.
- How to _make information understandable_ to end-users so that they can interpret the data correctly
- Local and remote analysis and visualization tools in a _distributed environment_ (i.e., subsetting, concatenating, regridding, filtering, ...)
- Integrating analysis into a distributed environment
- Providing climate diagnostics
- Delivering climate component software to the community

## Animation of AR4 model ensemble of “20c3m” (1880-1999) and “21st century” (2000-2100) experiment showing progressive warming of the planetary surface as greenhouse gases concentrations increase

**Earth System Grid Center for Enabling Technologies: (ESG-CET)**