# Presentation: 041573

##  Facilitating Distributed Climate Modeling Research and Analysis via the Climate Data eXchange 

** ****Facilitating Distributed Climate Modeling Research and Analysis via the ****Climate Data eXchange **

**18 September 2008**
**GO ESSP 2008 Workshop**

**Dan Crichton**
**Chris Mattmann**
**Amy Braverman**

## NASA’s Satellite Data and Climate Research

**Two major legacies from NASA’s Earth Observing System Data and Information System (EOSDIS)**
*Archiving of explosion in observational data in Distributed Active Archive Centers (DAACs)*
- Request-driven retrieval from archive is time consuming
*Adoption of Hierarchical Data Format (HDF) for data files*
- Defined by and unique to each instrument but not necessarily consistent between instruments
**What are the next steps to accelerating use of an ever increasing observational data collection?**
*What data are available?*
*What is the information content?*
*How should it be interpreted in climate modeling research?*

## EOSDIS DAAC’sEarth Observing System Data and Information System Distributed Active Archive Centers

**Notes:**
Data system developed as EOS instruments were planned. Thematic data centers created around the US; JPL has PO.DAAC since we are oceanography remote sensing center for NASA. Others: GSFC, NSIDC (snow and ice), LaRC (atmosphere), Sioux Falls (land surface), ORNL (bio-geo-chemical dynamics/CO2 information center). At the time the concept was to have repositories that support the community by serving requests for copies of data.

## Data Processing Levels

**Notes:**
Within NASA, the data levels are generically defined but volumes and content varies between projects.  Level 3 and 4 are not always created.

## EOSDIS DAAC’sEarth Observing System Data and Information System Distributed Active Archive Centers

**Notes:**
Since the launch of first Terra in 1999, Aqua in 2002, and finally Aqua in 2004 have accelerated the growth in the Level 2 data collections.  NASA’s decadal survey recommendations are, if anything, more data intensive than their predecessors in the EOS era.

## Researcher’s Challenge

**Scientists cannot easily locate, access, or manipulate observational data or model output necessary to support climate research**
*The latest data are available from independent instrument project data systems.*
*Scientists may not even be aware of what repositories or data exist*
*Observational data and model output data are heterogeneous in form and cannot be simply compared or combined.*
**Research data systems are often ad-hoc**
*They lack a modular approach limiting extensibility*
*They are designed individually rather than as a system*
*There are few capabilities in common between systems*
**They require “human-in-the-loop”**
*Web forms, manual ftp transfer*
*Rectification left to individual scientists*

**Notes:**
DAACs function as archives, not research environments. Serve products statically, etc. 

## Current Data System

-  *System serves static data products. User must find move, and manipulate all data him/herself.*
* **User must change spatial and temporal resolutions to match.*
* *_*User must understand instrument observation strategies and subtleties to interpret.*_

## Experience in Planetary Science: NASA’s PDS

- Pre-Oct 2002, no unified view across distributed operational planetary science data repositories
- Science data distributed across the country
- Science data distributed on physical media
- Planetary data archive increasing from 4 TBs in 2001 to 100+ TBs in 2008
- Traditional distribution infeasible due to cost and system constraints
- Mars Odyssey could not be distributed using traditional method
- PDS now has a distributed, federated framework in place
- Support online distribution of science data to planetary scientists
- Enable interoperability between nine institutions
- Support real-time access to distributed catalogs and repositories
- Uniform software interfaces to all PDS data holdings scientists and developers to link in their own tools
- Moving towards international standardization with the International Planetary Data Alliance
- Operational October 1, 2002

***2001 Mars Odyssey***

***PDS Federation***

**Experience in Planetary Science: NASA’s PDS**

## Experience in Cancer Research: NCI’s EDRN

- Provided the NCI with a bioinformatics infrastructure for establishing a virtual knowledge system
- Currently deployed at 15 of 31 NCI Research Institutions for the Early Detection Research Network (EDRN)
- Providing real-time access to distributed, heterogeneous databases
- Capturing validation study results, instrument results images, biomarkers, protocols, etc
- Funded 2001-2010 for NCI’s Early Detection Research Network
- Currently working with a new initiative in establishing an “informatics plan” for the Clinical Proteomics Technology Initiative

- Cancer Biomarkers Group
- Division of Cancer Prevention

**Experience in Cancer Research: NCI’s EDRN**

## CDX

-  **What: build open source software to **
*-- connect existing systems into a virtual network (big disk),*
*-- push as much computation as possible into remote nodes to minimize movement of data,*
*-- operators to rectify and fuse heterogeneous data sets, provide uncertainties.*
** ****Why: ****scientists need command line access to data sets (model output and observations) such that all data look local and rectified.**
** ****How: use technologies in new ways**
*-- distributed computing technologies already in place at JPL (OODT, others); Earth System Grid for parallel transfer,*
*-- rigorous mathematical/statistical methods for interpolation, transformation, fusion, and comparisons. Comparisons require new methods developed specifically for massive, distributed data sets. Uncertainties are key.*
** ****Why is this different: **
*-- system will capture intellectual capital of instrument scientists and modelers through multiple, flexible operators,*
*-- NOT trying to be all things to all people! *

## Climate Data eXchange Research Flow

## Climate Data eXchange Architecture

## Conclusions

**CDX is a paradigm shift in data access/delivery/analysis systems**
*Data analysis should not be decoupled from access and delivery*
*Should support interactive analysis*
**Distributed computing (e.g. web services) architecture is key**
*Support remote query, access, and computation*
*Not tied to any particular implementation*
*ESG is a success story for access and delivery*
*Partnership between JPL and LLNL to extend success to interactive, distributed data analysis*
**JPL will develop, deploy and test V1.0 of CDX over next 18 months**
*Funded NASA support to construct JPL ESG data node*
*Critical components proposed for internal support at JPL to enable model evaluations, validation, and projections*
**Feedback, suggestions, and collaborations welcome on path forward**

**Notes:**
We will create version 1.0 of what we think is an essential development needed to enable blah blah. We welcome collaboration and contributions from DOE national labs. Climate model simulation output is even bigger than observational set. Modelers typically reduce these output data sets to coarse resolution summaries because the raw output is too big. Destroys important high resolution information that is comparable to observations. 

## Backup

## Climate Research Use Case

**What is radiative effect of the vertical distribution of water vapor in the atmosphere under clear-sky conditions?**
*Warming by water vapor back to the surface could lead to increased evaporation and accelerate (positive feedback) the “greenhouse effect”*
**Investigation and validation of climate model representations of water vapor distributions can be made by comparison to both AIRS and MLS measurements of water vapor**
*AIRS provides water vapor measurements up to 200 mb (15km)*
*MLS  provides water vapor measurements from 300 mb to 100 mb (8km to 18km)*
*AIRS and MLS sample different states: each is capable of  measuring vapor in clear scenes, but under cloudy conditions they have different biases. *
*Need to combine these data to get the full picture.*

## Combining Instrument Data to enable Climate Research: AIRS and MLS

**Combining AIRS and MLS requires:**
*Rectifying horizontal, vertical and temporal mismatch*
*Assessing and correcting for the instruments’ scene-specific error characteristics**  (see left diagram)*