# Presentation: 370849

## The NOAA Operational Model Archive and Distribution System  NOMADSGlenn K. RutledgePrincipal Investigator Workshop on CLASS Status and Plans Suitland MD Dec 11-12 2003 

## Overview

-  Retrospective analysis and model inter-comparison are necessary to verify and improve short term NWP models, seasonal forecasts, climate simulations, assessment, and detection efforts.

-   University and Institutional research goes largely untapped by NOAA scientists. Effort is wasted on data receipt and format issues with no infrastructure to collaborate.

*Overview*

## Overview

-  **In 2000 NCDC, NCEP and GFDL initiated the NOAA Operational Model Archive and Distribution System. **
** ****NOMADS is a distributed data services pilot for format independent access to climate and weather models and data.**

*Overview*

## A Collaboration for Success

_**Core Collaborators**_
- CDC
- COLA
- FSL
- GFDL
- LLNL
- NCAR
- NCDC (PI)
- NCEP
- PMEL
- Unidata

_**Participating Collaborators**_
- CEOS (DataGrid)
- ESMF (for code standards)
- NASA/GCMD
- NASA/SI-ESIP (GMU)
- NSSL w/UW/SSEC
- NWS/COMET/HQ
- U. Of Alabama, Huntsville*
- U. of Washington

*A Collaboration for Success*

- DOE

- NCAR

- Unidata

- *  Exploring ESML for RADAR Data Mining

## NCDC NOMADS

*NCDC NOMADS*

## Core Collaborators:  FSL

*Core Collaborators:  FSL*

## Core Collaborators: NCEP

*Core Collaborators: NCEP*

## Core Collaborators: GFDL

*Core Collaborators: GFDL*

## Core Collaborators: NASA

*Core Collaborators: NASA*

## Core Collaborators: Unidata

*Core Collaborators: Unidata*

## International Collaborators

-   British Atmospheric Data Center & NERC Grid & e-Science
-   Climate Action Partnership" (CAP) under Commerce, Energy,
-     State, and EPA with Australia.
-   The GCOS, GODAE, US-GOOS, HAOC, NVODS Projects
-   European PRogram for Integrated Earth System Modeling
-    (PRISM) Project and US ESMF projects.
-   Committee of Earth Observation Satellites (CEOS)
-   Coordinated Enhanced Observing Period (CEOP)

*International Collaborators*

## NOMADS Momentum 

*NOMADS Momentum *

- Recent Invited Briefings include:
-  NOAA Climate and Global Change WG
-  NOAA Climate Monitoring WG
-  NOAA Science Advisory Board
-  NVODS/IOOS/GCOS
-  NWS Office Director and NWS CIO
-  NASA
-  Earth Science Portal (ESP- an International data
-    portal collaboration)
-  NOAA Tech and Web forums

## NOMADS Data Availability

- CDC:     Reanalysis, climate weather models, in-situ
- GFDL:   Coupled Models, Control and Perturbation
-  	 Integrations and historical 20th century simulations
- 	 using solar, volcano, GHG and aerosol forcings.
- FSL:      MADIS, Hi-Res RUC-II
- NCAR:  Community Climate System Model / Land Surface
-               CO2 predictive models (VEMAP), Reanalysis /  Eta
- NCDC:  Select observational datasets; LTA for NCEP/GFDL
- 	 Ocean Models, Ensembles, Reanalysis (NARR), SST’s
- NCEP:   R/T Model Input/Output, Reanalysis (I&II),  WRF
- LLNL:   AMIP / Probabilistic information
- PMEL:   Ocean and Climate datasets

*NOMADS Data Availability*

## Why Models?

- climate model output statistics from NWP run in quasi-real time to identify time-dependant biases in observations;
- assess the affect of missing data;
- for observing network design and operation, climate models can be used to guide the spatial and temporal sampling frequency to resolve distributions for specific variables (temp, precip).

*Why Models?*

-   NOMADS promotes systematic model evaluation and model inter-comparison; and a feedback mechanism from research to operations.

## Why Scientific Data Networking? 

*Why Scientific Data Networking? *

-  The users experience is often frustrating—
-   What data of interest exist?
-  Are they going to be useful to me?
-  How can I obtain them in a usable form?
-  Time and effort are wasted on data access and
-    inter-comparison issues.
-   As a result atmosphere/ocean/climate data
- are under-utilized.

## NOMADS is an agreement between agencies who participate to have common ...

- Data and observation distribution software, format independent and description  methods (metadata).
- Documentation and organization framework.
- Forum to plan and organize (science drivers).
- Funding avenue to direct $ for intra-Agency partnerships
- ... for university, federal agencies, and organizations, a mechanism to obtain support for the dissemination of their data sets.

*The NOMADS Framework*

## NOMADS Framework (cont) 

*NOMADS Framework (cont) *

-   NOMADS uses the Open Source XML based OPeNDAP
-  OPeNDAP is a binary-level protocol designed for the transport
-     of scientific data subsets over the Internet.  Through OPeNDAP
-     applications can open and read subsets from distributed
-     locations.
**  **Formats: FGDC, COARDS, CF, DIF, GRIB, GRIG2, BUFR,
-     HDF, NetCDF, others.....libraries built as necessary.
-   API’s: JAVA-OPeNDAP, C++-OPeNDAP, NetCDF, GRIB,
-    BUFR, THREDDS, Python.

## OPeNDAP Protocol

*OPeNDAP Protocol*

*Utilize Binary data and metadata through OPeNDAP ...*

## OPeNDAP Clients/Servers

- Access to remote data in the users normal application
- IDL / IDV
- Matlab
- Ferret
- GrADS (GRIB thru GDS)
- Any netCDF application i.e., AWIPS
- Netscape / Excel / etc.
- CDAT (PCMDI)
**Don’t need to know the format in which the data are stored.**

**Data publishing (server)**
- Can serve data in various formats
- netCDF / GRIB / (BUFR) (Grib2)
- HDF (3-5)
- SQL
- FreeForm
- JGOFS
- DSP
- ascii
**Data subsetting and host side computations on the fly.**

- NOMADS simplifies scientific data networking,   allowing simple access to remote data.

*OPeNDAP Clients/Servers*

## NOMADS Design:  Interoperability 

*NOMADS Design:  Interoperability *

- The “Grids”

- Rutledge 12/03

- The Grids include: CEOS-DataGrid,
- DOE’s ESG, NASA’s IPG, ADG, &
- UK’s NERC DataGrid

## OPeNDAP and Live Access Server

*OPeNDAP and Live Access Server*

**GDS**

## OPeNDAP and Grads-DODS Server

*OPeNDAP and Grads-DODS Server*

- GRIB, HDF

- NetCDF

- GrADS binary

- etc..

**Datasets in any format supported by GrADS**

- Result cache

**For GRIB and BUFR users, multi-dimensional data (space/time), and data subsets are extracted on the fly.**

- GrADS batch mode

- interface code

**DODS*** server libraries

- Server

**performs analysis operations**

**manages sessions, translates dataset names**

- Java servlet

**supports extended request types for analysis, upload**

- Encapsulated Analysis Requests

- Station data

- BUFR

- Internet

**LAS**

## NOMADS Core
Red Hat (tbd)
OPeNDAP (DODS)
GrADS-DODS Server -GDS
Live Access Server -LAS
LAS-Climate Data Analysis Tools (from LLNL)
Globus (Earth System Grid -ESG

- Red Hat (tbd)
- OPeNDAP (DODS)
- GrADS-DODS Server -GDS
- Live Access Server -LAS
- LAS-Climate Data Analysis Tools (from LLNL)
- Globus (Earth System Grid -ESG

*Production and Dvlp Applications *

## NOMADS currently leverages/collaborates with:
            CAP (NOMADS is a participant via Dept of State)
		CEOP (Coordinated Enhanced Observing Period)
		CEOS (Committee of Earth Observation Satellites) 		DMAC (collaborator via Hankin)
		ESP (founding member) 
		European PRISM and CAPRI (on Advisory Council)
		GODAE (collaborator)
		NASA GCMD and other efforts
		NERC DataGrid (on Advisory Committee)
		NSF Cyberinfrastructure (member)
		Unidata THREDDS  USWRP, NSDL, DLESSE
						The point is.....

**            ****CAP** (NOMADS is a participant via Dept of State)
- 		**CEOP** (Coordinated Enhanced Observing Period)
**	****	****CEOS** (Committee of Earth Observation Satellites) **	****	****DMAC** (collaborator via Hankin)
- 		**ESP** (founding member)
**	****	****European** PRISM and CAPRI (on Advisory Council)
**	****	****GODAE** (collaborator)
- 		**NASA GCMD** and other efforts
- 		**NERC DataGrid** (on Advisory Committee)
- 		**NSF Cyberinfrastructure** (member)
**	****	****Unidata THREDDS**  USWRP, NSDL, DLESSE
- 						The point is.....

*Existing and Collaborating Programs *

## Work with the broader development community and users to:

Define the functionality, tools, and science driven data requirements of the CLASS-NOMADS relationship based upon format neutral interoperability of OPeNDAP. 

By embracing the OPeNDAP CLASS becomes another member of the community– and compatible with existing NOMADS partners.  

- Define the functionality, tools, and science driven data requirements of the CLASS-NOMADS relationship based upon format neutral interoperability of OPeNDAP.
- By embracing the OPeNDAP CLASS becomes another member of the community– and compatible with existing NOMADS partners.

*Coordination of Development*

## What’s Next? (cont.)  

*What’s Next? (cont.)  *

-  DMAC, GODAE and NOMADS:  A template for CLASS?
-  WCRP supports NOMADS:  Incorporating NOMADS “Philosophy”
-    will advance International partnerships for CLASS data sets.
-  Long term “Bigger Pipe” options:  Earth System Grid.  Co-PI on
-   ESG is co-PI on NOMADS.
-  Agency-wide support for NOAA LO’s and non-profits involved in
-   the OPeNDAP consortium.
-  Leverage existing programs and collaborations into CLASS:
-    e.g., Earth System Portal (ESP):  _[http://](http://esportal.gfdl.noaa.gov/)__[esportal.gfdl.noaa.gov](http://esportal.gfdl.noaa.gov/)_

**Notes:**
The ESP is a superset of the original NOAMDS partners.  The goal is a common portal to model and observational datasets.  Key players involved.  See web site.  

## CLASS can collaborate by embracing OPeNDAP.  

Support NOMADS and NOMADS partners to ensure for data and metadata interoperability; and that search and use metadata are appropriate. 

Result
A NOMADS-CLASS Synergy will be established.

- Support NOMADS and NOMADS partners to ensure for data and metadata interoperability; and that search and use metadata are appropriate.
- Result
- A NOMADS-CLASS Synergy will be established.

*Recommendations   *

## See NOMADS at:
	Access:
	ncdc.noaa.gov/oa/model/model-resources.html

	NOMADS Program:
	ncdc.noaa.gov/oa/climate/nomads/nomads.html

			Glenn.Rutledge@noaa.gov

	NCDC Main Page  Climate    Model Resources

- 	_Access:_
- 	ncdc.noaa.gov/oa/model/model-resources.html
- 	_NOMADS Program:_
- 	ncdc.noaa.gov/oa/climate/nomads/nomads.html
- 			_[Glenn.Rutledge@noaa.gov](mailto:Glenn.Rutledge@noaa.gov)_
- 	NCDC Main Page  Climate    Model Resources

*More Information *

## QUESTIONS? 

*QUESTIONS? *