# Presentation: 690585

##  The NOAA Operational Model Archive and Distribution System		      NOMADS

- 	     	Overview and Plans
- 		       Glenn K. Rutledge
- 	   	          NOMADS  PI
-             National Oceanic and Atmospheric Administration
- 	              National Climatic Data Center
- 	              	      	 June, 2004

## Overview

-  Retrospective analysis and model inter-comparison are necessary to verify and improve short term NWP models, seasonal forecasts, climate simulations, assessment and detection efforts.

-   University and Institutional research goes largely untapped by NOAA scientists. Effort is wasted on data receipt and format issues with no infrastructure to collaborate.

*Overview*

## Overview

-  NCDC, NCEP and GFDL initiated the NOAA Operational Model Archive and Distribution System.
-  NOMADS is a distributed data services pilot for format independent access to climate and weather models and data.

*Overview*

## NOMADS Goals

- communities (ocean, weather, and climate)
- to study multiple earth systems using
- collections of distributed data,

-   promote product development,

-   expand institutional participation via distributed technologies.

-   The goals of NOMADS are to:

-   provide access to models,

*NOMADS Goals*

## The NOMADS Philosophy

- weather and climate information:  web browsers to advanced scientific data visualization packages.  NOMADS is a flexible approach to data access- promoting standards, based upon what users use most.
-   As users require new data types, under NOMADS, they can be added or replaced.  Over the years, NOAA has developed several different data conventions and data access protocols.  NOMADS allows the use of all these most widely used formats.
-   This will allow the users to make better, informed decisions about
- how nature will impact their future, either in their life, or business decisions.

*The NOMADS Philosophy*

## NOMADS is an agreement between agencies who participate to have common ...

- Data and observation distribution software, format independent and description  methods (metadata).
- Documentation and organizational framework.
- Forum to plan and organize (science drivers).
- Funding avenue to direct $ for intra-Agency partnerships
- ... for university, federal agencies, and organizations, a mechanism to obtain support for the dissemination of their data sets.

*A Distributed Framework*

## So What?  
		  	“But I just want the Data”
  The idea of pulling information, not bytes or data, is new and hard to convey
NOMADS is a pull technology.  Users can become data servers.    
Why transport millions of files if only a subset is needed? 
Will networks ever keep up with growing data sets? 
Data Management at the grass roots level with science driven requirements.
Many efforts in distributed access.  How best to coordinate efforts? 
Vision, planning, and cooperation needed w/ Agency attribution!
A dynamic system, not over engineered and built upon existing users needs but extensible for future requirements.  

**	****	****  ****	*****“******But I just want the ***_***Data***_***”***
***  ***The idea of pulling _information_, not bytes or data, is new and hard to convey
- NOMADS is a pull technology.  Users can become data servers.
- Why transport millions of files if only a subset is needed?
- Will networks ever keep up with growing data sets?
- Data Management at the grass roots level with science driven requirements.
- Many efforts in distributed access.  How best to coordinate efforts?
- Vision, planning, and cooperation needed w/ Agency attribution!
- A dynamic system, not over engineered and built upon existing users needs but extensible for future requirements.**  **

*Distributed Framework (cont.)*

## Scientific Data Networking?

-   	- What data of interest exist?
-   	- Are they going to be useful to me?
-   	- How can I obtain them in a usable form?
-    Time and effort are wasted on data access and format
-      issues.
-   As a result atmosphere/ocean/climate data are under-
-     utilized.  Model inter-comparison nearly impossible.

*Scientific Data Networking?*

## Scientific Data Networking...

- Access to remote data in the users normal application
- IDL / IDV / Matlab / Ferret
- GrADS (GRIB/BUFR w/ GDS)
- Netscape / Excel / http (wget)
- CDAT (PCMDI)
**Any netCDF application**      (i.e., AWIPS)
**Don’t need to know the format in which the data are stored.**

**Data publishing (server)**
**Can serve data in various formats**
- netCDF / GRIB / BUFR / GRIB2
- HDF (3-5) / EOS
- SQL / FreeForm
- JGOFS / NcML
- DSP
- ascii, others...
**Spatial and temporal sub-setting and host side computations on the fly.**

- NOMADS simplifies scientific data networking, allowing simple access to high volume remote data, unifying access to Climate and Weather models:

*Scientific Data Networking...*

## The Partnerships

- 	CDC
- 	COLA   co-PI
- 	FSL
- 	GFDL   co-PI
- 	LLNL
- 	NCAR
- 	NCDC   PI
- 	NCEP    co-PI
- 	PMEL    co-PI
- 	Unidata

**	**BADC (UK)
- 	CEOP
- 	CEOS
- 	LEAD & GTF (NSF)
- 	NASA GCMD & ESIP
- 	NERC DataGrid
- 	NSSL w/UW/SSEC
- 	United Nations FAO

*The Partnerships*

## Collaborating Programs

*Collaborating Programs*

- CAP     Climate Action Partnership  	       	DOC DOE EPA State Dept
- CDP     Community Data Portal 			NCAR
- CEOS   Committee on EO Satellites 		NOAA Representative
- CEOP   Coordinated Earth Obs Period    	NOAA Representative
- EPA     Air Quality Models 			(in progress)
- ESP      Earth Science Portal	 		Founding Member
- NASA GCMD    				Science Advisory Board
- NERC DataGrid 				Advisory Committee
- NSF Cyberinfrastructure 			Member
- NSF LEAD & Geo-Science Tech Forum	 (GTF)  Data / Planning Committee
- NVODS / US GODAE / GOOS			Data Provider
- Unidata THREDDS,  NSDL, DLESSE		Data Provider
- WCRP World Climate Research Program  	JSC/CLIVAR Briefings

## NOMADS is a founding member of the Group Organized for Earth Science Portal (GO-ESP).

ESP is a collaboration designed to build the infrastructure needed to create web portals to provide access to observed and simulated data within the climate and weather communities. 

The infrastructure created within ESP will provide a flexible framework that will allow interoperability between the front-end and back-end software components. ESP is an international collaboration involving software developers from both Europe and the United States.               See http://esportal.gfdl.noaa.gov/

- ESP is a collaboration designed to build the infrastructure needed to create web portals to provide access to observed and simulated data within the climate and weather communities.
- The infrastructure created within ESP will provide a flexible framework that will allow interoperability between the front-end and back-end software components. ESP is an international collaboration involving software developers from both Europe and the United States.               See http://esportal.gfdl.noaa.gov/

*A Bigger Picture*

## Uses

- Collaboration between Global Climate Model (GCM) and NWP
-     researchers using large data volumes of data.
- Assess the affect of inadequate spatial and temporal sampling.
- Models can be used to guide the spatial and temporal sampling frequency for observing network design and operation to resolve distributions for specific variables.
- Accurate estimates of future climate variability and trends.
- Long-term protection of climate simulations and NWP analysis.

*Uses*

## Uses (cont.)

- Departure of observations from NWP and a fixed reference
-     climatological background can help identify time-dependent
-     changes in the observations.
-   A Numerical Weather Prediction (NWP) re-run capability.
-    Model input Data Assimilation fields for Regional model
-    initialization (e.g., WRF, MM5,) and Climate Models.
-   Analysis of historical NWP for operational forecaster training.
-   Third-world internet access to NWP for forecasting operations.
-   Subsets of high volume NWP and GCM avbl over the Internet.

*Uses (cont.)*

## The NOMADS Philosophy

*The NOMADS Philosophy*

_**Multiple paths to format independent data access:**_

## GrADS Data Server- GDS

- GRIB, HDF

- NetCDF

- GrADS binary

- etc..

**Datasets in any format supported by GrADS**

- Result cache

**For GRIB and BUFR users, multi-dimensional data (space/time), and data subsets are extracted on the fly.**

- GrADS batch mode

- interface code

**DODS*** server libraries

- Server

**performs analysis operations**

**manages sessions, translates dataset names**

- Java servlet

**supports extended request types for analysis, upload**

- Encapsulated Analysis Requests

- Station data

- BUFR

- Internet

*GrADS Data Server- GDS*

- Source: B. Doty (COLA) & G. Rutledge

- ** using NetCDF interface

## Proposed NCDC Install of LAS

*Proposed NCDC Install of LAS*

**GDS**

- * GOAL: Retrieving and Using GRIB/BUFR through GDS and LAS

- Modified from Hankin ‘03

- *

## Framework

-   OPeNDAP is a binary-level protocol designed for the transport
-     of scientific data subsets over the Internet.   Provides server
-     side data manipulation on-the-fly (e.g., GrADS-DODS).
-   Data formats: GRIB, GRIB2, BUFR, HDF, NetCDF, ascii...
-    Conventions: COARDS, CF, FGDC, DIF....libraries built as
-    necessary.
-   APIs: JAVA-OPeNDAP, C++-OPeNDAP, NetCDF, GRIB,
-    BUFR, THREDDS, Python.

*Framework*

## Framework (cont.)

*Utilize Binary data and metadata through OPeNDAP ...*

*Framework (cont.)*

## Data Availability Overview

- GFDL:  Coupled Models, Control and Perturbation
-  	 Integrations and historical 20th century simulations
- 	 using solar, volcano, GHG and aerosol forcings.
- FSL:      MADIS mesoNets, Hi-Res RUC-II
- NCAR:  Community Climate System Model / Land Surface
-               CO2 predictive models (VEMAP), Reanalysis /  Eta
- NCDC:  Archive for NCEP model input/output / Select NCDC
-               Observation datasets, Ocean/Ice WAVE, NARR, SST’s...
- NCEP:   Real-time Input/Output, Reanalysis (I&II),  Ensembles, Sea
-               Ice Ocean, CDAS, Hourly Eta, Climate Forecast Models...
- LLNL:   AMIP / Probabilistic information
- PMEL:   Ocean and Climate datasets

*Data Availability Overview*

## NCDC and NCEP Data

*NCDC and NCEP Data*

-   NCDC NOMADS Archive
-  NWP from NCEP
-  POR: 2002 to Real-Time
-  Eta (12km); GFS (1 degree); GDAS; NARR 12km 30yrs
-  RUC-II 20/40km; Ocean and Ice WAVE Models
-  NCDC Reference Data Sets (Reynolds SST’s, GHCN...)
-  NCDC Mirror site to NCEP NOMADS for Eta & GFS
-  NCEP Real-Time NOMADS
-  Global Forecast System GFS 1 degree
-  Hourly Eta at 12km
-  Regional Spectral Model (RSM) and Ensembles
-  Climate Data Assimilation System (CDAS)
-  AMIP Climate Monitoring, Climate Forecast Model
-  NCEP/NCAR Global Reanalysis 1&2

## Model Input: NCEP GDAS

-   model output products as close as possible to NCEP operations.
-  The analysis files will be in the models own coordinate system.
-  Files are constructed with computer and computational efficiency
-   in mind, and not in standard coordinate systems.
-  Programs to convert  these files are available upon request:
-  spectral to gaussian
-  gaussian to lat/lon
-  sigma to pressure

*Model Input: NCEP GDAS*

## NCEP GDAS (cont.)

*NCEP GDAS (cont.)*

-   The minimum set for Global Spectral Forecast Model and the Spectral
- Statistical Interpolation Cycling Analysis System contains ~0.5Gb /run:

- - NOAA-15/16 AMSU-A/B TOVS 1B Radiances (IEEE)
- - Analysis Bias Corrected Information / Obs Toss List
- - SFC U/A, ACRS, Aircft (BUFR)
- - 6HR fcst guess from previous run (BUFR)
- - ERSCAT Sat obs / HIRS 14/15, MSU TOVS (IEEE)
- - Guess prep and and fcst guess output (BUFR)
- - Analysis ready QC’ed Obs. (prepBUFR)
- - Profiler, TOVS, Wind Obs. (BUFR)
- - SFC Analysis Restart Files
- - SST’s (GRIB), Radar VAD Winds (BUFR)

## NOMADS Archive and Users

*NOMADS Archive and Users*

-   Data Philosophy and Retention
-   Data are free.
-   NWP forecast data are retained for five years.
-   Analysis, Reanalysis, observations, and GDAS model input are retained for long term stewardship.
-   Data Users
-  Resolution of IP addresses indicate a broad range, and
-    consistent use of NOMADS available data:
-   U.S. Agencies, Academic Institutions: K-12 to Research
-   International governments, (Italy, Japan, countries within
-    South America and Africa.  Many others).
-   Private Sector and Non-Government Organizations NGO’s
-   World Bank, United Nations (FAO), others.

## NOMADS Archive and Users (cont.)

*NOMADS Archive and Users (cont.)*

- Existing and Projected Volume

- * 5-YR retention
-   of fcsts.  Long
-   term for anal.

- May 2004

## NCDC System Architecture

- NOAAPort

- Unidata IDD

- NCEP ftp

** ****Data & Directory **
**   ****structures “merged”**
** ****Daily Data Ingest **
**   ****inter-comparison **
** ****QC and R/T Monitoring**
** ****Index File generation**
** ****Control and OPeNDAP **
**   ****metadata generation  **
** ****CVS Backup (code)**
** ****NCDC Archive Interface**

** ****Earth System **
**Grid & **
**CEOS-Grid**

**Obs, Eta, **
**GFS, RUC**

**Hi-Res**
**GFS, Eta, **
**NARR and**
**GDAS**

_**Data Ingest**_

**Dual**
**Redundant**
**Ingest**

_**Data Management**_

_**Data Access**_

**NCDC**
**Archive**

**NOMADS**
**Web/DODS**

*NCDC System Architecture*

- GigaPOP

## Dynamic Ingest Monitoring

*Dynamic Ingest Monitoring*

- Dual Site Ingest and Dynamic error reporting for a serially complete archive.  QC architecture discovered NOAAPort labeling errors.

## NOMADS Main Page

*NOMADS Main Page*

## NCDC Web Interface

*NCDC Web Interface*

- Three primary
- methods for data
- access:
-  Web Interface
-  OPeNDAP
-  ftp w/ on the fly
-   Grib subsetting
- On-line or
- Off-line (archive)
- Server-side data
- Computations...

## At left is the complete script for generating 

- * January Mean 500 Height (1981 to 1989) minus (1990 to 1998)
- * Mean & Standard Deviation for all 10 ensembles
- * Time required: 60 secs
- 'reinit'
- '!date'
- * baseURL  = 'http://motherlode.ucar.edu:9090/dods/_expr_'
- * GKR 2/13/03  New NCAR URL
- baseURL  = 'http://dataportal.ucar.edu:9191/dods/'
- expr = 'ave(z,t=387,t=483,12)-ave(z,t=495,t=591,12)'
- xdim = '0:360'
- ydim = '20:90'
- zdim = '500:500'
- tdim = '1nov1978:1nov1978'
- 'sdfopen 'baseURL'_expr_{C20C/C20C_A}{'expr'}{'xdim','ydim','zdim','tdim'}'
- 'sdfopen 'baseURL'_expr_{C20C/C20C_B}{'expr'}{'xdim','ydim','zdim','tdim'}'
- 'sdfopen 'baseURL'_expr_{C20C/C20C_C}{'expr'}{'xdim','ydim','zdim','tdim'}'
- 'sdfopen 'baseURL'_expr_{C20C/C20C_D}{'expr'}{'xdim','ydim','zdim','tdim'}'
- 'sdfopen 'baseURL'_expr_{C20C/C20C_E}{'expr'}{'xdim','ydim','zdim','tdim'}'
- 'sdfopen 'baseURL'_expr_{C20C/C20C_F}{'expr'}{'xdim','ydim','zdim','tdim'}'
- 'sdfopen 'baseURL'_expr_{C20C/C20C_G}{'expr'}{'xdim','ydim','zdim','tdim'}'
- 'sdfopen 'baseURL'_expr_{C20C/C20C_H}{'expr'}{'xdim','ydim','zdim','tdim'}'
- 'sdfopen 'baseURL'_expr_{C20C/C20C_I}{'expr'}{'xdim','ydim','zdim','tdim'}'
- 'sdfopen 'baseURL'_expr_{C20C/C20C_J}{'expr'}{'xdim','ydim','zdim','tdim'}'
- 'define resa = result.1'
- 'define resb = result.2'
- 'define resc = result.3'
- 'define resd = result.4'
- 'define rese = result.5'
- 'define resf = result.6'
- 'define resg = result.7'
- 'define resh = result.8'
- 'define resi = result.9'
- 'define resj = result.10'
- say 'got data'
- 'set lev 500'
- 'set lat 20 90'
- 'define mean = (resa + resb + resc + resd + rese + resf + resg + resh + resi + resj)/10'
- 'define d1 = (pow(resa-mean,2))' ; 'define d2 = (pow(resb-mean,2))'
- 'define d3 = (pow(resc-mean,2))' ; 'define d4 = (pow(resd-mean,2))'
- 'define d5 = (pow(rese-mean,2))' ; 'define d6 = (pow(resf-mean,2))'
- 'define d7 = (pow(resg-mean,2))' ; 'define d8 = (pow(resj-mean,2))'
- 'define d9 = (pow(resi-mean,2))' ; 'define d10 = (pow(resj-mean,2))'
- 'define stddev = pow((d1 + d2 + d3 + d4 + d5 + d6 + d7 + d8 + d9 + d10)/10,0.5)'
- 'set gxout shaded'
- 'set mproj nps'
- 'display mean'
- 'draw title January Mean 500 Height (1981 to 1989) minus (1990 to 1998)'
- 'set string 3 bc 1'
- 'draw string 5.5 .5 Mean & Standard Deviation for all 10 ensembles:
- ‘C20C  Climate of the 20th Century Folland/Kinter'
- *'cbarn'
- 'set gxout contour'
- 'set ccolor 0'
- 'display stddev'
- '!date'

- At left is the complete script for generating
- mean and sdev at 500mb analyzing 18 years of
- “_Climate of the 20__th __Century_” over the Internet:
-            _Traditional_ vs.	_*NOMADS*_ methods:
- Data volume transported: 100Gb vs. 2Kb
- Time to access data:	   2 days vs. 60 sec
- Code development: 	   days    vs. minutes
- Fortran based LOC:	  1000    vs. 50 LOC

## NCDC Web Interface (cont.)

*NCDC Web Interface (cont.)*

- The NCDC Web Interface originally developed at NCEP:

- NOMADS leverages efforts across the community.

## NCEP “Web Plotter” 

*NCEP “Web Plotter” *

-   Developed at NCEP.
-   NCDC ingests 150K
- grids/day.  POR 2002
- to present.
-   Any one of these
- accessible in seconds
- Via:	OpENDAP
- 	GDS
-         	ftp
-         	Web Plotter
- 	LAS (soon)

## Providers: NCEP

*Providers: NCEP*

## Providers: GFDL

*Providers: GFDL*

## Providers: FSL

*Providers: FSL*

## Collaborator: NASA GCMD

*Collaborator: NASA GCMD*

## Other Portals: NCAR

*Other Portals: NCAR*

## Next Steps

-    North American Regional Reanalysis (NARR). Ingesting from NCEP.  NOMADS servicing.
-    NCDC NOMADS to archive/service new proposed NOAA Reanalysis (CDC/CPC/NCDC).
-    AWIPS looking into NOMADS:
-  Phoncon: FSL indicates next AWIPS will be OPeNDAP enabled.
-  FSL drafting Grid white paper to NOAA CIO (NOMADS part of the mix).
-  NOMADS to brief NOAA.   Request NCEP and GFDL Participation.
-    “Operationalize”
-  Move NOMADS services into the TOC?
-    Advance into Climate analysis and detection efforts
-  Science based user workshops and projects needed.
-    Backfill GFS & Eta to 2000.  Currently 2002 to present.

*Next Steps*

## Next Steps (cont.)

-   Adequate metadata for open access for NWP and GCM.  Groups working the GCM  but need NWP model metadata development.
-   Google like search engine through ESP community.
-   Data Management and Science oversight at NCDC, NCEP, and GFDL to advance, support, and coordinate various OPeNDAP projects:
-  allows climate / ocean / weather model inter-comparison and access.
-  Support software development institutions (e.g, COLA, PMEL, OPeNDAP).
-  Do not over engineer this process.  Build on simple successes with existing efforts including a new NOMADS capability within CLASS & thru Scientific Data Stewardship (SDS).

*Next Steps (cont.)*

## NOMADS:  OPeNDAP Enabled Portals

- NOMADS:  OPeNDAP Enabled* Portals*

- CDC:		_[http://www.cdc.noaa.gov/cgi-bin/nph-nc/Datasets/](http://www.cdc.noaa.gov/cgi-bin/nph-nc/Datasets/)_
- COLA: 	_[http://cola8.iges.org:9090/dods](http://cola8.iges.org:9090/dods)_
- FSL:		_[http://nomads.fsl.noaa.gov/](http://nomads.fsl.noaa.gov/)_
- GFDL:		_[http://nomads.gfdl.noaa.gov/](http://nomads.gfdl.noaa.gov/)_
- NCDC: 	_[http://nomads.ncdc.noaa.gov/](http://nomads.ncdc.noaa.gov/)_
- NCEP:		_[http://nomad1.ncep.noaa.gov/](http://nomad1.ncep.noaa.gov/)_
- Unidata:	_[http://www.unidata.ucar.edu/cgi-bin/dods/datasets/](http://www.unidata.ucar.edu/cgi-bin/dods/datasets/)_

## For more Program Information see:
		http://www.ncdc.noaa.gov/oa/climate/nomads/nomads.html

To get data:
	NOAA NCDC Main Page  Climate    Model Resources
				http://nomads.ncdc.noaa.gov

Or contact:
				Glenn.Rutledge@noaa.gov

Selected Publications:
				http://www.ncdc.noaa.gov/oa/model/publications/publications.html

**	****	****http://www.ncdc.noaa.gov/oa/climate/nomads/nomads.html**
**To get data:**
**	****NOAA NCDC Main Page ******** Climate  ********  *****Model Resources***
**	****	****	****	**_**[http://nomads.ncdc.noaa.gov](http://nomads.ncdc.noaa.gov/)**_
**Or contact:**
**	****	****	****	**_**[Glenn.Rutledge@noaa.gov](mailto:Glenn.Rutledge@noaa.gov)**_
***Selected Publications:***
***	******	******	******	******http://www.ncdc.noaa.gov/oa/model/publications/publications.html***

*For more information...*