# Presentation: 327188

## The NOAA Operational Model Archive and Distribution System NOMADS

- The NOAA Operational Model Archive and Distribution System **NOMADS**

- 	     	      	    **Glenn K. Rutledge**
- 	       National Oceanic and Atmospheric Administration
- 	                         National Climatic Data Center
- 	        2005 NOAA Data and Information Users’ Workshop
-     		                           Asheville, NC
- 			          May 11 - 13, 2004

- Image: Unidata idv

- NWP and Climate Model Data Access

## Overview

-  Retrospective analysis and model inter-comparison are necessary to verify and improve short term NWP models, seasonal forecasts, climate simulations, assessment, and detection efforts.

-   University and Institutional research goes largely untapped by NOAA scientists. Effort is wasted on data receipt and format issues with no infrastructure to collaborate.

*Overview*

## Overview

-  In 2001 NCDC, NCEP and GFDL initiated the NOAA Operational Model Archive and Distribution System.
-  Operational in 2003, NOMADS is a distributed data services pilot for format independent access to climate and weather models and data.

*Overview*

## NOMADS Goals

- communities (ocean, weather, and climate)
- to study multiple earth systems using
- collections of distributed data,

-   promote model evaluation and product development,

-   develop institutional partnerships via distributed open technologies.

-   provide distributed access to models and associated data,

*NOMADS Goals*

- _The NOAA Operational Model Archive and Distribution System_

## Model Data Access

-   	- What data of interest exist?
-   	- Are they going to be useful to me?
-   	- How can I obtain them in a usable form?
-    Time and effort are wasted on data access and format
-      issues.
-   As a result atmosphere/ocean/climate data are under-
-     utilized.  Model inter-comparison nearly impossible.

*Model Data Access*

## Uses

-   Long-term stewardship of NWP
-    analysis and forecasts.
-  Accurate estimates of future climate
-   variability and trends
-  Improved climate and weather
-   assessments.
-  Promote collaboration of climate
-   and weather researchers.

*Uses*

## Tools for Users

*Tools for Users*

- Pare down large file sizes of high resolution data and products.
- (re-) Group different data sets to create needed products – such as initialization files for model development, analysis, or by forecast projection.
- Subset the data:
-      - in parameter space
-  - in physical space
-  - in temporal space

## NOMADS Archive and Users

**2004-2005**

- Millions

**2004-2005**

- Tb

## Other Providers: NCEP

*Other Providers: NCEP*

- Distributed Access...

## Providers: FSL

*Providers: FSL*

## Providers: GFDL

*Providers: GFDL*

## Collaborator: NASA GCMD

*Collaborator: NASA GCMD*

## NWP Model
Global Forecast System (GFS), Model output and Input (SSI) with Restart 
North American Mesoscale (NAM, formally Eta) 
NOAAPort Rapid Update Cycle (RUC) forecast output (FSL)
NCEP Spectral Statistical Interpolation (SSI) Global Data Assimilation System  (GDAS) model input and restart files 
NCEP North American Regional Reanalysis (NARR) (Figure 3)
NCEP/NCAR R1/R2 Reanalysis (Climate Data Assimilation System -CDAS) 
NCEP Regional Special Model (RSM)
NCEP Global Ensembles
NCEP Climate Forecast System (CFS) coupled climate model.
NCEP Ocean Wave models
NCEP Sea Ice Models
 In situ
NCDC Global Historical Climate Network (GHCN) surface temperature and precipitation anomalies (Figure 4)
NCDC Integrated Global Radiosonde Archive (IGRA) upper air reference quality data set (formally the Comprehensive Aerological Data Set- CARDS)
NCDC Smith-Reynolds Extended Reconstructed Sea Surface Temperatures (ERSST) climatologies (see Figure 5)
 c. Satellite and Radar
National Ocean Data Center (NODC) Advanced Very High Resolution Radiometer (AVHRR), Pathfinder Sea Surface Temperature (SST) analysis datasets provided through a link to NODC
Other satellite and Radar data are currently being developed with various partners for access to limited Geostationary Operational Environmental Satellite (GOES), data and the National Aeronautics and Space Administration (NASA) satellite and other data via OPeNDAP.   
Climate: Coupled AOGCM 
GFDL GCM
GFDL CM2.0 and CM2.1 Climate Experiments
GFDL_R15_b ‐ Low resolution (R15L9 atmosphere, 4 degL12 ocean)
GFDL_R30_c ‐ medium resolution (R30L14 atmosphere, 2 degL18 ocean).
...

**Global Forecast System (GFS), Model output and Input (SSI) with Restart **
**North American Mesoscale (NAM, formally Eta) **
**NOAAPort Rapid Update Cycle (RUC) forecast output (FSL)**
**NCEP Spectral Statistical Interpolation (SSI) Global Data Assimilation System  (GDAS) model input and restart files **
**NCEP North American Regional Reanalysis (NARR) (Figure 3)**
**NCEP/NCAR R1/R2 Reanalysis (Climate Data Assimilation System -CDAS) **
**NCEP Regional Special Model (RSM)**
**NCEP Global Ensembles**
**NCEP Climate Forecast System (CFS) coupled climate model.**
**NCEP Ocean Wave models**
**NCEP Sea Ice Models**
*** ******In situ***
**NCDC Global Historical Climate Network (GHCN) surface temperature and precipitation anomalies (Figure 4)**
**NCDC Integrated Global Radiosonde Archive (IGRA) upper air reference quality data set (formally the Comprehensive Aerological Data Set- CARDS)**
**NCDC Smith-Reynolds Extended Reconstructed Sea Surface Temperatures (ERSST) climatologies (see Figure 5)**
** *****c. Satellite and Radar***
**National Ocean Data Center (NODC) Advanced Very High Resolution Radiometer (AVHRR), Pathfinder Sea Surface Temperature (SST) analysis datasets provided through a link to NODC**
**Other satellite and Radar data are currently being developed with various partners for access to limited Geostationary Operational Environmental Satellite (GOES), data and the National Aeronautics and Space Administration (NASA) satellite and other data via OPeNDAP.   **
**Climate: Coupled AOGCM **
***GFDL GCM***
**GFDL CM2.0 and CM2.1 Climate Experiments**
**GFDL_R15_b ‐ Low resolution (R15L9 atmosphere, 4 degL12 ocean)**
**GFDL_R30_c ‐ medium resolution (R30L14 atmosphere, 2 degL18 ocean).**
**...**

*     **NCDC NOMADS Data*

## N.A. Regional Reanalysis

*N.A. Regional Reanalysis*

-  Create a long-term set of consistent climate data on a
-   regional scale on a North American domain.
-  Superior to NCEP/NCAR Global Reanalysis (GR):
-  use of a regional model (the Eta model)
-  advances in modeling and data assimilation since
-   1995 especially:
-  Precipitation assimilation
-  Direct assimilation of radiances
-  Land-surface (NOAH) model updates

- NOAA Access Point for NARR

## Scientific Data Networking

- Access to remote data in the users normal application
- IDL / IDV / Matlab / Ferret
- GrADS (GRIB/BUFR w/ GDS)
- Netscape / Excel / http (wget)
- CDAT (PCMDI)
**Any netCDF application**      (i.e., AWIPS)
**Don’t need to know the format in which the data are stored.**

**Data publishing (server)**
**Can serve data in various formats**
- netCDF / GRIB / BUFR / GRIB2
- HDF (3-5) / EOS
- SQL / FreeForm
- JGOFS / NcML
- DSP
- ascii, others...
**Spatial and temporal sub-setting and host side computations on the fly.**

-  NOMADS simplifies scientific data networking, allowing simple access to high volume remote data, unifying access to Climate and Weather models:

*Scientific Data Networking*

- Advancing Collaborations

## System Architecture

- NOAAPort

- Unidata IDD

- NCDC ftp

** ****Data & Directory **
**   ****structures “merged”**
** ****Daily Data Ingest **
**   ****inter-comparison **
** ****QC and Monitoring**
** ****Index File generation**
** ****Control and DODS **
**   ****metadata generation  **
** ****CVS Backup (code)**
** ****HDSS/HAS Injection**

**CEOS-Grid**
**(exploratory)**

**Obs, Eta, **
**GFS, RUC**

**Hi-Res**
**GFS, Eta, **
**NARR and**
**GDAS**

_**Data Ingest***_

**Dual**
**Redundant**
**Ingest**

_**Data Management**_

_**Data Access**_

**DSI 6172**
**DSI 6173**
**DSI 6174**
**DSI 6175**

**NOMADS**
**Access**

*System Architecture*

- * NOMADS provides NCDC NOAAPort ingest processing

- NCDC Archive

- AAB Ingest Processes

- DAB/NOMADS Ingest Processes

- AAB Archive Processes

## The NOMADS Philosophy

*The NOMADS Philosophy*

_**Multiple paths to format independent data access:**_

- Advancing Collaborations

## NCDC Web Interface

*NCDC Web Interface*

- Three primary
- methods for data
- access:
-  Web Interface
-  GDS OPeNDAP
-  ftp w/ on the fly
-   Grib subsetting
- On-line or
- Off-line (archive)
-  Server-side data
- computations.

## NCDC Web Interface (cont.)

- The NCDC Web Interface originally developed at NCEP:

- NOMADS leverages efforts across the community.

- Promoting Model Collaborations

## NOMADS “Web Plotter” 

*NOMADS “Web Plotter” *

-  NCDC NOMADS
- ingests 250K grids
- day.  POR 2002
- to present.
-   Any one of these
- accessible in seconds
- Via:	OpENDAP
- 	GDS
-         	ftp
-         	Web Plotter
- 	LAS
- 	mySQL

## Slide 21

## Slide 22

## NCDC Reference Datasets

*NCDC Reference Datasets*

- NCDC reference
- And others
- datasets also available:
-  CARDS (IGRA)
-  GHCN
-  NARR
-  Ocean WAVE
- Accessible via
-   OPeNDAP

- Promoting Model to Obs. Intercomparisons

## Unidata’s

**Unidata’s**
**IDV & NOMADS **
**Access**

## Slide 25

## NOMADS Ensemble Access

- NOMADS Ensemble Probabilities on the fly

*NOMADS Ensemble Access*

- URL is: _[http://nomad3.ncep.noaa.gov:9090/dods/enshires/archive/ens20040809/ensc0_](http://nomad3.ncep.noaa.gov:9090/dods/enshires/archive/ens20040809/ensc0_)_
- 00z_1x1.ascii?pratesfc[3:3][125:125][277:277]

- Enabling private sector access: An example

** ****No need for image generation of ensembles...**

**OPeNDAP constraint expression**

## At left is the complete script for generating 

- * January Mean 500 Height (1981 to 1989) minus (1990 to 1998)
- * Mean & Standard Deviation for all 10 ensembles
- * Time required: 60 secs
- 'reinit'
- '!date'
- * baseURL  = 'http://motherlode.ucar.edu:9090/dods/_expr_'
- * GKR 2/13/03  New NCAR URL
- baseURL  = 'http://dataportal.ucar.edu:9191/dods/'
- expr = 'ave(z,t=387,t=483,12)-ave(z,t=495,t=591,12)'
- xdim = '0:360'
- ydim = '20:90'
- zdim = '500:500'
- tdim = '1nov1978:1nov1978'
- 'sdfopen 'baseURL'_expr_{C20C/C20C_A}{'expr'}{'xdim','ydim','zdim','tdim'}'
- 'sdfopen 'baseURL'_expr_{C20C/C20C_B}{'expr'}{'xdim','ydim','zdim','tdim'}'
- 'sdfopen 'baseURL'_expr_{C20C/C20C_C}{'expr'}{'xdim','ydim','zdim','tdim'}'
- 'sdfopen 'baseURL'_expr_{C20C/C20C_D}{'expr'}{'xdim','ydim','zdim','tdim'}'
- 'sdfopen 'baseURL'_expr_{C20C/C20C_E}{'expr'}{'xdim','ydim','zdim','tdim'}'
- 'sdfopen 'baseURL'_expr_{C20C/C20C_F}{'expr'}{'xdim','ydim','zdim','tdim'}'
- 'sdfopen 'baseURL'_expr_{C20C/C20C_G}{'expr'}{'xdim','ydim','zdim','tdim'}'
- 'sdfopen 'baseURL'_expr_{C20C/C20C_H}{'expr'}{'xdim','ydim','zdim','tdim'}'
- 'sdfopen 'baseURL'_expr_{C20C/C20C_I}{'expr'}{'xdim','ydim','zdim','tdim'}'
- 'sdfopen 'baseURL'_expr_{C20C/C20C_J}{'expr'}{'xdim','ydim','zdim','tdim'}'
- 'define resa = result.1'
- 'define resb = result.2'
- 'define resc = result.3'
- 'define resd = result.4'
- 'define rese = result.5'
- 'define resf = result.6'
- 'define resg = result.7'
- 'define resh = result.8'
- 'define resi = result.9'
- 'define resj = result.10'
- say 'got data'
- 'set lev 500'
- 'set lat 20 90'
- 'define mean = (resa + resb + resc + resd + rese + resf + resg + resh + resi + resj)/10'
- 'define d1 = (pow(resa-mean,2))' ; 'define d2 = (pow(resb-mean,2))'
- 'define d3 = (pow(resc-mean,2))' ; 'define d4 = (pow(resd-mean,2))'
- 'define d5 = (pow(rese-mean,2))' ; 'define d6 = (pow(resf-mean,2))'
- 'define d7 = (pow(resg-mean,2))' ; 'define d8 = (pow(resj-mean,2))'
- 'define d9 = (pow(resi-mean,2))' ; 'define d10 = (pow(resj-mean,2))'
- 'define stddev = pow((d1 + d2 + d3 + d4 + d5 + d6 + d7 + d8 + d9 + d10)/10,0.5)'
- 'set gxout shaded'
- 'set mproj nps'
- 'display mean'
- 'draw title January Mean 500 Height (1981 to 1989) minus (1990 to 1998)'
- 'set string 3 bc 1'
- 'draw string 5.5 .5 Mean & Standard Deviation for all 10 ensembles:
- ‘C20C  Climate of the 20th Century Folland/Kinter'
- *'cbarn'
- 'set gxout contour'
- 'set ccolor 0'
- 'display stddev'
- '!date'

- At left is the complete script for generating
- mean and sdev at 500mb analyzing 20 years of
- “_Climate of the 20__th __Century_” from NCAR.
-            _Traditional_ vs. _*NOMADS*_ method:
- Data volume transported:	100Gb vs. 2Kb
- Time to access data:		2 days  vs. 60 sec
- Code development: 		days     vs. minutes
- Fortran based LOC:		1000    vs. 50 LOC

## For NOMADS Program Information see:

http://www.ncdc.noaa.gov/oa/climate/nomads/nomads.html

For NOMADS Model Data Access:
	NOAA NCDC Main Page  Upper-Air    Data and Products

Or contact:
				Glenn.Rutledge @ noaa.gov

**	****	**
**http://www.ncdc.noaa.gov/oa/climate/nomads/nomads.html**
**For NOMADS Model **_**Data Access**_**:**
**	****NOAA NCDC Main Page ******** Upper-Air  ********  Data and Products**
**	****	****	**
**Or contact:**
**	****	****	****	****Glenn.Rutledge @ noaa.gov**

*For more information...*