# Presentation: 327206

## The NOAA National Operational Model Archive and Distribution System  NOMADS

**The NOAA National Operational Model Archive and Distribution System **** **NOMADS

- 	     	      	    **Glenn K. Rutledge**
-                        National Oceanic and Atmospheric Administration
- 	                           National Climatic Data Center
- 		      	    Informational Briefing
- 			     June 16th, 2006

- Client: Unidata IDV

- Catalog: THREDDS

- Data:  NCDC NOMADS

## Overview

-  Retrospective analysis and model inter-comparison are necessary to verify and improve short term NWP models, seasonal forecasts, climate simulations, assessment, and detection efforts.

-   University and Institutional research goes largely untapped by NOAA scientists. Effort is wasted on data receipt and format issues with no infrastructure to collaborate.

*Overview*

## Overview

-  NCDC, NCEP and GFDL initiated the NOAA Operational Model Archive and Distribution System.
-  NOMADS is a distributed data services pilot for format independent access to climate and weather models and data.

*Overview*

## Model Data Access

-   	- What data of interest exist?
-   	- Are they going to be useful to me?
-   	- How can I obtain them in a usable form?
-    Time and effort are wasted on data access and format
-      issues.
-   As a result atmosphere/ocean/climate data are under-
-     utilized.  Model inter-comparison nearly impossible.

*Model Data Access*

## Some Assumptions

-  Ensemble Predictions:  flow-dependant prediction of weather    and climate risk- nowcasting, medium range and seasonal.

*Some Assumptions*

-  Atmospheric and Oceanic Research-
-  Scalar and Vector processing and Workstation models
-  Model output statistics; data assimilation techniques

-  Global Climate Change and Advanced Analysis-
-  Clouds, initial conditions, true coupled simulations.
-  Long term climate monitoring: in-situ analysis, trends, data homogeneity, extremes, downscaling, reducing uncertainty...
-  On-demand Data Mining and Product Generation.

## NOMADS Goals

- communities (ocean, weather, and climate)
- to study multiple earth systems using
- collections of distributed data,

-   promote model evaluation and product development,

-   develop institutional partnerships via distributed open technologies.

-   provide distributed access to models and associated data,

*NOMADS Goals*

- _The NOAA Operational Model Archive and Distribution System_

## Uses

-   Long-term stewardship of NWP
-    analysis and forecasts.
-  Accurate estimates of future climate
-   variability and trends
-  Improved climate and weather
-   assessments.
-  Promote collaboration of climate
-   and weather researchers.

*Uses*

## NOMADS Archive and Users

*NOMADS Archive and Users*

-   Data Philosophy and Retention
-   Data are free.
-   NWP forecast data are retained for five years.
-   Analysis, Reanalysis, observations, and GDAS model input are retained for long term stewardship.
-   Data Users
-  Resolution of IP addresses indicate a broad range, and
-    consistent use of NOMADS available data:
-   U.S. Agencies, Academic Institutions: K-12 to Research
-   International governments, (Italy, Japan, countries within
-    South America and Africa.  Many others.
-   Private Sector and Non-Government Organizations NGO’s
-   World Bank, United Nations (FAO), others.

## Tools for Users

***Tools for Users***

**Pare down large file sizes of high resolution data and products- and provide flexible inter-operable access.**
**(re-) Group different data sets to create needed products – such as initialization files for model development, analysis, or by forecast projection.**
**Subset the data: **
**         ****- in parameter space**
**    ****- ****in physical space **
**    ****- in temporal space**

## NOMADS is an agreement between agencies who participate to have common ...

**Data and observation distribution software, format independent and description  methods (metadata).**
**Documentation and organizational framework.**
**Forum to plan and organize (science drivers). **
**Funding avenue to direct $ for intra-Agency partnerships**
**... ****for university, federal agencies, and organizations, a mechanism to obtain support for the dissemination of their data sets.**

*A Distributed Framework*

## Partners 

- GFDL:  Coupled Models, Control and Perturbation
-  	 Integrations and historical 20th century simulations
- 	 using solar, volcano, GHG and aerosol forcings.
- ESRL:    MADIS mesoNets, Hi-Res RUC-II
- NCAR:  Community Climate System Model / Land Surface
-               CO2 predictive models (VEMAP), Reanalysis /  Eta
- NCDC:  Archive for NCEP model input/output / Select NCDC
-               Observation datasets, Ocean/Ice WAVE, NARR, SST’s...
- NCEP:   Real-time Input/Output, Reanalysis (I&II),  Ensembles, Sea
-               Ice Ocean, CDAS, Hourly Eta, Climate Forecast Models...
- LLNL:   AMIP / Probabilistic information
- PMEL:   Ocean and Climate datasets
-                                               Now NODC......

***Partners ***

- Advancing collaborations: Multidisciplinary Research

## Collaborating Programs

- CAP     Climate Action Partnership  	       	DOC DOE EPA State Dept
- CEOS   Committee on EO Satellites 		NOAA Representative
- CEOP   Coordinated Earth Obs Period    	NOAA Representative
- EPA     Air Quality Models 			NOAA Representative
- GO-ESSP  Earth Science Portal	 		Founding Member
- NASA GCMD    				User Advisory Committee
- NERC DataGrid (e-science)			Advisory Board
- NSF LEAD					Data Provider
- NSF Forum for Geosciences Information	Planning Committee
- Unidata THREDDS,  NSDL, DLESSE		Data Provider
- US-GEO IEOS 					Sited as Potential Pilot
- WCRP World Climate Research Program  	JSC/CLIVAR

## Scientific Data Networking

- Access to remote data in the users normal application
- IDL / IDV / Matlab / Ferret
- GrADS (GRIB/BUFR w/ GDS)
- Netscape / Excel / http (wget)
- CDAT (PCMDI)
- ArcView (Spring 06)
**Any netCDF application**      (i.e., AWIPS)
**    ****Don’t need to know the format in which the data are stored.**

**Data publishing (server)**
**Can serve data in various formats**
- netCDF / GRIB / BUFR / GRIB2
- HDF (3-5) / EOS
- SQL / FreeForm
- JGOFS / NcML
- DSP
- ascii, others...
**    ****Spatial and temporal sub-setting and host side computations on the fly.**

-  NOMADS simplifies scientific data networking, allowing simple access to high volume remote data, unifying access to data:

*Scientific Data Networking*

- Advancing Collaborations

## Framework

-   OPeNDAP is a binary-level protocol designed for the transport
-     of scientific data subsets over the Internet.   Provides server
-     side data manipulation on-the-fly (e.g., GrADS-DODS).
-   Data formats: GRIB, GRIB2, BUFR, HDF, NetCDF, ascii...
-    Conventions: COARDS, CF, FGDC, DIF....libraries built as
-    necessary.
-   APIs: JAVA-OPeNDAP, C++-OPeNDAP, NetCDF, GRIB,
-    BUFR, THREDDS, Python.

*Framework*

## Framework 2

*Utilize Binary data and metadata through OPeNDAP ...*

*Framework 2*

## System Architecture

- NOAAPort*

- Unidata IDD

- NCEP

** ****Data & Directory **
**   ****structures “merged”**
** ****Daily Data Ingest **
**   ****inter-comparison **
** ****QC and Monitoring**
** ****Index File generation**
** ****Control and DODS **
**   ****metadata generation  **
** ****CVS Backup (code)**
** ****HDSS/HAS Injection**

**CEOS-Grid**
**(exploratory)**

**Obs, Eta, **
**GFS, RUC**

**Hi-Res**
**GFS, Eta, **
**NARR and**
**GDAS**

_**Data Ingest**_

**Dual**
**Redundant**
**Ingest**

_**Data Management**_

_**Data Access**_

**80Tb**
**(FY06)**

**NOMADS**
**40Tb**

*System Architecture*

- * NOMADS NOAAPort ingest processing discontinued 5/05

- NCDC Archive

-  ftp

- On-Line

## The NOMADS System Design

*The NOMADS System Design*

_**Multiple paths to format independent data access**_

**Notes:**
The next step in NOMADS is the availability of access by OGC standards for Web Mapping and Web Coverage Services.
The NASA funded project includes NASA, NOMADS, Unidata, OPeNDAP and GMU for the development of these services using an OPeDNAP “gateway” and Unidata’s THREDDS Data Server (TDS)
NCDC is funded FY06 to develop a pilot grid activity (earmarked money).  NCDC’s Bruce Barkstrom (a former NASA Langley employee) is now developing a SOW.  NOMADS will continue to explore a CEOS Gird activity as times permits and also work to fuse the efforts.  Glenn Rutledge is POC for CEOS-Grid.  
NOMADS is pleased to be a benefactor of this s/w development activity.
TDS is under development and plans are underway to provide model forecast aggregation across multiple models for on-line intercomparison
LAS has a limited capability already.   
All system components firewalled.  Data requests/raid is NFS mounted across a firewall.  Deep archive has separate firewall. 

##                The NCDCNOMADS Architecture June 2006

*               **The NCDCNOMADS Architecture **June 2006*

## System Architecture (cont.)

*System Architecture (cont.)*

- ...

- ...

- Design aligned to user specific targets:
-  Dedicated GDS
-   for each large DS
-  “Load Balanced”
-   for Web requests
-  Automated peer-
- to-peer subscriptions
- Scalable

## GrADS Data Server- GDS

- GRIB, HDF

- NetCDF

- GrADS binary

- etc..

**Datasets in any format supported by GrADS**

- Result cache

**For GRIB / 2 and BUFR users, multi-dimensional data (space/time), and data subsets are extracted on the fly.**

- GrADS batch mode

- interface code

**DODS*** server libraries

- Server

**performs analysis operations**

**manages sessions, translates dataset names**

- Java servlet

**supports extended request types for analysis, upload**

- Encapsulated Analysis Requests

- Station data

- BUFR

- Internet

*GrADS Data Server- GDS*

- Source: B. Doty (COLA) & G. Rutledge

- ** using NetCDF interface

## User Statistics

*User Statistics*

- Highest month:  5.3Tb    w/4 million downloads

- NCDC
-  Only

- BAMS Paper

**Notes:**
Other NOMADS sites include GFDL and NCEP.  NCEP has ~1-5Tb /month

## Dynamic Ingest Monitoring

*Dynamic Ingest Monitoring*

- Dual Site Ingest and Dynamic error reporting for a serially complete archive.  QC architecture discovered NOAAPort labeling errors.

## NOMADS

*NOMADS*
*Web Page*

-   Completely
- re-designed.
-  Uses “php”
-  Easier navigation
-  Better
-   Documentation
-  NOMADS is
- renamed:
- NOAA _*National*_*...*

## Other Providers: NCEP

*Other Providers: NCEP*

- Distributed Access...

## Providers: FSL

*Providers: FSL*

## Providers: GFDL

*Providers: GFDL*

## Collaborator: NASA GCMD

*Collaborator: NASA GCMD*

## NWP Model
Global Forecast System (GFS), 1 degree (1/2 degree in Sept 05)
NCEP Spectral Statistical Interpolation (SSI) Global Data Assimilation System (GDAS) w/ restart files
North American Mesoscale (NAM, formally Eta) 
NOAAPort Rapid Update Cycle (RUC) forecast output (FSL)
NCEP North American Regional Reanalysis (NARR) 
NCEP/NCAR R1/R2 Reanalysis (Climate Data Assimilation System -CDAS) 
NCEP Regional Special Model (RSM)
NCEP Global Ensembles and SREF
NCEP Climate Forecast System (CFS) coupled climate model.
NCEP Ocean Wave models
NCEP Sea Ice Models
NWS Analysis of Record Real-time Mesoscale Analysis (RTMA downscaled RUC to 5km)
 In situ
NCDC Global Historical Climate Network (GHCN) surface temperature and precipitation anomalies 
NCDC Integrated Global Radiosonde Archive (IGRA) upper air reference quality data set (formally CARDS)
NCDC Smith-Reynolds Extended Reconstructed Sea Surface Temperatures (ERSST) climatologies )
NOAA Satellite and Radar (II-III)
NODC AVHRR Pathfinder Sea Surface Temperature (SST) analysis; GOES and Radar in beta 
Climate Models / Coupled AOGCM 
GFDL CM2.0 and CM2.1 Climate Experiments
Paleoclimate Model Intercomparison Project (PMIP) 

- Global Forecast System (GFS), 1 degree (1/2 degree in Sept 05)
- NCEP Spectral Statistical Interpolation (SSI) Global Data Assimilation System (GDAS) w/ restart files
- North American Mesoscale (NAM, formally Eta)
- NOAAPort Rapid Update Cycle (RUC) forecast output (FSL)
- NCEP North American Regional Reanalysis (NARR)
- NCEP/NCAR R1/R2 Reanalysis (Climate Data Assimilation System -CDAS)
- NCEP Regional Special Model (RSM)
- NCEP Global Ensembles and SREF
- NCEP Climate Forecast System (CFS) coupled climate model.
- NCEP Ocean Wave models
- NCEP Sea Ice Models
- NWS Analysis of Record Real-time Mesoscale Analysis (RTMA downscaled RUC to 5km)
* **In situ*
- NCDC Global Historical Climate Network (GHCN) surface temperature and precipitation anomalies
- NCDC Integrated Global Radiosonde Archive (IGRA) upper air reference quality data set (formally CARDS)
- NCDC Smith-Reynolds Extended Reconstructed Sea Surface Temperatures (ERSST) climatologies )
*NOAA Satellite and Radar (II-III)*
- NODC AVHRR Pathfinder Sea Surface Temperature (SST) analysis; GOES and Radar in beta
- Climate Models / Coupled AOGCM
- GFDL CM2.0 and CM2.1 Climate Experiments
- Paleoclimate Model Intercomparison Project (PMIP)

*     **NCDC NOMADS Data*

## NCEP GDAS (cont.)

*NCEP GDAS (cont.)*

-   The minimum set for Global Spectral Forecast Model and the Spectral
- Statistical Interpolation Cycling Analysis System contains ~0.5Gb /run:

- - NOAA-15/16 AMSU-A/B TOVS 1B Radiances (IEEE)
- - Analysis Bias Corrected Information / Obs Toss List
- - SFC U/A, ACRS, Aircft (BUFR)
- - 6HR fcst guess from previous run (BUFR)
- - ERSCAT Sat obs / HIRS 14/15, MSU TOVS (IEEE)
- - Guess prep and and fcst guess output (BUFR)
- - Analysis ready QC’ed Obs. (prepBUFR)
- - Profiler, TOVS, Wind Obs. (BUFR)
- - SFC Analysis Restart Files
- - SST’s (GRIB), Radar VAD Winds (BUFR)

## Model Input: NCEP GDAS

-   model output products as close as possible to NCEP operations.
-  The analysis files will be in the models own coordinate system.
-  Files are constructed with computer and computational efficiency
-   in mind, and not in standard coordinate systems.
-  Programs to convert  these files are available upon request:
-  spectral to gaussian
-  gaussian to lat/lon
-  sigma to pressure

*Model Input: NCEP GDAS*

## NCDC Web Interface (cont.)

- The _[NOMADS](http://nomads.ncdc.noaa.gov/cgi-bin/ncdc-ui/define-collection.pl?model_sys=nam&model_name=nam&grid_name=218)__ _Web Interface:

## NOMADS “Web Plotter” 

*NOMADS “Web Plotter” *

-  NCDC NOMADS
- ingests 250K grids
- day.  POR 2002
- to present.
-   Any one of these
- accessible in seconds
- Via:	OpENDAP
- 	GDS
-         	ftp
-         	Web Plotter
- 	LAS
- 	mySQL

## Traditional Tools

*Traditional Tools*

- Hovmoller

- Time Series

- GDS and GrADS

- Host Side Services

- Client Side

## NOMADS Real-time Grib sub-setting: By variable/location/time

## NCDC Reference Datasets

*NCDC Reference Datasets*

- NCDC reference
- And others
- datasets also available:
-  CARDS (IGRA)
-  GHCN
-  NARR
-  Ocean WAVE
- Accessible via
-   OPeNDAP

- Promoting Model to Obs. Intercomparisons

## N.A. Regional Reanalysis

*N.A. Regional Reanalysis*

-  Create a long-term set of consistent climate data on a
-   regional scale on a North American domain.
-  Superior to NCEP/NCAR Global Reanalysis (GR):
-  use of a regional model (the Eta model)
-  advances in modeling and data assimilation since
-   1995 especially:
-  Precipitation assimilation
-  Direct assimilation of radiances
-  Land-surface (NOAH) model updates

- NOAA’s Primary Access Point for NARR

## Unidata’s  IDV

**Unidata’s  IDV**
**THREDDS Data**
**       ****Server**

## NOMADS Ensemble Access

- _[NOMADS](http://nomad5.ncep.noaa.gov/cgi-bin/var/ensprob2.pl)__ Ensemble Probabilities on the fly_

*NOMADS Ensemble Access*

- URL is: _[http://nomad3.ncep.noaa.gov:9090/dods/enshires/archive/ens20040809/ensc0_](http://nomad3.ncep.noaa.gov:9090/dods/enshires/archive/ens20040809/ensc0_)_
- 00z_1x1.ascii?pratesfc[3:3][125:125][277:277]

** ****No need for image generation of ensembles...**

**OPeNDAP constraint expression**

## At left is the complete script for generating 

- * January Mean 500 Height (1981 to 1989) minus (1990 to 1998)
- * Mean & Standard Deviation for all 10 ensembles
- * Time required: 60 secs
- 'reinit'
- '!date'
- * baseURL  = 'http://motherlode.ucar.edu:9090/dods/_expr_'
- * GKR 2/13/05  New NCAR URL
- baseURL  = 'http://dataportal.ucar.edu:9191/dods/'
- expr = 'ave(z,t=387,t=483,12)-ave(z,t=495,t=591,12)'
- xdim = '0:360'
- ydim = '20:90'
- zdim = '500:500'
- tdim = '1nov1978:1nov1978'
- 'sdfopen 'baseURL'_expr_{C20C/C20C_A}{'expr'}{'xdim','ydim','zdim','tdim'}'
- 'sdfopen 'baseURL'_expr_{C20C/C20C_B}{'expr'}{'xdim','ydim','zdim','tdim'}'
- 'sdfopen 'baseURL'_expr_{C20C/C20C_C}{'expr'}{'xdim','ydim','zdim','tdim'}'
- 'sdfopen 'baseURL'_expr_{C20C/C20C_D}{'expr'}{'xdim','ydim','zdim','tdim'}'
- 'sdfopen 'baseURL'_expr_{C20C/C20C_E}{'expr'}{'xdim','ydim','zdim','tdim'}'
- 'sdfopen 'baseURL'_expr_{C20C/C20C_F}{'expr'}{'xdim','ydim','zdim','tdim'}'
- 'sdfopen 'baseURL'_expr_{C20C/C20C_G}{'expr'}{'xdim','ydim','zdim','tdim'}'
- 'sdfopen 'baseURL'_expr_{C20C/C20C_H}{'expr'}{'xdim','ydim','zdim','tdim'}'
- 'sdfopen 'baseURL'_expr_{C20C/C20C_I}{'expr'}{'xdim','ydim','zdim','tdim'}'
- 'sdfopen 'baseURL'_expr_{C20C/C20C_J}{'expr'}{'xdim','ydim','zdim','tdim'}'
- 'define resa = result.1'
- 'define resb = result.2'
- 'define resc = result.3'
- 'define resd = result.4'
- 'define rese = result.5'
- 'define resf = result.6'
- 'define resg = result.7'
- 'define resh = result.8'
- 'define resi = result.9'
- 'define resj = result.10'
- say 'got data'
- 'set lev 500'
- 'set lat 20 90'
- 'define mean = (resa + resb + resc + resd + rese + resf + resg + resh + resi + resj)/10'
- 'define d1 = (pow(resa-mean,2))' ; 'define d2 = (pow(resb-mean,2))'
- 'define d3 = (pow(resc-mean,2))' ; 'define d4 = (pow(resd-mean,2))'
- 'define d5 = (pow(rese-mean,2))' ; 'define d6 = (pow(resf-mean,2))'
- 'define d7 = (pow(resg-mean,2))' ; 'define d8 = (pow(resj-mean,2))'
- 'define d9 = (pow(resi-mean,2))' ; 'define d10 = (pow(resj-mean,2))'
- 'define stddev = pow((d1 + d2 + d3 + d4 + d5 + d6 + d7 + d8 + d9 + d10)/10,0.5)'
- 'set gxout shaded'
- 'set mproj nps'
- 'display mean'
- 'draw title January Mean 500 Height (1981 to 1989) minus (1990 to 1998)'
- 'set string 3 bc 1'
- 'draw string 5.5 .5 Mean & Standard Deviation for all 10 ensembles:
- ‘C20C  Climate of the 20th Century Folland/Kinter'
- *'cbarn'
- 'set gxout contour'
- 'set ccolor 0'
- 'display stddev'
- '!date'

- At left is the complete script for generating
- mean and sdev at 500mb analyzing 20 years of
- “_Climate of the 20__th __Century_” from NCAR.
-            _Traditional_ vs. _*NOMADS*_ method:
- Data volume transported:	100Gb vs. 2Kb
- Time to access data:		2 days  vs. 60 sec
- Code development: 		days     vs. minutes
- Fortran based LOC:		1000    vs. 50 LOC

## Retrieving ascii from binary GRIB ulwrfsfc (OLR) from NCEP 

-  wget w/ OLR example
- File (url): level, lat, lon ()time is known using [Fn]
- -- I need to know what variable I want and where in the given coordinate system:
-       ULWRFsfc 1 level * surface Upward long wave flux [W/m^2]  (lat/lon)
- -- and where the data is located (url):
- _http://nomad2.ncep.noaa.gov:9090/dods/gdas/rotating/gdas2003092000/ _
- -- then pull an ascii value directly from grib binary using OPeNDAP constraint:
- _.ascii?ulwrfsfc[0:1][140:140][200:200]\_
- Returned by http:
- 256
- This is ULWRFsfc  in W/m^2  !!!

## New NOMADS LAS Server
On-the-fly model inter-
comparisons

Limited GFDL CM2.x
Intend to download most 
           requested CM runs
Climate Indices Scripts
New NCDC Mirror Backup 
     Server for NCEP

- On-the-fly model inter-
- comparisons
- Limited GFDL CM2.x
- Intend to download most
-            requested CM runs
- Climate Indices Scripts
- New NCDC Mirror Backup
-      Server for NCEP

*New Data and Services*

## NOMADS requested by NWS to Service hi-volume 5km NDFD. 

NDFD is a seamless mosaic of digital
     forecasts from NWS field offices 
     working in collaboration with NCEP.  

Users (forecasters) of NDFD can 
     create their own products using GDS
     server accessing only data they need.  

NOMADS reduces the bandwidth
     needed to create products in
     weather service operations.

**NDFD is a seamless mosaic of digital**
**     ****forecasts from NWS field offices **
**     ****working in collaboration with NCEP.**** **
**Users (forecasters) of NDFD can **
**     ****create their own products using GDS**
**     ****server accessing only data they need.  **
**NOMADS reduces the bandwidth**
**     ****needed to create products in**
**     ****weather service operations.**

- National Digital Forecast Database

- NDFD 5km Sensible Wx Condition

## NASA MAPS (G4) minus NCEP GFS

- 7

*NASA MAPS (G4) minus NCEP GFS*

- Real-Time Model
- Intercomparisons

## A NOMADS OGC Extension

-  It is flexible enough to add a Open GIS Consortium (OGC) Web Coverage Service (WCS), and Web Feature Service (WFS) needed for GIS users (ArcView/GRASS/ GMT/ Intergraph/others).
-  No need for a complete GIS “Server” effort for NOMADS.  The Interfaces through OPeDNAP present the data to GIS clients from NOMADS.  Community driven but satellite issues (geo-rectification, mapping etc) remain.

***A NOMADS OGC Extension***

## The THREDDS Data Server:
Catalog + Ag Services
Test out with all data forms
Reload capability: how scalable? 

- Catalog + Ag Services
- Test out with all data forms
- Reload capability: how scalable?

***A NOMADS TDS***

## NOMADS is a founding member of the Global Organization for Earth System Science Portals (GO-ESSP).

GO-ESSP is a collaboration designed to build the infrastructure needed to create web portals to provide access to observed and simulated data within the climate and weather communities. 

The infrastructure created within GO-ESSP will provide a flexible framework that will allow interoperability between the front-end and back-end software components. ESP is an international collaboration involving software developers from both Europe and the US. See http://esportal.gfdl.noaa.gov/

**  **
**GO-ESSP is a collaboration designed to build the infrastructure needed to create web portals to provide access to observed and simulated data within the climate and weather communities. **
**The infrastructure created within GO-ESSP will provide a flexible framework that will allow interoperability between the front-end and back-end software components. ESP is an international collaboration involving software developers from both Europe and the US. See http://esportal.gfdl.noaa.gov/**

*The** **Bigger** **Picture*