# Presentation: 096734

## (NOAA Operational Model Archive and Distribution System)

**Real Time **
**NOMADS Component**

- Jordan C. Alpert  _[jordan.alpert@noaa.](mailto:jordan.alpert@noaa.gov)__[gov](mailto:jordan.alpert@noaa.gov)_
- Jun Wang                  _[jun.wang@naa.gov](mailto:jun.wang@naa.gov)_
-  Earth Science Portal  06/9//04

**where the nation’s climate and weather services begin**

**(NOAA Operational Model Archive and Distribution System)**

## Real-Time NOMADS is a service for science, development and public access.

*To distribute NCEP’s Operational data sets to researchers and the public.  *
*To provide server software to NOAA projects who need to serve their data through web services.*
- NCEP NOMADS servers are the intermediary between NCEP operational computers and NCDC NOMADS archive servers.

## Servers provide Tools for Users who need digital data

- Pare down large file sizes of high resolution  initialization and model results.
- (re-) Group different data sets to create needed products – such as initialization files for model development.
- Subset the data sets in parameter space
- Subset the data sets in physical space
-  (subset the grid in 4 dimensions)

## NCEP Real-Time NOMADS Server

- Real time.
- Operational Suite of Model products.
- Forecast information for model comparison.
- Operational initialization and observations files.
- Maintenance of codes.
- Utilities for data manipulation and grid transformation & sub-setting.
- Documentation.
- Serves as transfer point to NCDC for NOMADS archive of reruns and retrospectives.  The goal is a seamless data archived from 2000 to the present.

## The Server – Client distributed relationship:A Common Sense Approach to Data Distribution

- _[ftp2u](ftp://ftp2u/)_, Great Displays (pdisp plot), and GrADS allow sub-setting of data by parameter, by 3D region, and by time.
- Allows for results with low band width!  Eg., NASA/GSFC takes 1 hour to ftp entire GFS (AVN) forecasts or 13 seconds to download their  desired fields (_[ftp2u](ftp://ftp2u/)_).
- “Great Displays” (pdisp) A display program operating from the same database as ftp2u to  display any part of the NCEP data base by time, region, vertical level and variable.

*Examples of Clients :*

**Notes:**
NASA and NOAA have fast lines between them!

## ftp2u is a client application that filters GRIB files in the 3 space dimensions, time and parameter space.

- _[ftp2u](ftp://ftp2u/)_ will send the repackaged GRIB file to your ftp server or ours for later download via http or ftp services.
- The original GRIB files can be obtained with *ftp* at nomad1[2][3].ncep.noaa.gov (eg., nomad3 has ensembles)
- cd /pub/ens/archive/ensyyyymmdd              for low resolution
- cd /pub/enshires/archive/ensyyyymmdd      for high resolution

- More on _[ftp2u](ftp://ftp2u/)__ ..._
*It will slice! It will dice! It will repackage GRIB files! And send the results to your ftp server or ours.  Place the resulting URL in a Unix script “for” loop and watch it churn out data!*

## The URL from the ftp2u session 

*(for those with Unix knowledge)*

- The URL from a successful _[ftp2u](ftp://ftp2u/)_ session can be changed and re-issued in any browser.
- In Unix script the URL may become the “quoted object” of a *‘wget’* (or wwwgrab, etc...) command.  Control the script actions with script ($...) variables.
- Unix script ‘*for loops’* can allow users to obtain many files but we ask users to please include a ‘*sleep 30’* (wait 30 seconds) inside the loop to stop tight loops from accidentally occurring, otherwise throttles will apply.
- Place scripts in ‘*cron’* for automatic file retrieval.

## The GrADS-DODS  Server (GDS)

-  NOMADS participants serve their data sets through a client-server relationship, that is, the data sets are internet ready and the display is done by the client.
- GDS combines both GrADS, a freeware client (from COLA) and DODS (OPeN-DAP) server to compress and exchange data in many formats with http.
**This means that NCEP data can appear to the user or client application as a **_**local file**_**!**
*                     **Like Network File System over the web!*
- DODS requests are made by many freeware and commercial high level language clients.
- Simple http queries to the DODS server can create value added products.

## Slide 10

## NCEP Model Data Sets

- Each RT-NOMAD server contains a header web page (under construction) pointing to various documentation, explanations, and status links and....
- A table listing several data sets. By clicking on the appropriate command, you can (1) make plots, (2) *ftp2u* the files to your computer or (3) obtain DODS metadata descriptions or other data set documentation.

## Slide 12

## Slide 13

## Slide 14

## Slide 15

## An Aggregate Server Based on

- D ISTRIBUTED  M ETADATA  SERVER
- D  I  M  E  S

- (Yang, R, X. Deng, M. Kafatos, C. Wang, S. Wang, 2001
- An XML-Based Distributed Metadata Server (DIMES)
- Supporting Earth Science Metadata” in Proc. 13th Intl. Conf.
- on Sci. and Stat. Database Manag.  pp. 251-256, IEEE,
- Comp Soc.)

## A DIMES Search Page - 

- A DIMES Search Page -

## Result of DIMES search for tmp2m and  ETA for 5/1/2004

## Slide 19

## Metadata Descriptor Problems

- Descriptions of Variable names, Units, Text and other keywords are not uniform.
- Much of the description at NCEP is “Title” and defaults, for example, the vertical coordinate defaults to Pressure in mb! *(...lucky for us)*
- DODS descriptions do not always contain enough information, for example, how to describe unequally space vertical coordinate.
- Project to correct the data descriptions is long term but data managers have to be willing to make corrections consistent with scientific community.
- What are the community standards that we should use!

## Excerpt from the metadata info section.

- GrADS-DODS Server -
- info for /gfs/gfs20031021/gfs_00z : dds  das
- DODS URL: http://nomad1.ncep.noaa.gov:9090/dods/gfs/gfs20031021/gfs_00z
- Description: AVN fcst starting from 00Z21oct2003,
- Longitude: 0°E to 359°E
-  (360 points, avg. res. 1.0°)
- Latitude: -90°N to 90°N
-  (181 points, avg. res. 1.0°)
- Altitude: 1000 to 10
-  (26 points, avg. res. 39.6)
- Time: 00Z21OCT2003 to 12Z28OCT2003
-  (61 points, avg. res. 3.0 hours)
- Variables: (total of 114)
- absv ** absolute vorticity [/s]

## More from a metadata info section.

- complete metadata listing:
- Global attributes:
- title: "NCEP/DOE Reanalysis (R-2), 4x daily rotating archive"
- convention: "COARDS"
- Variables
- 	absvprs: Grid
- 		_FillValue: 9.999E20
- 		long_name: "** absolute vorticity [/s] "
- 		absvprs: Array of 32 bit Reals [time = 0..287][lev = 0..16][lat = 0..72][lon = 0..143]
- 			_FillValue: 9.999E20long_name: "** absolute vorticity [/s] "
- 		   time: Array of 64 bit Reals [time = 0..287]
- 			units: "days since 1-1-1 00:00:0.0"
- 			long_name: "Time"
- 			minimum: "00Z22MAR2004"
- 			maximum: "18Z01JUN2004"
- 		   lev: Array of 32 bit Reals [lev = 0..16]
- 			units: "mb"
- 			long_name: "altitude"
- 			minimum: 1000
- 			maximum: 10
- 		   lat: Array of 32 bit Reals [lat = 0..72]
- 			units: "degrees_north"
- 			long_name: "latitude"
- 			minimum: -90maximum: 90
- 		   lon: Array of 32 bit Reals [lon = 0..143]
- 			units: "degrees_east"
- 			long_name: "longitude"
- 			minimum: 0
- 			maximum: 357.5

## Result of a Constrained Query from the web browser

- The wget command:
- wget -O -  "http://nomad1.ncep.noaa.gov:9090/dods/gfs/gfs20040604/gfs_00z.ascii?tmp[0:0][0:5][110:110][0:0]"
- returned the following data:
- tmp, [1][6][1][1]
- [0][0][0], 312.2
- [0][1][0], 310.7
- [0][2][0], 310.2
- [0][3][0], 309.5
- [0][4][0], 307.5
- [0][5][0], 302.7
- time, [1]
- 731737.0
- lev, [6]
- 1000.0, 975.0, 950.0, 925.0, 900.0, 850.0
- lat, [1]
- 20.0
- lon, [1]
- 0.0

## DODS Servers (GDS) makes NCEP disk storage directly accessible to your PC!

- ... Using http to extract data from the server:
- http://140.90.198.156:9090/dods/eta/eta20030527/eta_00z.ascii?tmpprs[0:0][0:0][130:130] [290:290]
- the order of the square bracked values is:
-  [time][level][lat][lon]
-  where lat is measured from the south pole (0) to NP (180)
- In units represented by the metadata descriptor file.
- (Use a “wget” and the URL in cron or a cgi-bin script provides needed values.)

## ...and Value Added Products!

- Other Commercial and Freeware client applications can access the GrADS-DODS GDS server and use the functionality built into the advanced software.
- As a VAR you can make VAPs from our GDS.
- Use GDS to construct a value added product:
*The probability of any weather element event, say for example, high or low temperature*
- E X A M P L E
- Lets use the NCEP 1x1 degree (high resolution) Ensembles

## NOMADS Ensemble Data Sets at NCEP 

- Real Time Operational Ensemble model data sets from GFS in 5 day rotating archive.
- Ensemble at Low resolution: 2.5 degrees, 00, 06, 12 and 18Z Ensemble Cycles, (ensemble control, “c0” at 00Z and GFS (MRF) for other cycles)  out to 16 day (384-h) forecasts.
- Ensemble at High Resolution: 1 degree, 00, 06, 12, 18Z cycles out to 96-h forecasts.

*Metadata descriptor files describe all aspects of the data sets and are created uniquely from the headers of GRIB files. *

## .... Ensemble Data Sets on nomad3

*http://nomad3.ncep.noaa.gov/ncep_data*

## User Selects the Station Name from a list and set the 

- User Selects the Station Name from a list and set the
- date and forecast time, and the details of the desired event.

## The (Perl) web page inserts the lon/lat of the station and creates constrained queries for each of the 11 Ensemble forecast components.  A probability is constructed as frequency of the user picked event.  Here the last 5 of the 11 ensembles are shown along with the probability of the selected event.  

- The (Perl) web page inserts the lon/lat of the station and creates constrained queries for each of the 11 Ensemble forecast components.  A probability is constructed as frequency of the user picked event.  Here the last 5 of the 11 ensembles are shown along with the probability of the selected event.

## NOMADS Real Time Project at NCEP 2004-2005

- Increased time and space resolution Operational model datasets, 1x1 deg Regional Ensembles, GFS 1x1 deg, 3hr, Eta 12km (#218 grid) 1and 3 hr and special cases, AMIP, re-analysis,...
**BUFR Observations** “ready for the NCEP cycling analysis” with complex quality control – all observations; conventional, like ADPUPA, and non-conventional like Satellite and radar winds.  (Certain restrictions apply for “restricted data sets” from Europe!)
- Integration of the BUFR obs  and model grid fields using GDS server/clients.

## “Operational – lizing” NOMADS 

- Many and varied public and scientific community clients are utilizing NOMADS!
- We need to be more reliable.  Operational budget for fail-over and operational management.
- Operational NOMADS for weather Service Regional centers and Weather Forecast Offices, for example, to allow the National Digital Forecast Database (5 km manual forecast products) interaction with forecasters for watches and warnings.

## Other Real Time NOMADS Servers at NCEP

- In addition to _[http://nomad3.ncep.noaa.](http://nomad3.ncep.noaa.gov/)__[gov](http://nomad3.ncep.noaa.gov/)_
- _[http://nomad5.ncep.noaa.gov](http://nomad5.ncep.noaa.gov/)_
- _[http://nomad2.ncep.noaa.gov](http://nomad2.ncep.noaa.gov/)_
- _[http://nomad1.ncep.noaa.gov](http://nomad1.ncep.noaa.gov/)_
- nomad4 is a disk storage appliance soon to be on line like the above servers.

## Slide 33