# Presentation: 809270

## (NOAA Operational Model Archive and Distribution System)

**and**
**Real Time NOMADS**

- Jordan C. Alpert  _[jordan.alpert@noaa.](mailto:jordan.alpert@noaa.gov)__[gov](mailto:jordan.alpert@noaa.gov)_
- Jun Wang                  _[jun.wang@naa.gov](mailto:jun.wang@naa.gov)_
- Bhavani Balasubramaniyan Bhavani.Balasubramaniyan@noaa.gov
- 						MMB  11/08//05

**where the nation’s climate and weather services begin**

**(NOAA Operational Model Archive and Distribution System)**

## NOMADS – What is it?

*A *_*service orientated architecture*_* prototype to provide distributed format neutral access to model data.*
- Data distribution software, format independent and description methodology,
- Documentation and organization framework,
- Forum to plan and organize,
*... **for university, federal agencies and organizations, a framework for the dissemination of their data sets.*

*For Data producers – e.g., NCEP models and projects...*

**Notes:**
There are many participants in NOMADS – See the talk from NCDC. 

## RT-NOMADS at NCEP (jargon)

- 	Distributed Ocean Data Server  - OPeNDAP (DODS) based on metadata file descriptors and XML under the GrADS Data Server (GDS).
- Now known as the Open-Source Project for a Network Data Access Protocol.
- NCEP NOMADS servers are the intermediary between NCEP operational computers and NCDC NOMADS archive servers.
- 3Yrs of operation, many partners, countless users.

*Consists of*

## RT-NOMADS is a service for science, development and public access for data users.

*Tools on the server – **	**C**lient Application Examples:*
*	**	*_[ftp2u](ftp://ftp2u/)_*, http(fast ftp) or plain ftp*
- Pare down large file sizes of high resolution  initialization and model results.
- (re-) Group different data sets to create needed products – such as initialization files for model development.
- Subset the data sets in parameter space,
- Subset the data sets in physical space and time.
-  (subset the grid in 4 dimensions)
*The user’s resulting URL address containing  user control variables, can be placed in a suitable script with an interactive web download command and placed in cron for custom automatic data flow.*

## The Server – Client distributed relationship:A Common Sense Approach to Data Distribution

- _[ftp2u](ftp://ftp2u/)_, Great Displays, and GDS allow sub-setting of data by parameter, by 3D region, and by time.
- Fast ftp using Curl comands (http) see
- http://www.cpc.ncep.noaa.gov/products/wesley/fast_downloading_grib.html
- “Great Displays” (pdisp) A display program operating from the same database as ftp2u to display any part of the NCEP data base by time, region, vertical level and variable.
- GDS or DODS or OPeNDAP ...
*ftp* server is present for data sets.

*Tools on the server – **Examples of Clients  (con’t):*

**Notes:**
NASA and NOAA have fast lines between them!

## Tools on the server – 		Client Application Examples:			pdisp (“Great” displays)

- Many features:  Plot type:  map, time series, lat/lon vs time and Animation.
- User selected projection map areas, contours, colors, units, averages ....
- URL address can be copy-pasted into browser or unix script using non-interactive web download command with user changes.  Create products and embed results in other web pages.

## The NCEP server (nomad5) data entry table

## Slide 8

## Slide 9

## Slide 10

## The GrADS-DATA Server (GDS)

- NOMADS participants serve their data sets through a client-server relationship, that is, the data sets are internet ready and the display is done by the client.
- GDS combines both GrADS, a freeware client and DODS (OPeNDAP) server to exchange data in many formats with http protocol.
**This means that server data can appear to the user or client application as a **_**local file**_**!**
*                     **Like Network file system over the web!*
- DODS requests are made by many freeware and commercial high level language clients.
- Simple http queries to the DODS server can create value added products.

- ...and any browser

## ... A  GrADS client example script that animates 3 years of reanalysis height fields and places the date in the title and a color bar below.  A user with GrADS freeware can run this script.   Note the first line, as directed by metadata description in next slide....

- Sdfopen http://nomad3:9090/dods/reanalysis-2/pgb/hgt
- t = 1
- 'set gxout shaded'
- while (t <= 1068)
-   'set t 't
-   'q time'
-   date=subwrd(result,3)
-   'draw title hgtprs 'date' 't
-   'd hgtprs'
-   '/home/wd23ja/grads/lib/cbarn.gs'
- t = t + 1
- endwhile

## Metadata description includes data set location URL address 

*Metadata description includes data set location URL address *
*and all other information about the file and variables.*

## A commercial or freeware client is not necessary to access data...Using http to extract data from the server:

- http://nomad3.ncep.noaa.gov:9090/dods/eta/eta20030527/eta_00z.ascii?tmpprs[0:0][0:0][130:130] [290:290]
- the order of the square bracked values is:
-  [time][level][lat][lon]
-  where lat is measured from the south pole (0) to NP (180)
- In units represented by the metadata descriptor file.
- (Use a “wget” and the URL in cron or a cgi-bin script provides needed values.)

## MATLAB: A Commerical Client

- server=‘nomad2:9090’;
- directory=‘/dods/gdas/rotating/’;  fext=‘ascii’;
- model=‘gdas’;  datestr=‘YYYYMMDDHH’;  varname=‘hgt’;
- varlevs=‘1’;
- Arrc=[‘[‘,t1,’:’,t2,’][‘,nlev1,’:’,nlev2,’][‘,ilat1,’:’,ilat2,’][‘,ilon1,’:’,ilon2,’]’];
- urlstring=[http://’,server,directory,model,datestr,’.’,fext,’?’,varname,arrc];
- url = java.net.URL(urlstring);    % connect to web page
- Try
-      stream = openStream(url)
-      ireader = java.io.InputStreamReader(stream);
-      breader = java.io.BufferedReader(ireader);
- Line = readLine(breader);       disp(line);

## NOMADS Data Sets at NCEP 

-  Wave model forecast      6 hours
-   RTOFS model forecast  daily
- Example of pdisp (“Great” displays) on the Wave model Forecast enp_00z.ctl HTSGWsfc 1 00Z08nov2005

*Metadata descriptor files describe all aspects of the data sets and are created uniquely from the headers of GRIB files.  Data sets at NCEP:*

- http://nomad5.ncep.noaa.gov/cgi-bin/pdisp_wave.sh?ctlfile=enp_00z.ctl&ptype=map&povlp=noovlp&psfile=off&var=HTSGWsfc&level=1&op1=none&op2=none&hour=00Z&day=08&month=nov&year=2005&proj=custom&lon0=-170&dlon=60&lat0=0&dlat=60&type=shaded&cint=def&white=def&fwrite=off&plotsize=800x600&title=&dir=%2Fenp%2Fenp20051108

## RT NOMADS Server at NCEP

- _[http://nomad5.ncep.noaa.gov](http://nomad5.ncep.noaa.gov/)_
- _[http://nomad3.ncep.noaa.gov](http://nomad3.ncep.noaa.gov/)_
- http://nomads6.ncdc.noaa.gov

## Demonstration

##  cat pdisp_script.sh

- wget -o result_log -O- "http://nomad5.ncep.noaa.gov/cgi-bin/pdisp_aofs.sh?ctlfile=rtofs_atl.ctl&ptype=map&povlp=noovlp&psfile=off&var=WTMPsfc&level=1&op1=none&op2=none&hour=00Z&day=03&month=dec&year=2005&proj=atlantic&lon0=280&dlon=50&lat0=-60&dlat=60&type=shaded&cint=def&white=def&fwrite=off&plotsize=800x600&title=&dir=%2Faofs20051201"

- The successful *pdisp* client/user interaction which produced the
- URL address/command in quotes above, is placed
- (copy-pasted) into the script file *pdisp_script.sh *as
- the object of an interactive web download command (like wget)*.*

## Slide 21

##  ./pdisp_script.sh

- <head>
- <title>Display WTMPsfc: NCEP RTOFS(Atlantic) Model</title>
- </head>
- <body bgcolor="#ffffff">
- <h3 align="center">NCEP RTOFS(Atlantic) Model rtofs_atl.ctl</h3>
- <h3 align="center">WTMPsfc 1 00Z03dec2005</h3>
- <FORM NAME="form1" action="pdisp_aofs.sh">
- <center>
- <img src="/tmp/ATEST12963.png" alt="plot">
- </center>
- <pre>
- image was generated
- </pre>
- <small><p>&nbsp&nbsp <p align=left>Problems with the datasets, contact: Jordan.alpert@noaa.gov,bhavani.balasubramaniyan@noaa.gov<br>
- pdisp version v0.9.6.7: Jun.Wang@noaa.gov,Wesley.Ebisuzaki@noaa.gov
- </body>

*The plot is on the server with a*
*unique file name given in the *
*URL address commands: *
*/tmp/ATEST12963.png*
*We use a second wget to get the*
*png file (on the next slide).*

*Pdisp* returns the display plot
- as a web page as shown on
- the previous slide.  The html
- Text on this page results from
- resending the same URL
- command with the
-    interactive web download
-    program wget.

## #!/bin/sh

- set -x
- locnam=`/home/wd23ja/info/rtofs/pdisp_script.sh |grep "\/tmp\/"|cut -d\" -f2`
- echo "locnam $locnam "
- wget -O out.png -o log1.txt "http://nomad5.ncep.noaa.gov${locnam}"

- First execute the *pdisp_script.sh* script above finding the
- line with the characters *“/tmp*/” in it and from it cut away
- the server directory/filename, */tmp/ATEST12963.png* and
- send that location in another *wget* to get the .png picture file
- into the file *out.png* for further viewing.