# Presentation: 882231

## Jordan Alpert, NCEP/EMCPaula Freeman1, Lin Gan1, Chi Kang2and Rick Hackenberg1

- Jordan C. Alpert	_[jordan.alpert@noaa.gov](mailto:jordan.alpert@noaa.gov)_
- 1 NCEP/NCO, 2 NOAA/CIO
- 				4th Ensemble User Workshop 5/14/2008

**where the nation’s climate and weather services begin**

**Jordan Alpert, NCEP/EMC****Paula Freeman****1****, Lin Gan****1****, Chi Kang****2****and Rick Hackenberg****1**

## A digital archive of NOAA’s operational weather models, and an “innovative data access philosophy to promote interoperable access across the geosciences” (BAMS, Rutledge et. al., 2006) 

- A digital archive of NOAA’s operational weather models, and an “innovative data access philosophy to promote interoperable access across the geosciences” (*BAMS, Rutledge et. al., 2006)*
- A Pilot project at NCEP that advances integration of real time model data.
- NOMADS:  An integrator of common web services infrastructure to support the discovery, access and transport of data (*NOAA GEO-IDE Concept of Operations Rept. to the DMC*).

*is a pilot project designed to provide real-time and retrospective format independent access to climate and weather model data:*

## Motivation

- “_Completing the Forecast_” (NRC 1/2006) Recommendation 3.4:   *“NOMADS should be maintained and extended to include *
* **(a) long-term archives of the global and regional*_*** ***__***ensemble ***_*forecasting systems at their native resolution, and *
* **(b) re-forecast datasets to facilitate post-processing.”*
- _The NOAA FY07-11 Baseline Assessment_ (pp10) points out that there is a *“...pressing need for enhancement to present capabilities needs through: improving capabilities to process and disseminate increasing volumes of environmental data.”*
- Recommendation 5 of the National Research Council’s _Fair Weather: Effective Partnerships in Weather and Climate Services states:_ “*The National Weather Service should make its data and products available in accessible digital form*”.

*NOAA’s current needs and the needs of those who “want data”...*

## NCEP 2008 Annual Plan includes the Implementation of NOMADS applicationsin a high availability environment within the NOAA WOC

- NOMADS operating 5 years in experimental development mode
- Proved a Concept
- GDS/OPeN(DAP) DODS:  250,000 queries/day.
- pdisp (“Great Displays”) plot program, 2694/day.
- http access_log  ~200,000 hits/day
- ftp2u, slice, dice and subset files, 8,623/month  repackaged GRIB downloads.

## Real Time NOMADS component at NCEP:

-    *NOMADS offers guidelines, server software and client examples to participants.*
- Serves operational model data sets, GFS, NAM(Eta), Ensembles, RSM, RUC, Reanalysis, etc...,
- “Slice and Dice” ftp with *ftp2u & ftp *application clients, “http”,
- Plot (“Great”) Displays with* pdisp, *application client (_not Impl_),
- GrADS-Data Server (DODS) (OPEN-DAP and GDS) access using open source and XML, to make our server disk storage appear as if it is attached your workstation disk storage!
-  *The GDS framework allows many users to have direct access to the model data base (disk storage) to make the products users need at the users location including automated custom data flow.*

*For the Project that needs to Serve Data and ...**For the Users who Need to Receive Data:*

## NOMADS is a bunch of servers from different locations using the same protocol to distribute model data.

- National Center’s for Environmental Prediction (NCEP) Real Time model data from NOAA’s Operational Models
- Archives of model and other data at the  National Climate Data Center (NCDC) (Separate Ops. Implemented in 2007)
- Other participations are GFDL, PMEL, NCAR, NASA...

## Slide from 

- Slide from
- G. Rutledge

- As of last year....
- (year before that)

## Status of NOMADS High Availability Implementation

_**Lead:**_** **
- Rick Hackenberg, Project Manager
_**Scope:**_** **
- Creating the high availability environment within the NOAA WOC, including porting NOMADS v1.0 functionality to this environment
- Creating the necessary operational support documentation
- Implementing the support model and cost structure developed with the NOAA  WOC, including documented MOU and SLA
- Ensuring that the NOMADS application is properly performance-tuned and  supportable. This includes creating and executing performance testing/tuning  plans.
- Working with the NOAA NCIRT to ensure that the application is secure

## Schedule: On target but ... Range from Date shown to add 1/4

| Milestone | Date | Status |
| --- | --- | --- |
| Project Chartered | 30 Sept 07 | Complete |
| Build Development Environment Development Begins | 22 Oct 2007 | Complete |
| NCIRT Security Review Completed | 15 Feb 2008
26 May 2008 | Analysis complete, NCIRT to report |
| Development Completed | 29 Feb 2008 | Complete |
| Port Application to WOC Development | 07 March 2008 | Complete |
| Integration Testing begins | 15 May 2008 | On Target |
| WOC procure high availability environments | 31 March 2008
15 May 2008 | On Target |
| Performance Testing begins | 09 June 2008 | On Target |
| Experimental Parallel Begins | 23 June 2008 | On Target |
| Evaluation Survey Begins | 21 July 2008 | On Target |
| NOMADS High Availability Live | 15 August 2008 | On Target |

**Status of NOMADS High Availability Implementation, continued...**

## There is a Budget

_**Associated Costs:**_** **
-         NCEP resources needed to support development of applications and install in WOC environment and provide continued application support.
- WOC will procure the IT infrastructure with reoccurring annual upgrade costs (every three years) and support IT infrastructure. EMC provides ongoing develop support.

## Conclusion

- There is a plan for the implementation of NOMADS applications in a high availability environment within the NOAA WOC (also known as ftpprd).
- Scheduled by the end of 2008
- Potentially all Ensemble grib files in Operations (/com/...)
- Normal Operational procedure (JIF) required for data and other changes through Data Review Group.

## Background Slides

## Here is the location of the data  

*Here is the location of the data  *
*in html, shown as a web page, *
*for access from ftp or http.*

*This is the URL with key words *
*which can be modified and re-sent *
*to perform many tasks. *
*Use copy-paste and modify.*

- Parameters:
- file: the data file user selected
- wildcard: the wildcard string user used to select the files
- lev_(lev name):the levels user needs to subset the data file
- var_(var name): the variables user needs
- subregion: if user need extract subregion, on means do extract subregion
- leftlon: left longitude of subregion
- rightlon: right longitude of subregion
- toplat: top latitude of subregion
- bottomlat: bottom latitude of subregion

## Parameters (con;t): 

- results: ways to get data, if results is set to SAVE, save user's data to NOMAD anonymous servers; if results is set to FTP, save user's data to user's sFTP servers.
- rtime: the time to keep data on NOMAD anonymous server
- machine: ftp server for user to save data on user's ftp server,
- user: user name on user's ftp server for user to save data on user's ftp server,
- passwd: user's password on user's ftp server for user to save data on user's ftp server,
- ftpdir: the directory on user's ftp server for user to save data on user's ftp server,
- prefix: the prefix to the data file name on user's ftp server
- dir: the directory on NOMAD where the data user request are located, usually contain the date information.
- ...And if you know a little UNIX scripting language or know some one who does....

## stuff=`wget -O - "http://nomad3.ncep.noaa.gov/cgi-bin/ftp2u_nam.sh?file=nam.t00z.awphys06.tm00&wildcard=&lev_500_mb=on&var_HGT=on&leftlon=0&rightlon=360&toplat=90&bottomlat=-90&results=SAVE&rtime=1hr&machine=140.90.192.197&user=anonymous&passwd=&ftpdir=%2Fincoming_1hr&prefix=&dir=%2Fnam20061030"`

- stuff=`wget -O - "http://nomad3.ncep.noaa.gov/cgi-bin/ftp2u_nam.sh?file=nam.t00z.awphys06.tm00&wildcard=&lev_500_mb=on&var_HGT=on&leftlon=0&rightlon=360&toplat=90&bottomlat=-90&results=SAVE&rtime=1hr&machine=140.90.192.197&user=anonymous&passwd=&ftpdir=%2Fincoming_1hr&prefix=&dir=%2Fnam20061030"`
- echo stuff=$stuff
- location=`echo $stuff|cut -d\> -f8`
- echo location=$location
- export loc2=${location%???}
- echo location=$loc2
- wget "$loc2"

- A UNIX script from a successful ftp2u URL is re-sent to the server by
- a non-interactive web download command, wget, and the results
- retrieved, to find the actual web location of the data, and a second
- wget of that location to download the data. (4 real lines of script...)
- ## for cycle in 00 06 12 18 ; do ; echo “cycle=${cycle}”

*Put the $cycle unix variable in *
*place of the cycle “00” in ...t00z...*
* **and add to the script “sleep 20” ; *
*“**done” to end the loop.  This gets *
*the 6-h forecast from 4 model runs.*

## ftp2u example: Summary

- The unix script can be placed inside a loop with changing values for any of the keywords.
- If these are placed in loops, then we ask that users add a “sleep 20” to stop run away loops, otherwise a NOMADS throttle (denial of service stopper) will add the proper wait.
- Custom and automated dataflow can be accomplished if the script is placed in cron.

## NOMADS and OPENDAP(DODS)/GDS

- Distributed Ocean Data System

- Grads Data Server

## Grads Data Server in NOMADS

*Direct links to access data  (in multiple formats)*

- GDS and Data Application Protocol (DAP)

- This works for
- files that conform to
- the CF standard.

## 	The GrADS-Data Server GDS:

-  NOMADS participants serve their data sets through a client-server relationship.  The data sets have machine and man readable metadata descriptions.
- The display is done by the client.
-  GDS combines both GrADS, a freeware client (from COLA) and DODS (OPeN-DAP) server to compress, cache and exchange data in many formats with http.
**This means that data appears to the user or client application as a **_**local file**_**!**
- DODS requests are made by many freeware and commercial high level language clients like GrADS and MATLAB.
*http queries to the DODS server can create value added products.*

*Open Source*

- The NOMADS matrix of data holdings
**appears to the user or client application **
**as a **_**local file**_**!**
*One was to do this is by running a user client application like GrADS*
- Sdfopen http://nomad3:9090/dods/reanalysis-2/pgb/hgt
- ... another example ....
- Sdfopen http://nomad5.ncep.noaa.gov:9090/dods/enshires/archive/ens20060331/ensn1_00z_1x1
- Sdfopen http://nomad5.ncep.noaa.gov:9090/dods/enshires/archive/ens20060331/ensn2_00z_1x1
-     .
-     .
-     .
**Enter any GrADS commands...**

## ... A  GrADS client example....

- Sdfopen http://nomad3:9090/dods/reanalysis-2/pgb/hgt
- t = 1
- 'set gxout shaded'
- while (t <= 1068)
-   'set t 't
-   'q time'
-   date=subwrd(result,3)
-   'draw title hgtprs 'date' 't
-   'd hgtprs'
-   '/home/wd23ja/grads/lib/cbarn.gs'
-   'swap'
-   t = t + 1
- endwhile

*This script, in GrADS, *
*animates 3 years of *
*reanalysis height data*.
- GrADS and MATLAB
- math functions can be
- applied across the data sets.

## Direct Queries to access data  (format independent)

- Using OPENDAP(DODS)/GDS the user (client) data request is delivered as:
- ascii values (1.234....)
- Units and other information as given in the metadata description file.
- Including the time, level, longitude(s) and latitude(s) of the returned data.
*From the information in the metadata description one could construct a netCDF file.*

## Data location

- Description
- Extent
- Variables, Units...
- Geo-spatial

## ... Using http URLs to extract data from the GDS server:

- http://nomad5.ncep.noaa.gov:9090/dods/enshires/archive/ens20060331/ensn1_00z_1x1.ascii?ugrdprs[2:2][0:0][122:122][255:255]
- the order of the square bracketed values is:
-  [time][level][lat][lon] where for example:
-   lat is measured from the SP (0) to NP (180 on 1x1 degree)
- in units represented in the metadata descriptor file.
- Use a non-interactive web download program like “wget” and the URL in cron or a cgi-bin script to provide new values and custom data flow.
- This simple approach works with ftp2u and pdisp.

## Distinction between nowcast and forecast products

- The entire matrix of initial conditions and R/T NCEP model forecasts for all cycles and past NCDC holdings (archives) are accessible!
- One can use this paradigm for NOAA to create products or users to produce products in real time!
- Done by the user inside the web/script making language of their choice, embed the “wget http://Address_of_the_constrained_query to get information from the matrix of NOAA holdings.

*For Products, if NOMADS was Operational...*

## Access to derived products

- GDS/OPENDAP offers users (client) accessibility to the data which can be in different physical locations simultaneously:
- The archive at NCDC
- The real time NOMADS at NCEP
- Anywhere else there is a DAP server

## Global Ensembles and Event Probability Example:

- NCEP ensembles are constructed from many (10) independent model integrations by changing the initial conditions from adding (p1-p5) and subtracting (n1-n5) the fastest growing global modes from 6-h forecasts.  11 in all with the control.
- Ensembles attempt to span the space of possible forecasts.
- Ensemble members are equally probable forecasts – if not, we apply a suitable normalization.
- Probability estimates are defined simply as the percentage of forecasts of the total number that satisfy a specified weather event.
- We apply this to weather elements like surface  temperature, wind speed or precipitation at a location and future model forecast time.

## Screen shot of the web page containing prompts and user 

- Screen shot of the web page containing prompts and user
- entered responses for the probability of frost at day 4 1⁄2.

- One can try this at:* *
*  **http://nomad5.ncep.noaa.gov/cgi-bin/var/ensprob4.pl*

## Web page showing the URL queries of the ensemble database.  The

- (Probability of frost was 37%.)

## The OPEN-DAP/DODS Constrained Query

- 	http://nomad5.ncep.noaa.gov:9090/dods...
- URL contains the the server address including the port (:9090) /enshires/archive/ens20041014/ensc0_00z_1x1 ...
- 	.ascii?tmin2m[18:18][130:130][255:255]
- The server unpacks GRIB files, finds the minimum 2 meter temperature, tmin2m, but sends only data to the client specified by the constraints in the square brackets for time and location (and level if applicable):
- [18:18] 		indicates the 4 1⁄2 day forecast (6-hr intervals from IC)
- [130:130]   	# of lats north of the South Pole on a 1x1 degree grid.
- [244:244]	# of lons measured eastward from Greenwich meridian.

- Repeat (loop)  for all forecasts from f00 (initial condition) to f180.
- Make a bar graph of results.

## Slide 32

## Summary

- It slices, it dices and more....
- If  you need ftp and more than common; ftp “fast ftp”, then NOMADS is for you.
- Scalable and distributive....  Purchase your own server(s) and (quality) control your own access.
- NOMADS will set up the GDS framework and supply client (user) application examples, like* ftp2u* and *pdisp*.
- NCEP suite of model operations in real time or archives are available as described.
- Conforms to NOAA Data Management Standards (GEO-IDE).

## RT NOMADS Server at NCEP

- http://nomad1.ncep.noaa.gov    (Data)
- http://nomad3.ncep.noaa.gov    (Data)
- http://nomad5.ncep.noaa.gov    (Data)
- http://nomads6.ncdc.noaa.gov  (Backup)
- http://nomads.ncdc.noaa.gov    (Archive)
- MIDAS - DIMES Aggregate Servers for NCEP NOMADS running at GMU:
- http://spring.scs.gmu.edu:8199/nomad

## Slide 35