# Presentation: 271678

##  NCDC GRID and NOMADS Approach TeamKickoff Meeting

- June 18th, 2004
- Participants:
- _CSD              DO__	__               DOD__	__           ScSD__	__          RSAD               SSD_
- Vicki Wright   Tom Karl           Wayne Faas        Glenn Rutledge      Brian Nelson     Preston Carter
- 	      Sharon LeDuc   Gus Shumbura    Dave Anderson
- 	      John Jenson

## Advancement of GRID & NOMADS Related Activities

- Purpose
- Identify areas of internal and external activities to advance both NOMADS (Web/GIS), and GRID based technologies
- Scope
-  NOMADS
-  Work to add new datasets including Satellite,   Radar, and in-situ.  Dvlp linkages to CLASS.
- GRID
- Participate in existing Gird partnerships and work to promote these activities at the NOAA level.

## The Partnerships

- 		**CDC **
**	****	****COLA   ****co-PI**
**	****	****FSL**
**	****	****GFDL   co-PI **
**	****	****LLNL   co-PI**
**	****	****NCAR**
**	****	****NCDC   PI**
**	****	****NCEP    co-PI**
**	****	****PMEL    ****co-PI**
**	****	****Unidata**
-   **United Nations FAO (in progress)**

**	**

*The Partnerships*

## NCDC System Architecture

- NOAAPort

- Unidata IDD

- NCEP ftp

** ****Data & Directory **
**   ****structures “merged”**
** ****Daily Data Ingest **
**   ****inter-comparison **
** ****QC and R/T Monitoring**
** ****Index File generation**
** ****Control and OPeNDAP **
**   ****metadata generation  **
** ****CVS Backup (code)**
** ****NCDC Archive Interface**

** ****Earth System **
**Grid & **
**CEOS-Grid**

**Obs, Eta, **
**GFS, RUC**

**Hi-Res**
**GFS, Eta, **
**NARR and**
**GDAS**

_**Data Ingest**_

**Dual**
**Redundant**
**Ingest**

_**Data Management**_

_**Data Access**_

**NCDC**
**Archive**

**NOMADS**
**Web/DODS**

*NCDC System Architecture*

- GigaPOP

## The NOMADS Philosophy

*The NOMADS Philosophy*

_**Multiple paths to format independent data access:**_

## NCDC Grid & NOMADS Team

- into NCDC Divisions for both the sake of clarity, and to
- provide Division level oversight.  It is also provided to initiate team discussion.
-  The activities are based upon an analysis of what is doable under the current NOMADS and exiting Grid efforts as I perceive them.
- The activities are divided into two sections: “NOMADS” and “Grid”

## NOMADS Related Activities 1

- Focus: ScSD-DOD-CSD
- NCDC (Faas/Lott/Rutledge) are advancing NOMADS into CSD.  A comprehensive plan has been developed and most functionally already tested that will provide NOMADS capabilities within CSD.  This effort will use NOMADS to service SRRS model and bufr observations; and NDFD Grib2 products for internal and external users.  It is partially funded by NWS.  This activity is on track.
*Recommendation*:  None.

## NOMADS Related Activities 2

- Focus: ScSD-CSD
- If NCDC is to advance distributed access, beyond SRRS and NDFD datasets especially in the OGC (GIS) environment, an additional CSD resource is suggested on the team.
*Recommendation:*  A new CSD member to attend and participate in NOMADS activities in the broader external community specifically ESP, THREDDS, and OPeNDAP communities, to leverage the work of Web-OGC-Grid and relational database efforts already underway.

## NOMADS Activities 3

- Focus: All Divisions
- The ScSD NOMADS activity draws upon resources across all NCDC Divisions.  This inter-Divisional effort, thru cooperation at the working level has proven quite successful.  On a monthly basis, NOMADS currently provides 100,000’s of downloads per month.
- Active participation include: Systems, Ingest, Access (thru NOMADS and now thru the CSD)  and Archive branches have all contributed to this success.
*Recommendation:*  New data/product development

## NOMADS Activities 4

- Focus: ScSD
- The development of a ScSD “Reanalysis” effort, which includes a model-to-observational inter-comparison effort, is currently stalled.  Oversight of the NOMADS activities, including the Grid, has been more time consuming then anticipated.
*Recommendation:*  A new modeler resource to develop long-term dynamically driven improvements to observations.  Engage others ScSD.

## NOMADS Activities 5

- Focus: RSAD, ScSD
- CLASS has the (longer term) capability to provide distributed (OPeNDAP) and Grid-based technologies (XML/NetCDF, etc.).
- Recommendation: Work toward the converge of a NOMADS-CLASS interface.  Metadata “standards” & conventions are key.

## The Grid

- Focus: RSAD, SSD and ScSD
- This effort will require collaboration with external Grid partners.  There are two major groups that NOMADS is currently engaged: Earth System Grid through the ESP; and the CEOS-Grid.
- Caution: Direct involvement with the Earth System Systems Grid and CEOS-Grid should maintained since COES-Grid and ESG may be on divergent paths (at least for the present time).  CEOS has a focus on satellite, while ESG has a focus on Model.  This needs to be monitored for a comprehensive long-term NOAA gird infrastructure to emerge.

## Grid Activities 1

- Focus: RSAD, ScSD, SSD
- NOMADS is an active participant within the CEOS-Grid.  NCDC has a Globus server w/ GridFTP capabilities in a pilot environment across the CEOS-Grid partners (several groups for the specific application of data transfer and inter-comparison).
*Recommendation:*  RSAD, and SSD join in on CEOS-Grid monthly telecoms and ESP/NOMADS users and working groups already established.  SSD involvement would be significant.

## Grid Activities 2

- Focus: RSAD, SSD, ScSD
- Forecast System Lab (FSL) has initiated a NOAA gird infrastructure effort.  NCDC thru NOMADS-ESP has been requested to collaborate with GFDL and FSL.
*Recommendation:*  RSAD, SSD join in on ScSD discussions toward a “DataGrid” pilot within NOAA.  NCDC (w/DO), FSL, and GFDL conduct NOAA level briefings on both grid, ESP and NOMADS.

## Grid Activities 3

- Focus: DOD, ScSD
- Distributed access to Radar data could be a DOD activity.  U of Alabama has developed an ESML-OPeNDAP Radar Level-II interface.  ESG (NCAR) has developed an OPeNDAP-G interface.
*Recommendation:*  DOD and ScSD develop “NOMADS Radar Server” based on these open technologies with U of AL and ESG.

## Grid Activities 4

- Focus: RSAD ScSD and SSD
- Model to Satellite data effort w/ GMU.  GMU has successfully collaborated with NOMADS for a satellite (modis) to model (Eta) format “converter to display and make avbl these data through OGC (GIS) services.
*Recommendation:* RSAD and SSD participate in CEOS-Grid (w/ ScSD) for the development of metadata and the NOMADS-GMU collaboration.

## External Collaborations

- NOMADS has developed many external collaborations.  NCDC can participate in those that advance NCDC and NOAA goals.
- The following programs are already interested in partnering.  They include National and International projects:

## Collaborating Programs

*Collaborating Programs*

**CAP     Climate Action Partnership                 DOC DOE EPA State Dept**
- CDP     Community Data Portal 			NCAR
**CEOSGrid  Committee on EO Satellites****	****NOAA ****Representative**
- CEOP   Coordinated Earth Obs Period    	NOAA Representative
- EPA     Air Quality Models 			(in progress)
**ESP      Earth Science Portal****	**** ****	****	****Member**
- European PRISM
- NASA GCMD    				Science Advisory Board
- NERC DataGrid + BADC			Advisory Committee
- NSF Cyberinfrastructure 			Member
**NSF** **LEAD GTF GeoScience Tech Forum**	Planning Committee
- NVODS / US GODAE / GOOS			Data Provider
**Unidata THREDDS,  NSDL,  DLESSE****	****	****Data Provider**
**WCRP World Climate Research Program  ****	****JSC/CLIVAR**
**WMO**						(in progress)

## http://lead.ou.edu/gtf.htm

## The Grid: An IT Infrastructure for NOAA in the 21st Century Mark Govett, Mike Doney, Paul Hyder

**To meet challenges facing NOAA in the next decade and beyond**
**100 Fold Increase in Data Volume in 10 years**
**GOES-R, NPOESS, IEOS, Radar, GPS**
**More Complex Modeling Systems**
**higher resolutions, ensembles, data assimilation, more data**
**Proposes an Integrated IT Infrastructure based on Grids **
**Build on existing NOAA Programs (e.g. CLASS, NOMADS)**
**Develop Compute, Data and Service Grids**
**Enable Dynamic Data Discovery, Access, Integration**
**Utilize / Develop Web Services, Grid Portals**

## Suggested Reading

- NOMADS
- _[http://www.ncdc.noaa.gov/oa/model/model-resources.html](http://www.ncdc.noaa.gov/oa/model/model-resources.html)_
- http://www.ncdc.noaa.gov/oa/climate/nomads/nomads.html
- _[http://www.ncdc.noaa.gov/oa/climate/nomads/program-plan.html](http://www.ncdc.noaa.gov/oa/climate/nomads/program-plan.html)_
- _[http://www.ncdc.noaa.gov/oa/model/publications/publications.html](http://www.ncdc.noaa.gov/oa/model/publications/publications.html)_
- Earth Science Portal
- _[http://esportal.gfdl.noaa.gov/](http://esportal.gfdl.noaa.gov/)_
- Earth System Grid
- _[https://www.earthsystemgrid.org/](https://www.earthsystemgrid.org/)_
- CEOS-Grid
- _[http://www.ncdc.noaa.gov/oa/climate/nomads/ceos-grid-plan-20020905.pdf](http://www.ncdc.noaa.gov/oa/climate/nomads/ceos-grid-plan-20020905.pdf)_ and
- _[http://grid-](http://grid-tech.ceos.org/gridwiki)__[tech.ceos.org/gridwiki](http://grid-tech.ceos.org/gridwiki)_ (username/password - ceos-grid/grid-tech)

## Next Steps?