# Presentation: 906356

## NOAA IOOS Program DIF & Standards

**NOAA IOOS Program ****DIF & Standards**

**23 October 2007**
**CAPT Chris Moore**
**Charles Alexander**
**Marcia Weaks**
**Rebecca Shufford**
**Rob Ragsdale**

## Need a plan!

## Documentation

**NOAA**
- Integrated Ocean Observing System (IOOS) Program Office
**Data Integration Framework (DIF)**
**DIF As-Is Baseline Systems Document**
- (DRAFT Version 0.6)
- September 19, 2007

**NOAA**
- Integrated Ocean Observing System (IOOS) Program Office
**Data Integration Framework **
**(DIF)**
**Functional Requirements Document**
- Version 0.1
- September 17, 2007

**NOAA**
- Integrated Ocean Observing System (IOOS) Program
**Data Integration Framework **
**(DIF)**
**Master Project Plan**
- (Draft Version 0.1)
- September 12, 2007

## NOAA’s Data Integration Framework (DIF) Project Management Structure

- NOAA IOOS Program Senior Staff

- NOAA Observing Systems Council (NOSC)

- Integrated Products Team (IPT)

- Program Support Contractor

- John H. Dunnigan, NOS Assistant Administrator

- NWS System Engineering Center Contractor

- NOAA Chief Information Officer (CIO)

- NOAA Ocean Council (NOC)

- Project Review Team

- Advisory/ Consultation

- Technical Support Contractor(s)

- NOAA IOOS Operations Division Chief

- Project Staff

- Captain Christopher Moore, IOOS DIF Project Manager/ Deputy Director

- Zdenka Willis, IOOS Program Director

- Data Standards Working Group

- Concept of Operations Working Group

- Working Group N

## IPT and Working Groups

## Teams

**DATA**

**DIF**

**CUSTOMERS**

**LMI**

**DIF Team**

**Skjei Telecom**

**IOOS Program**

**IPT**
***Data Standards ***
***CONOPS***

**Project Review Team**
**NOAA Oversight Council**

**Teams**

## Temperature Test May/ June 2007

_**Data**_
** ****Incompatible data formats**
** ****Inconsistent Data structures**
** ****Sources & Queries**
** ****Different forms of temporal/spatial expressions **
** ****All different data dictionaries/vocabularies.**
** ****Varying formats for lat/long**

_**Metadata**_
** ****Metadata generally missing**
_**Information Technology**_** **
**OPeNDAP servers unavailable**
** ****Transport protocols generally correct, some outdated.**
** ****Data not ‘integratable’**

**OGC WFS**

**W3C SOAP Web services**

**OPeNDAP**

**NWS**
**OGC WFS **
**WFS version 1.**

**CO-OPS**
**W3C SOAP**

**NERRS/ CDMO**
**W3C SOAP **

**SEACOOS**
**OPeNDAP DRDS **

**GoMOOS**
**OPeNDAP**

**CORMP**
**OPeNDAP**

**NCDDC**
**OGC WFS **
**WFS version 1.0.0**

**Temperature**
**CSC**
**Jim Boyd, John Ulmer, **
**Daniel Martin**

**Sea level**

**Salinity**

**Temp**

**Currents**

**Color**

**DATA**

_**Status:  Report Complete**_

**Notes:**
Incompatible data formats and content prohibited integration of multiple data sources without programming support and/or advanced end-user client tools (e.g. Matlab, Arc GIS).
Data structures (e.g. row/ column structure) across all services were generally inconsistent.
Very few sources of water temperature data exist where access is through a user-friendly, queriable services based interface.
Each data source uses different forms of temporal and spatial expressions and vocabularies; all different data dictionaries.
Spherical coordinates, latitude and longitude, were generally used, however in varying formats.

Metadata

 Metadata was generally missing from all data sources or difficult to find in conjunction with the source.

Information Technology 
Federal OPeNDAP 3 and 4 servers were unavailable due to a recent security breach.

 Implementation of the actual transport protocols (WFS, SOAP, OPeNDAP) were generally correct, however in some cases outdated versions were in use.

 Utilization of the tested transport standards did not result in data sources that were compatible for integration.

So what did we accomplish?

Volunteer effort
Documentation

CIO status on OPENDAP use still pending

## Alternatives

- Do Nothing
- Hardwire
- Do Something

## Something

- Data Content
- Data Transport
- Metadata
- QA/QC

**Common Data Sharing Infrastructure**

## Data Content

- Vocabulary/Dictionary
- Location Convention (Decimal Degrees, etc)
- Date-Time Convention (JD/Year, UTC, etc.)
- Station ID
- Vertical Datum
- Vertical Position
- Units

- 2007-05-25T12:00:00:00
- <ncddc:ioos.madis_last.ddmmmyyyy>25-May-2007</ncddc:ioos.madis_last.ddmmmyyyy>
-    <ncddc:ioos.madis_last.hour>11</ncddc:ioos.madis_last.hour>
-    <ncddc:ioos.madis_last.minute>56</ncddc:ioos.madis_last.minute>
- 05/29/07 07:45
- 1180695640*
- 52342.000011574**
- * Seconds since January 1, 1970
- ** Decimal days since November 17, 1858

*(Who, what, where, when)*

## Data Content Standard example

## Candidate systems

## Slide 13

## Slide 14

## Standards

***Fall 2007 Submission***

***Fall 2007 Submitted-Proposed Poll***

***Fall 2007 Public Comment***

***Fall 2007 Proposed-Recommended Poll***

***Spring 2008 Submission***

***Spring 2008 Submitted-Proposed Poll***

***Spring 2008 Public Comment***

***Spring 2008 Proposed-Recommended Poll***

***Fall 2008 Submission***

***Fall 2008 Submitted-Proposed Poll***

***Fall 2008 Public Comment***

***Fall 2008 Proposed-Recommended Poll***

***Spring 2009 Submission***

***Spring 2009 Submitted-Proposed Poll***

***Spring 2009 Public Comment***

***Spring 2009 Proposed-Recommended Poll***

## Collaborative Website Support Tool

**Collaborative Website Support Tool**

## IOOS DMAC Standards Process Collaborative Workspace Tutorial

## ?

**?**

** **

|  | Data Discovery | Data Transport | Metadata
 | Uniform on-line Browse
 | Data Archive & Access
 | IT Security | QC/QA |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Grids |  |  |  |  |  |  |  |
| Scattered Points |  |  |  |  |  |  |  |
| Trajectories |  |  |  |  |  |  |  |
| Time Series |  |  |  |  |  |  |  |
| Moving-sensor |  |  |  |  |  |  |  |
| Profiles |  |  |  |  |  |  |  |
| Geospatial Mapping data |  |  |  |  |  |  |  |
| Metadata |  |  |  |  |  |  |  |

**D**
**A**
**T**
**A**
**T**
**Y**
**P**
**E**
**S**

**Sea level**

**Salinity**

**Temp**

**Currents**

**Color**

**DATA**

**Notes:**
Uniform on-line browse, Data Archive & Access, IT security, and QC/QA will not be considered by the DIF

Grids (structured and unstructured – 2 different types) and collections of grids (models, gridded data products)
Scattered points (random sampling, tsunami, species sightings)
Trajectories (u/w ship data, aircraft track data, ocean surface drifters) 
Time series (ocean moorings, fish landings, stream flow) 
Moving-sensor multidimensional fields and collections (satellite, HF radar, etc)
Profiles (salinity profiles, profiling floats)
Geospatial mapping data (coastlines)
And yes – you can have metadata about metadata

## Acknowledgements

- Kurt J Schnebele
- Marcia Weaks
- Dr. Rebecca Shufford
- Rob Ragsdale
- Carmel Ortiz (Skjei Telecom)
- Bill Swart (Skjei Telecom)
- Dr. Chung Wu (NWS SEC)
- Tom Crowley (LMI)
- Delta Pelgrim (LMI)
- Kim Lambert (LMI)
- Dan Henderson
- Dan Martin
- Ken McDonald
- Dr. Roy Mendelssohn
- Steve Hankin
- Ted Habermann
- And all y’all.

## Observation Name, Unit, Domain example

## Vertical Datum Domain example

## Timeline

***Requirements Complete***

***Initial System Ready for Use***

***Development****** Complete***

***Design ******Complete***

## CO-OPS Timeline

- XML Definition (18 days)
- 	Define WSDL (4 days)
- 	Vertical Position and Vertical Datums (5 days)
- 	Testing (4 days)
- Write store procedures and retrieve tasks (22 days)
- 	Get data uses (3 days)
- 	Testing of Stored Procedures (3 days)
- 	Lat/Lon/speed Unit Conversion (5 days)
- 	Time String Manipulation (2 days)
- 	Testing of Procedures (5 days)
- Java Client (8 days)
- 	Write Java Client (4 days)
- 	Testing of Java client (2 days)
- 	Completed Basic Current Web Services (HAB DIF)
- Phase 2 – Web Page Addition (18 days)
- 	Write Try Me Web Page (4 days)
- 	Submit to HTML (5 days)
- 	Plain Text Web service (5 days)
- 	Test Plain Text Web Service (2 days)

**CO-OPS Timeline**

- |October	               |November                           |December

**2007**

## Working Group Plan

**SCOPE OF WORK**
- between data provider and data consumer
- independent of data provide and data consumer native systems
- inclusive of five core parameters and others as defined by
**TASK 1: DATA CONTENT STANDARDS**
- Identify “minimum” content standards for the five core variables
- recognize feature and coverage data models (ISO)
- Core variable vocabularies
- Location convention
- Date-time convention
*	**	**	**	*_*Deliverable:*__ A proposal for common content specifications_
* *
**TASK 2: ENDORSEMENT OF PREVAILING STANDARDS**
- These are well recognized standards that need to be considered as-is, although they may not immediately provide deployment level specificity.
- examine and propose adoption of initial transport standards
- examine and propose adoption of initial metadata standards
*	**	**	**	*_*Deliverable:*__ Annual summary of endorsed standards_
**TASK 3: IMPLEMENTATION SPECIFICATIONS**
- Specializes and/or expands the prevailing standards to a level of detail that is adequate to support deployment.
- Scoping of specification requirements
- Specification development
- Develop and maintenance of reference implementations
- Implementation guidance and other training material
**TASK 4: DATA PROVIDER IMPLEMENTATION**
**Schedule**
- Task 1a – 1e			February
- Task 2					February
- Task 3a
- Task 3b
- Task 3c
- Task 4
**Budget**
**Communications**
**Points of Contact**

## NDBC Timeline

- 1.  Senior System Analysis – IOOS Coordination  - NDBC architect 		($175K)
- 2.  Convert NDBC data to new data base
- 	based on new common data model 					($100K)
- 3. Publish services for data transport ( THREDDS ) 			($152K)
- 4. Publish services for data transport ( SOAP ) 				($100K)
- 5  First Year Meta-data effort 						($475K)
- 6.  Second Year  Meta-data completion effort 				($425K)
- 7.  Maintenance, review and adaptation for compliance
- 	with systems that have required operational data push
- 	( NWS AWIPS, GTS, EMWIN, ...etc...)				_( $200K)_

- $1627K