# Presentation: 397751

## NOAA IOOS Data Integration Framework:Draft Architecture andRecommended Services

- Jeff de La Beaujardière, PhD
- NOAA IOOS Program
- DIF Sr Systems Architect

## Primary DIF Partners

- Data Providers
- NWS NDBC (National Data Buoy Center)
- NOS CO-OPS (Ctr for Operational Oceanographic Prod & Svcs)
- NESDIS CoastWatch
- Customer Focus Areas
- HAB (Harmful Algal Bloom Forecast System)
- IEA (Integrated Ecosystem Assessments)
- CI (Coastal Inundation)
- HI (Hurricane Intensification)
- Regional associations
- As represented in Web Services working group
- ... and in new Regional Implementation group

## Recommended Web Services and Data Encodings

**Recommended Web Services and Data Encodings**

- In-situ data (buoys, piers, towed sensors)

- Gridded data (model outputs, satellite)

- OGC Sensor Observation Service (SOS)

- OpenDAP and/or
- OGC Web Coverage Service (WCS)

- XML based on OGC Observations and Measurements (O&M)

- NetCDF using Climate and Forecast (CF) conventions

- Images of data

- OGC Web Map Service (WMS)

- GeoTIFF, PNG etc.-possibly with standardized styles

*[*OGC = Open Geospatial Consortium]*

**Data Type**

**Web Service**

**Encoding**

**Notes:**
To promote standardized access to data we have recommended particular web services and data encodings.
 First of all, we broadly classify data into three types: in-situ data, gridded data, and images of data. Each of these uses a different type of service.
 For in situ data, we recommend the Open Geospatial Consortium (OGC) Sensor Observation Service. SOS lets you ask for data from one or more sensors, and to ask for detailed information about the characteristics of a particular sensor. For gridded data, we recommend the OGC Web Coverage Service. WCS lets you ask for data values in the geographic region of interest to you. Finally, for serving images of data we recommend the OGC Web Map Service.
 Each of these services can send data in different encoding formats. For in-situ data, we are recommending an XML (Extensible Markup Language) format based on the OGC Observations and Measurements specification. For gridded data, we use the NetCDF format coupled with the Climate & Forecast conventions. Finally, for images formats such as GeoTIFF and PNG are suitable.

## Component Implementations for DIF (2008)

**Component Implementations for DIF (2008)**

- Client
- Components

- Browser

- G.Earth

*ESRI*

**HAB-FS**

- IEA

- CI

*NOAA Customers*

- ERDDAP

- OSMC

- HI

***Technology Viewpoint from Reference Model for Open Distributed Processing (RM-ODP)***

**Notes:**
This slide shows the actual implementations of various components that are now either underway or in planning as part of the DIF.
We are now testing OGC Sensor Observation Service (SOS) implementations at two major NOAA data providers, NDBC and CO-OPS, providing access to data from NDBC moorings, IOOS RCOOSes, and DART, TAO, NWLON and PORTS stations.
We are also testing OpenDAP and OGC Web Coverage Service (WCS) access to HFR data at NDBC.
CoastWatch is working on OpenDAP and WCS access to MODIS ocean color data.
We are working with the IOOS RAs to have them also establish the same data access services if they have not already done so.
NDBC will establish a Web Map Service (WMS) to provide visualizations of  their data holdings.
ERDDAP has just been enhanced to support WMS, and is also being configure as a translation service.
The Harmful Algal Bloom Forecast System (HAB-FS) has been enhanced to obtain in situ ocean currents data from the DIF SOS implementations.

## Possible Next StepsTentative, partial list; dependent on funding & requirements

- SOS DescribeSensor + SensorML implementation
- Testing/evaluation/refinement of existing work
- Software reference implementations
- Metadata for discovery and QA/QC
- Model Data Access
- Additional customers
- Additional variables & data providers
- Build or borrow needed components in the architecture
- Catalog of available data
- Registry of terms, relationships, sensors
- Data translation service

## Unused Slides

## Outline

- DIF overview
- Draft architecture diagrams *(partial set)*
- Recommended web services for data access
- Data provider status
- Next steps

## NOAA IOOS Goals

- Modernize the way NOAA collects, shares, and uses ocean information
- Increase data interoperability and efficiency of operations across NOAA and the eleven IOOS regions
- Expedite access to data for improved decision making

## NOAA IOOS Program Components

- Standards
- Develop and maintain a process to document, coordinate, evaluate and accept standards
- Regions
- Invest in Regional Association efforts that contribute to the definition and implementation of IOOS
- Data Integration Framework (DIF)
- Conduct a limited-scope implementation of IOOS as a proof of concept and to gather lessons to inform the larger IOOS effort

**Notes:**
...is focusing its efforts in 3 areas....read slide

## Data Integration Framework (DIF)

- First spiral of IOOS development
- Standardize on small number of services & encodings
- Implement at selected provider & customer sites
- Start with several core variables
- Currents
- Temperature
- Salinity
- Water Level
- Winds
- Waves
- Ocean Color (chlorophyll)
- Evaluate in FY 2010

## Architectural Layers

- Observing Systems

- Data Providers

- Data Access Services

- Utility Services

- Client Components

- _ISO 3-Layer Model_

- Data Access Tier

- Business Process Tier

- User Interface Tier

*Observing Systems*

*Data Management*
*and Communications*
*(DMAC)*

*Modeling & Analysis*

_*IOOS “Subsystems”*_

- DRAFT

**Notes:**
One architectural approach is to decompose the system into layers providing different functionalities.
The middle stack shows the architectural layers considered in IOOS.
These are mapped on the left to the traditional ISO 3-layer model, with two additional layers below to account more explicitly for the data production, assembly and archiving functions.
These are mapped on the right to the so-called IOOS “subsystems.” Note that the “DMAC” subsystem actually embodies several distinct layers in the architecture.

## Crosscutting Layers 

- Observing Systems

- Data Providers

- Data Access Services

- Utility Services

- Client Components

- IT Security

- Operational Reliability

- Data Quality Assurance

- Metadata

- DRAFT

**Notes:**
Several crosscutting layers or themes must also be considered.
These include IT security, operational reliability, and data quality.
In practice, in a distributed system IT security and operational reliability are often applied at the individual component level (e.g., every server must conform to IT security regulations, even in the absence of a system-wide IT Security guardian).
Data quality control may be performed at a variety of layers and components, but it is important to have an end-to-end capability for accessing data lineage and QA/QC metadata.

## Component Types Needed for IOOS

***Engineering Viewpoint from Reference Model for Open Distributed Processing (RM-ODP)***

**Component Types Needed for IOOS**

- DRAFT

**Notes:**
This slide shows the types of components that will be needed to create a functioning IOOS. This is only one viewpoint of the system architecture, and may not be complete (comments are welcomed).
The bottom layer represents observing systems, including assets in the water and satellite observations. Models are listed here to represent their role as sources of numerical information.
The next layer represents data providers – the organizations that actually gather and serve data.
The middle layer indicates some of the kinds of data access services that are needed. Users must be able to request in-situ data, remote-sensing data, and model outputs. Those are not necessarily all provided by different services. There may also be a gateway that translates requests between different kinds of services.
The next layer shows additional services needed beyond basic data access. We need services to visualize data, perhaps to convert data into other formats, to integrate data from multiple sources into a single package, a Catalog to enable users to find data, and a Registry of all services and definitions.
Finally, the upper layer represents various types of customers. The DIF is focusing initially on 4 NOAA customers, but is designed to support all potential users of oceanographic data.

## “Map” = georeferenced picture of data
GetCapabilities operation: “table of contents” in standardized format
GetMap operation:image of data customizedaccording to:
Variable(s) of interest
User-specified bounding box
User-specified time
Image size
File format (e.g., PNG,GetTIFF, JPEG, GIF) 

- GetCapabilities operation: “table of contents” in standardized format
- GetMap operation:image of data customizedaccording to:
- Variable(s) of interest
- User-specified bounding box
- User-specified time
- Image size
- File format (e.g., PNG,GetTIFF, JPEG, GIF)

**WMS for Images of Data*****OGC Web Map Service***

- ← width →

- ← height →

## WCS for Gridded DataOGC Web Coverage Service

| 27 | 1828 | 1828 | 4590 | 4523 | 5360 | 2874 | 7135 | 2662 | 4977 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 9676 | 2772 | 4076 | 6303 | 5354 | 7594 | 5713 | 8217 | 8525 | 1664 |
| 9218 | 1741 | 3596 | 6290 | 4357 | 2900 | 3342 | 9526 | 595 | 6307 |
| 2338 | 2988 | 753 | 1952 | 5101 | 9011 | 5738 | 3418 | 7930 | 7021 |
| 2447 | 6146 | 668 | 822 | 6480 | 168 | 4774 | 1185 | 3742 | 3454 |
| 695 | 5170 | 2761 | 8386 | 626 | 1331 | 3845 | 8300 | 752 | 449 |
| 2007 | 932 | 8709 | 1274 | 4374 | 7047 | 2306 | 9697 | 7209 | 3101 |
| 6574 | 6377 | 2111 | 2523 | 8978 | 4425 | 569 | 5369 | 6770 | 7854 |
| 9879 | 3163 | 6889 | 2300 | 9879 | 3127 | 7361 | 7821 | 5424 | 9992 |
| 1936 | 6803 | 3182 | 5288 | 6939 | 8496 | 4651 | 582 | 939 | 2398 |
| 1173 | 123 | 8197 | 684 | 1614 | 397 | 198 | 3767 | 9320 | 6832 |
| 3287 | 8250 | 9819 | 4558 | 1530 | 1756 | 7173 | 6133 | 2069 | 8112 |
| 3515 | 9888 | 8519 | 3458 | 727 | 3866 | 7385 | 8942 | 2879 | 2284 |
| 6104 | 8419 | 8444 | 3634 | 6324 | 4968 | 4875 | 6023 | 3624 | 8270 |
| 2353 | 436 | 9941 | 8491 | 4631 | 4093 | 4317 | 3814 | 3640 | 5462 |
| 167 | 6839 | 6424 | 3781 | 4059 | 2714 | 5635 | 4906 | 1303 | 1072 |
| 7041 | 7189 | 8610 | 6873 | 9696 | 5521 | 2671 | 5468 | 8957 | 350 |
| 3210 | 6817 | 121 | 56 | 2788 | 235 | 1930 | 3322 | 4745 | 158 |
| 5036 | 6041 | 6997 | 3297 | 2508 | 8687 | 6966 | 4035 | 5570 | 7162 |
| 7871 | 3419 | 5124 | 6652 | 103 | 592 | 1236 | 6771 | 9432 | 5278 |

**WCS for Gridded Data*****OGC Web Coverage Service***

- Coverage ~ array of gridded data values
*(simplified viewpoint for this discussion – coverage can be more complex)*
- GetCapabilities operation: “table of contents”
- GetCoverage operation: data filecontaining header and array(s) ofnumbers customized for:
- Variable of interest
- User-specified bounding box
- User-specified time
- File format (e.g, NetCDF,HDF, floating-point TIFF)
- DescribeCoverage operation:metadata about a specific dataset

## SOS for In-Situ DataOGC Sensor Observation Service

- Sampling feature = discrete location(s) of measurements
*Point, Vertical or Horizontal Profile, Trajectory (e.g., ship track)*
*...**and Time Series or Collections thereof *
- GetCapabilities operation: “table of contents”
- GetObservation operation: XML data filecontaining observation values for desired:
- Variable(s) of interest
- Bounding box
- Or perhaps named geographic feature of interest
- Or perhaps a single sensor
- Time
- DescribeSensor operation:XML providing detailed information about a specific sensor(or platform or group of sensors)

**SOS for In-Situ Data*****OGC Sensor Observation Service***

## SOS Concept

**Collection**
**Of Sensors**

**SOS Concept**

**S**ensor
**O**bservation
**S**ervice

- Provides access to

- SOS
- Client

**GetObservation request**

**GetObservation response**

- Observation

- Units
- Dictionary

- Phenomena
- Dictionary

- Schema

**References**

**Constrained by**

**References**

*Based on slide from Open Geospatial Consortium*

## SOS GetObservation ResponseXML Encoding of In-Situ Data

- XML
- _Extensible Markup Language_Generic method for structuring text data

- OGC GML_Geography Markup Language_XML that can represent any geospatial feature

- OGC O&M_Observations and Measurements Model_GML that describes the act of measuring real-world phenomena and the result of the measurement

*specialized**by*

*specialized**by*

## Data Provider Status(expected by end of CY2008)

**Sensor Observation Service**

**NDBC DAC****- buoys, **
**C-MAN, VOS, NOS NERRS, etc.**

**IOOS DAC****- Regional observations**

**TAO DAC****- Tropical Atmosphere Ocean**

**DART DAC****- Deep-Ocean Assessment and Reporting of Tsunamis**

**National Water Level Observation Network**

**PORTS®**

**NDBC**

**CO-OPS**

***User***
***Requests***

**Sensor Observation Service**

**High-Frequency Radar (HFR) surface currents**

**Web Coverage Service**
**& OPeNDAP**

*Data Content & Encoding Standards*

*Data Content & Encoding Standards*

**Web Coverage Service**
**& OPeNDAP**

**Satellite Ocean Color (Aqua MODIS, Terra MODIS, SeaWiFS)**

**CoastWatch**

*Data Content & Encoding Standards*

*Data Integrated from One Provider*

*Data Integrated from One Provider*

*Data Integrated from One Provider*

- NDBC Data Assembly Centers (DACs)

_*IOOS Variables*_*: temperature, salinity, currents,**water level, winds, waves, ocean color (chlorophyll)*