**UML Description: Level 2 Use Cases and Analysis View**

Boeing Corporation and Battelle Ocean Sciences

An IOOS Demonstration Project

July 2006

NOAA Coastal Services Center

2234 South Hobson Avenue • Charleston, SC 29405-2413\
843-740-1200 (phone) • 843-740-1224 (fax)

This page intentionally blank

Table of Contents

1 Scope [1](#scope)

1.1 Identification [1](#identification)

1.2 ICOOS IIS Data Mart Use Cases [1](#icoos-iis-data-mart-use-cases)

1.2.1 Manage Web Page Data Transfer (Use Case)
[3](#manage-web-page-data-transfer-use-case)

1.2.2 Conduct Web Page Transfer (Association)
[5](#conduct-web-page-transfer-association)

1.2.2.1.1 Provide Web Page (Acquire Data Role)
[5](#provide-web-page-acquire-data-role)

1.2.2.1.2 Capture Web Page (Manage Web Page Data Transfer Role)
[5](#capture-web-page-manage-web-page-data-transfer-role)

1.2.3 Supply Mixed-Content Web Page Data (Association)
[5](#supply-mixed-content-web-page-data-association)

1.2.3.1.1 Extract Data From Web Page (Manage Web Page Data Transfer
Role)
[6](#extract-data-from-web-page-manage-web-page-data-transfer-role)

1.2.3.1.2 Standardize Extracted Web Page Data (Integrate Mixed-Content
Data Role)
[6](#standardize-extracted-web-page-data-integrate-mixed-content-data-role)

1.2.4 Integrate Mixed-Content Data (Use Case)
[6](#integrate-mixed-content-data-use-case)

1.2.5 Supply Standardized Data (Association)
[7](#supply-standardized-data-association)

1.2.5.1.1 Provide Standardized Data (Supply Standardized Data Role)
[7](#provide-standardized-data-supply-standardized-data-role)

1.2.5.1.2 Store Standardized Data (Record Standardized Data Role)
[8](#store-standardized-data-record-standardized-data-role)

1.2.6 Conduct OPeNDAP Data Transfer (Association)
[8](#conduct-opendap-data-transfer-association)

1.2.6.1.1 Provide Data Per OPeNDAP (Acquire Data Role)
[8](#provide-data-per-opendap-acquire-data-role)

1.2.6.1.2 Capture OPeNDAP Data Files (Manage OPeNDAP Data Transfer Role)
[8](#capture-opendap-data-files-manage-opendap-data-transfer-role)

1.2.7 Manage OPeNDAP Data Transfer (Use Case)
[8](#manage-opendap-data-transfer-use-case)

1.2.8 Supply Mixed-Content OPeNDAP Data (Association)
[11](#supply-mixed-content-opendap-data-association)

1.2.8.1.1 Extract Data From OPeNDAP Download (Manage OPeNDAP Data
Transfer Role)
[11](#extract-data-from-opendap-download-manage-opendap-data-transfer-role)

1.2.8.1.2 Standardize Extracted OPeNDAP Data (Integrate Mixed-Content
Data Role)
[11](#standardize-extracted-opendap-data-integrate-mixed-content-data-role)

2 ETL Program Level 2 Analysis View
[12](#etl-program-level-2-analysis-view)

2.1 Timer (Class) [15](#timer-class)

2.1.1 Period (Timer Attribute) [15](#period-timer-attribute)

2.1.2 Start Signal (Timer Attribute) [15](#start-signal-timer-attribute)

2.2 Initiates (Association) [15](#initiates-association)

2.2.1 Issue Start (Timer Role) [15](#issue-start-timer-role)

2.2.2 Start ETL Program (Transformation Manager Role)
[16](#start-etl-program-transformation-manager-role)

2.3 Transformation Manager (Class) [16](#transformation-manager-class)

2.3.1 Executive (Transformation Manager Attribute)
[16](#executive-transformation-manager-attribute)

2.3.2 Process Monitor (Transformation Manager Attribute)
[16](#process-monitor-transformation-manager-attribute)

2.3.3 Scraper Sequencer (Transformation Manager Attribute)
[17](#scraper-sequencer-transformation-manager-attribute)

2.3.4 Retrieval Sequencer (Transformation Manager Attribute)
[17](#retrieval-sequencer-transformation-manager-attribute)

2.3.5 Storage Sequencer (Transformation Manger Attribute)
[17](#storage-sequencer-transformation-manger-attribute)

2.3.6 OPeNDAP Interface Sequencer (Transformation Manger Attribute)
[17](#opendap-interface-sequencer-transformation-manger-attribute)

2.4 Coordinates Actions of (Association)
[17](#coordinates-actions-of-association)

2.4.1 Control Data Flow (Transformation Manager Role)
[17](#control-data-flow-transformation-manager-role)

2.4.2 Provide Interface (Data Storage Interface Role)
[18](#provide-interface-data-storage-interface-role)

2.5 Data Storage Interface (Class) [18](#data-storage-interface-class)

2.5.1 Station Table Interface (Data Storage Interface Attribute)
[18](#station-table-interface-data-storage-interface-attribute)

2.5.2 Location History Table Interface (Data Storage Interface
Attribute)
[18](#location-history-table-interface-data-storage-interface-attribute)

2.5.3 Collection Platform Table Interface (Data Storage Interface
Attribute)
[18](#collection-platform-table-interface-data-storage-interface-attribute)

2.5.4 Operational Status Table Interface (Data Storage Interface
Attribute)
[18](#operational-status-table-interface-data-storage-interface-attribute)

2.5.5 Collection Sensor Table Interface (Data Storage Interface
Attribute)
[19](#collection-sensor-table-interface-data-storage-interface-attribute)

2.5.6 Sensor Table Interface (Data Storage Interface Attribute)
[19](#sensor-table-interface-data-storage-interface-attribute)

2.5.7 Measurement Table Interface (Data Storage Interface Attribute)
[19](#measurement-table-interface-data-storage-interface-attribute)

2.5.8 Station Type Table Interface (Data Storage Interface Attribute)
[19](#station-type-table-interface-data-storage-interface-attribute)

2.5.9 Benchmark Reference Table Interface (Data Storage Interface
Attribute)
[19](#benchmark-reference-table-interface-data-storage-interface-attribute)

2.5.10 Benchmark Table Interface (Data Storage Interface Attribute)
[19](#benchmark-table-interface-data-storage-interface-attribute)

2.5.11 Extraction Type Table Interface (Data Storage Interface
Attribute)
[19](#extraction-type-table-interface-data-storage-interface-attribute)

2.5.12 Participating Station Table Interface (Data Storage Interface
Attribute)
[20](#participating-station-table-interface-data-storage-interface-attribute)

2.5.13 Observation Type Table Interface (Data Storage Interface
Attribute)
[20](#observation-type-table-interface-data-storage-interface-attribute)

2.5.14 Observation Structure Table Interface (Data Storage Interface
Attribute)
[20](#observation-structure-table-interface-data-storage-interface-attribute)

2.5.15 Observation Attribute Table Interface (Data Storage Interface
Attribute)
[20](#observation-attribute-table-interface-data-storage-interface-attribute)

2.5.16 Observed Data Table Interface (Data Storage Interface Attribute)
[20](#observed-data-table-interface-data-storage-interface-attribute)

2.5.17 Range Edit Table Interface (Data Storage Interface Attribute)
[20](#range-edit-table-interface-data-storage-interface-attribute)

2.5.18 Discrete Edit Table Interface (Data Storage Interface Attribute)
[21](#discrete-edit-table-interface-data-storage-interface-attribute)

2.5.19 Exclude Edit Table Interface (Data Storage Interface Attribute)
[21](#exclude-edit-table-interface-data-storage-interface-attribute)

2.5.20 OPeNDAP and Predictions Tables Interfaces(Data Storage Interface
Attribute)
[21](#opendap-and-predictions-tables-interfacesdata-storage-interface-attribute)

2.6 Exchanges Observation Data with (Association)
[21](#exchanges-observation-data-with-association)

2.6.1 Issue Requests and Data (Data Storage Interface Role)
[21](#issue-requests-and-data-data-storage-interface-role)

2.6.2 Store and Retrieve (Relational Data Base Manager Role)
[21](#store-and-retrieve-relational-data-base-manager-role)

2.7 Are Fetched by (#1) (Association)
[21](#are-fetched-by-1-association)

2.7.1 Retrieve (Data Storage Interface Role)
[22](#retrieve-data-storage-interface-role)

2.7.2 Wait for Use (Scraper Settings Role)
[22](#wait-for-use-scraper-settings-role)

2.8 Are Fetched by (#2) (Association)
[22](#are-fetched-by-2-association)

2.8.1 Retrieve (Data Storage Interface Role)
[22](#retrieve-data-storage-interface-role-1)

2.8.2 Wait for Use (OPeNDAP Settings Role)
[22](#wait-for-use-opendap-settings-role)

2.9 Scraper Settings (Class) [22](#scraper-settings-class)

2.9.1 Extraction Type (Scraper Settings Attribute)
[23](#extraction-type-scraper-settings-attribute)

2.9.2 Participating Station (Scraper Settings Attribute)
[23](#participating-station-scraper-settings-attribute)

2.9.3 Station Summary (Scraper Settings Attribute)
[23](#station-summary-scraper-settings-attribute)

2.9.4 Observation Type (Scraper Settings Attribute)
[23](#observation-type-scraper-settings-attribute)

2.9.5 Observation Structure (Scraper Settings Attribute)
[23](#observation-structure-scraper-settings-attribute)

2.9.6 Attribute Summary (Scraper Settings Attribute)
[23](#attribute-summary-scraper-settings-attribute)

2.9.7 Observation Attribute (Scraper Settings Attribute)
[23](#observation-attribute-scraper-settings-attribute)

2.9.8 Range Edit (Scraper Settings Attribute)
[24](#range-edit-scraper-settings-attribute)

2.9.9 Discrete Edit (Scraper Settings Attribute)
[24](#discrete-edit-scraper-settings-attribute)

2.9.10 Exclude Edit (Scraper Settings Attribute)
[24](#exclude-edit-scraper-settings-attribute)

2.9.11 Station URL (Scraper Settings Attribute)
[24](#station-url-scraper-settings-attribute)

2.10 OPeNDAP Settings (Class) [24](#opendap-settings-class)

2.10.1 Extraction Type (OPeNDAP Settings Attribute)
[25](#extraction-type-opendap-settings-attribute)

2.10.2 Participating Data Source (OPeNDAP Settings Attribute)
[25](#participating-data-source-opendap-settings-attribute)

2.10.3 Data Source Summary (OPeNDAP Settings Attribute)
[25](#data-source-summary-opendap-settings-attribute)

2.10.4 Observation Type (OPeNDAP Settings Attribute)
[25](#observation-type-opendap-settings-attribute)

2.10.5 Observation Structure (OPeNDAP Settings Attribute)
[25](#observation-structure-opendap-settings-attribute)

2.10.6 Attribute Summary (OPeNDAP Settings Attribute)
[25](#attribute-summary-opendap-settings-attribute)

2.10.7 Observation Attribute (OPeNDAP Settings Attribute)
[26](#observation-attribute-opendap-settings-attribute)

2.10.8 Range Edit (OPeNDAP Settings Attribute)
[26](#range-edit-opendap-settings-attribute)

2.10.9 Discrete Edit (OPeNDAP Settings Attribute)
[26](#discrete-edit-opendap-settings-attribute)

2.10.10 Exclude Edit (OPeNDAP Settings Attribute)
[26](#exclude-edit-opendap-settings-attribute)

2.10.11 Data Source URL (OPeNDAP Settings Attribute)
[26](#data-source-url-opendap-settings-attribute)

2.11 Guide Operation of (#1) (Association)
[26](#guide-operation-of-1-association)

2.11.1 Support Scraper Access (Scraper Settings Role)
[27](#support-scraper-access-scraper-settings-role)

2.11.2 Consult Settings (Web Page Scraper Role)
[27](#consult-settings-web-page-scraper-role)

2.12 Guide Operation of (#2) (Association)
[27](#guide-operation-of-2-association)

2.12.1 Support OPeNDAP Access (OPeNDAP Settings Role)
[27](#support-opendap-access-opendap-settings-role)

2.12.2 (OPeNDAP Interface Role) [27](#opendap-interface-role)

2.13 Base Template (Class) [27](#base-template-class)

2.13.1 Applicable Agent (Base Template Attribute)
[27](#applicable-agent-base-template-attribute)

2.13.2 Pattern Definition (Base Template Attribute)
[27](#pattern-definition-base-template-attribute)

2.14 OPeNDAP Base Template (Class) [28](#opendap-base-template-class)

2.14.1 Host Agent (OPeNDAP Base Template Attribute)
[28](#host-agent-opendap-base-template-attribute)

2.14.2 File Pattern (OPeNDAP Base Template Attribute)
[28](#file-pattern-opendap-base-template-attribute)

2.15 Enables Web Page Parsing by (Association)
[28](#enables-web-page-parsing-by-association)

2.15.1 Reveal Pattern (Base Template Role)
[28](#reveal-pattern-base-template-role)

2.15.2 Read Pattern (Web Page Scraper Role)
[28](#read-pattern-web-page-scraper-role)

2.16 Enables File Parsing by (Association)
[28](#enables-file-parsing-by-association)

2.16.1 Consult Pattern (OPeNDAP Interface Role)
[28](#consult-pattern-opendap-interface-role)

2.16.2 Provide Pattern (OPeNDAP Base Template Role)
[29](#provide-pattern-opendap-base-template-role)

2.17 Coordinates ETL Activities of (Association)
[29](#coordinates-etl-activities-of-association)

2.17.1 Control Scraper (Transformation Manager Role)
[29](#control-scraper-transformation-manager-role)

2.17.2 Parse Web Page (Web Page Scraper Role)
[29](#parse-web-page-web-page-scraper-role)

2.18 Coordinates ETL OPeNDAP Activities of (Association)
[29](#coordinates-etl-opendap-activities-of-association)

2.18.1 Parse Data File (OPeNDAP Interface Role)
[29](#parse-data-file-opendap-interface-role)

2.18.2 Control OPeNDAP Interface (Transformation Manager Role)
[29](#control-opendap-interface-transformation-manager-role)

2.19 Web Page Scraper (Class) [30](#web-page-scraper-class)

2.19.1 Target Station List (Web Page Scraper Attribute)
[30](#target-station-list-web-page-scraper-attribute)

2.19.2 Station Connector (Web Page Scraper Attribute)
[30](#station-connector-web-page-scraper-attribute)

2.19.3 Web Page Acquisition (Web Page Scraper Attribute)
[30](#web-page-acquisition-web-page-scraper-attribute)

2.19.4 Template Loader (Web Page Scraper Attribute)
[30](#template-loader-web-page-scraper-attribute)

2.19.5 Web Page Parser (Web Page Scraper Attribute)
[30](#web-page-parser-web-page-scraper-attribute)

2.19.6 Observation Extractor (Web Page Scraper Attribute)
[31](#observation-extractor-web-page-scraper-attribute)

2.19.7 Settings Acquisition (Web Page Scraper Attribute)
[31](#settings-acquisition-web-page-scraper-attribute)

2.19.8 Exceptions Handler (Web Page Scraper Attribute)
[31](#exceptions-handler-web-page-scraper-attribute)

2.19.9 Data Checker (Web Page Scraper Attribute)
[31](#data-checker-web-page-scraper-attribute)

2.20 OPeNDAP Interface (Class) [31](#opendap-interface-class)

2.20.1 Target Data Source (OPeNDAP Interface Attribute)
[32](#target-data-source-opendap-interface-attribute)

2.20.2 Data Source Connector (OPeNDAP Interface Attribute)
[32](#data-source-connector-opendap-interface-attribute)

2.20.3 OPeNDAP Query Builder (OPeNDAP Interface Attribute)
[32](#opendap-query-builder-opendap-interface-attribute)

2.20.4 Template Loader (OPeNDAP Interface Attribute)
[32](#template-loader-opendap-interface-attribute)

2.20.5 Data File Parser (OPeNDAP Interface Attribute)
[32](#data-file-parser-opendap-interface-attribute)

2.20.6 Data Extractor (OPeNDAP Interface Attribute)
[32](#data-extractor-opendap-interface-attribute)

2.20.7 Settings Acquisition (OPeNDAP Interface Attribute)
[32](#settings-acquisition-opendap-interface-attribute)

2.20.8 Exceptions Handler (OPeNDAP Interface Attribute)
[33](#exceptions-handler-opendap-interface-attribute)

2.20.9 Data Checker (OPeNDAP Interface Attribute)
[33](#data-checker-opendap-interface-attribute)

2.20.10 File Handler (OPeNDAP Interface Attribute)
[33](#file-handler-opendap-interface-attribute)

2.20.11 Query Sender (OPeNDAP Interface Attribute)
[33](#query-sender-opendap-interface-attribute)

2.21 Is Created by (#1) (Association) [34](#is-created-by-1-association)

2.21.1 Extract Observation (Web Page Scraper Role)
[34](#extract-observation-web-page-scraper-role)

2.21.2 Contain Observation Data (Extracted Observation Role)
[34](#contain-observation-data-extracted-observation-role)

2.22 Is Created by (#2) (Association) [34](#is-created-by-2-association)

2.22.1 Read Observation Data (OPeNDAP Interface Role)
[34](#read-observation-data-opendap-interface-role)

2.22.2 Contain Observation Data (Extracted Observation Role)
[34](#contain-observation-data-extracted-observation-role-1)

2.23 Extracted Observation (Class) [34](#extracted-observation-class)

2.23.1 Type (Extracted Observation Attribute)
[35](#type-extracted-observation-attribute)

2.23.2 Short Name (Extracted Observation Attribute)
[35](#short-name-extracted-observation-attribute)

2.23.3 Value (Extracted Observation Attribute)
[35](#value-extracted-observation-attribute)

2.23.4 Date (Extracted Observation Attribute)
[35](#date-extracted-observation-attribute)

2.23.5 Time (Extracted Observation Attribute)
[36](#time-extracted-observation-attribute)

2.24 Provides Extracted Observation to (#1) (Association)
[36](#provides-extracted-observation-to-1-association)

2.24.1 Prepare Extracted Observation (Web Page Scraper Role)
[36](#prepare-extracted-observation-web-page-scraper-role)

2.24.2 Acquire Extracted Observation (Data Storage Interface Role)
[36](#acquire-extracted-observation-data-storage-interface-role)

2.25 Provides Extracted Observation to (#2) (Association)
[37](#provides-extracted-observation-to-2-association)

2.25.1 Prepare Extracted Observation (OPeNDAP Interface Role)
[37](#prepare-extracted-observation-opendap-interface-role)

2.25.2 Acquire Extracted Observation (Data Storage Interface Role)
[37](#acquire-extracted-observation-data-storage-interface-role-1)

2.26 Relational Database Manager (Class)
[37](#relational-database-manager-class)

2.26.1 Scraper Service (Relational Database Manager Attribute)
[37](#scraper-service-relational-database-manager-attribute)

2.26.2 Loader Interface (Relational Database Manager Attribute)
[37](#loader-interface-relational-database-manager-attribute)

2.26.3 OPeNDAP Interface Service (Relational Database Manager Attribute)
[37](#opendap-interface-service-relational-database-manager-attribute)

2.27 Controls Operation of (Association)
[38](#controls-operation-of-association)

2.27.1 Manage Observations Data Flows (Relational Database Manager Role)
[38](#manage-observations-data-flows-relational-database-manager-role)

2.27.2 Store Observations Data (Working Database Role)
[38](#store-observations-data-working-database-role)

2.28 Working Database (Class) [38](#working-database-class)

2.28.1 Observations Data (Working Database Attribute)
[38](#observations-data-working-database-attribute)

2.28.2 Scraper Data (Working Database Attribute)
[38](#scraper-data-working-database-attribute)

2.28.3 Business Schema (Working Database Attribute)
[39](#business-schema-working-database-attribute)

2.28.4 OPeNDAP Data (Working Database Attribute)
[39](#opendap-data-working-database-attribute)

2.29 Loader (Class) [39](#loader-class)

2.29.1 Query Generator (Loader Attribute)
[39](#query-generator-loader-attribute)

2.29.2 Geographic Positioner (Loader Attribute)
[39](#geographic-positioner-loader-attribute)

2.29.3 GIS Output (Loader Attribute) [39](#gis-output-loader-attribute)

2.30 Coordinates Activation of (Association)
[39](#coordinates-activation-of-association)

2.30.1 Control Loading (Transformation Manager Role)
[40](#control-loading-transformation-manager-role)

2.30.2 Prepare Geospatial Data (Loader Role)
[40](#prepare-geospatial-data-loader-role)

2.31 Obtains New Observations Data From (Association)
[40](#obtains-new-observations-data-from-association)

2.31.1 Issue Retrieval Requests (Loader Role)
[40](#issue-retrieval-requests-loader-role)

2.31.2 Retrieve Observations Data (Relational Database Manager Role)
[40](#retrieve-observations-data-relational-database-manager-role)

2.32 Geospatial Interface (Class) [41](#geospatial-interface-class)

3 Geospatial Data Repository Level 2 Analysis View
[42](#geospatial-data-repository-level-2-analysis-view)

3.1 Geospatial Database Manager (Class)
[43](#geospatial-database-manager-class)

3.1.1 ETL Program Interface (Geospatial Database Manager Attribute)
[44](#etl-program-interface-geospatial-database-manager-attribute)

3.1.2 Client Interface (Geospatial Database Manager Attribute)
[44](#client-interface-geospatial-database-manager-attribute)

3.2 Geospatial Interface (Class) [44](#geospatial-interface-class-1)

3.3 Controls Operations in (Association)
[44](#controls-operations-in-association)

3.3.1 Manage Geospatial Data Flows (Geospatial Database Manager Role)
[44](#manage-geospatial-data-flows-geospatial-database-manager-role)

3.3.2 Store Geospatial Data (Geospatial Database Role)
[44](#store-geospatial-data-geospatial-database-role)

3.4 Geospatial Database (Class) [44](#geospatial-database-class)

3.4.1 Geospatial Data (Geospatial Database Attribute)
[45](#geospatial-data-geospatial-database-attribute)

3.4.2 Geospatial Schema (Geospatial Database Attribute)
[45](#geospatial-schema-geospatial-database-attribute)

3.5 Web Feature Service (Class) [45](#web-feature-service-class)

List of Figures

Figure 1‑1. ICOOS IIS Level 2 Use Cases for \"Stage Data.\"
[2](#__RefHeading___Toc137958522)

Figure 2‑1. ICOOS IIS ETL Program Level 2 Classes.
[14](#__RefHeading___Toc137958523)

Figure 3‑1. ICOOS IIS Geospatial Data Repository Level 2 Classes.
[41](#__RefHeading___Toc137958524)

Abstract

The ICOOS Integrated Information System demonstration process described
herein encompasses services that integrate mixed-content source data
into a current standardized-content geospatial collection and make the
collection available to coastal inundation models and tools employed by
user personnel on remote terminals.

This document presents the level 2 use cases and analysis view of the
business policies and processes employed by the Extract, Transfer, and
Load (ETL) Program in the ICOOS IIS Data Mart.

# Scope

This document presents the level 2 object oriented use cases and
analysis view of the ICOOS Data Mart for review and comment. This is the
policies and processes (requirements) view of the ICOOS Data Mart
product.

This document examines only the ICOOS IIS Data Mart at Level 2
magnification. The companion document, *Integrated Coastal and Ocean
Observing Systems (ICOOS) Integrated Information System (IIS) UML
Description: Level 1 Use Cases and Analysis View,* looks at the overall
ICOOS and shows the Data Mart\'s occupancy within it.

## Identification

The formal title of the system-level entity is \"Integrated Information
System for the Integrated Coastal and Ocean Observing Systems.\" The
title is abbreviated to \"ICOOS IIS x.y\" where \"x\" represents the
version number and \"y\" represents the release number.

The ICOOS IIS is also known as \"ICOOS IIS Data Mart,\" \"ICOOS Data
Mart,\" and \"Data Mart.\"

## ICOOS IIS Data Mart Use Cases

The Level 2 view of the Stage Data use case is shown in Figure 1-1
below. The figure shows associations between the Level 1 use cases
Acquire Data, Publish Data, and Manage Storage and the Level 2 use cases
that make up Stage Data.

The Stage Data use case consists of the following Level 2 use cases:

> 1\. Manage Web Page Data Transfer
>
> 2\. Integrate Mixed-Content Data
>
> 3\. Record Standardized Data
>
> 4\. Manage OPeNDAP Data Transfer

![](media/image1.wmf)

[]{#__RefHeading___Toc137958522 .anchor}Figure 1‑1. ICOOS IIS Level 2
Use Cases for \"Stage Data.\"

The Level 2 use cases interact with each other through the following
associations:

> 1\. Conduct Web Page Data Transfer
>
> 2\. Supply Mixed-Content Web Page Data
>
> 3\. Supply Standardized Data
>
> 4\. Conduct OPeNDAP Data Transfer
>
> 5\. Supply Mixed-Content OPeNDAP Data

The Level 2 use cases and their associations represented in the diagram
are discussed in the paragraphs below.

### Manage Web Page Data Transfer (Use Case)

SUMMARY.

The functionality of this use case periodically connects with Data
Warehouses that communicate data via web pages.

Incoming web page code is parsed for its data content. Observation data
are copied out and are made available for standardization.

USE CASE #1.

Manage Web Page Data Transfer.

ACTORS.

Acquire Data (Level 1), Integrate Mixed-Content Data.

DESCRIPTION.

Check the predefined set of data providers. For each data provider,
establish a communication link to be used at appropriate times
consistent with the provider\'s update cycle. Bring the incoming web
page code into a work area. Terminate the link when the data transfer is
complete. Find mixed-content observation data within the web page code.
Copy the mixed-content data and apply appropriate identification tags.
Provide the mixed-content data to the Integrate Mixed-Content Data
functionality.

PRECONDITIONS.

1\. Identities of the data providers are known.

2\. Communication channels between the Data Mart and the data providers
are not established.

3\. The physical means exists for establishing the communication
channels.

4\. The procedures for establishing the communication channels are
known.

5\. The Data Mart can acquire data using web page scraping processes.

6\. The Manage Web Page Data Transfer function is idle until notified
that the data channel is to be opened.

7\. The observation data are embedded within the web page code.

8\. Data embedded in the web page code are in mixed-content format.

POSTCONDITIONS.

1\. Mixed-content observation data are separated from the web page code.

2\. The mixed-content data are accessible for standardization.

3\. The Manage Web Page Data Transfer functionality is ready to handle
web page code from a new provider.

4\. The connection to the data provider is closed when the data transfer
is complete.

PRIORITY.

Manage Web Page Data Transfer is a Priority 1 function crucial to the
operation of the first instantiation of the Data Mart prototype.

NORMAL COURSE.

Be idle while watching for indication that a web page data transfer is
imminent. Determine the identity of the data provider to be contacted.
When the proper time arrives, establish the communication channel with
the data provider. Capture the incoming web page code. Close the
connection with a data provider when the data transfer process is
finished. Find observation data embedded in the web page code. Copy out
the observation data. Make the observation data available for downstream
processes. Clear internal processes. Return to idle.

ALTERNATIVE COURSES.

1\. When data are missing from the web page code, flag the missing data.

2\. If the web page code format has changed since the last access, flag
the missing data.

3\. When the data provider has changed in an unrecognizable way,
terminate the connection process with this provider and initiate the
connection process for the next provider at the appropriate time.

4\. When the data provider is not available, terminate the connection
process with this provider and initiate the connection process for the
next provider at the appropriate time.

SPECIAL REQUIREMENTS.

1\. No format or data requirements are to be propagated back to the web
page provider.

2\. Associate extracted data with appropriate identification
information.

3\. Provide indication when the processing of a web page processing is
completed.

ASSUMPTIONS.

1\. Web page formats change infrequently.

2\. Error checking occurs in the communication services levels.

3\. The identifications of data providers are known.

4\. Each data provider is uniquely described for identification,
address, interface protocol, contact times, and any other parameter
relevant to establishing the communication channel within the Data Mart
(searching is not required).

5\. The population of data providers is small enough to avoid impacting
time-based access.

### Conduct Web Page Transfer (Association)

The functionality resident in Manage Web Page Data Transfer carries out
data transfers with data providers that communicate using web pages.

##### Provide Web Page (Acquire Data Role)

Communication functions in Acquire Data assemble the web page code and
send it to the ICOOS Data Mart when the data channel is opened.

##### Capture Web Page (Manage Web Page Data Transfer Role)

The Manage Web Page Data Transfer function stores the incoming web page
code where it can be examined.

### Supply Mixed-Content Web Page Data (Association)

Mixed-content data extracted from the web page code are made available
for standardization in units of measure and numerical formats.

##### Extract Data From Web Page (Manage Web Page Data Transfer Role)

Mixed-content data are copied out of the web page code and associated
with identification information in preparation for standardization.

##### Standardize Extracted Web Page Data (Integrate Mixed-Content Data Role)

The mixed-content web page data are made consistent with units and
formats used by the Data Mart.

### Integrate Mixed-Content Data (Use Case)

SUMMARY.

This functionality converts mixed-content data into the standardized
units and formats used in the data repository. The standardized data are
made available for storage in the data repository.

USE CASE #2.

Integrate Mixed-Content Data.

ACTORS.

Manage Web Page Data Transfer, Manage OPeNDAP Data Transfer, Record
Standardized Data.

DESCRIPTION.

Watch for a signal from the Manage Web Page Data Transfer functionality
or the Manage OPeNDAP Data Transfer functionality indicating that
mixed-content data are available for standardization processing. Operate
on the mixed-content data to make them consistent in units and formats
with Data Mart standards policies. Indicate that the data are ready for
storage when the standardization process is complete.

PRECONDITIONS.

1\. Mixed-content observation data are ready for standardization
processing.

2\. ICOOS Data Mart standards policies are known and implemented in the
integration functions.

3\. The mixed-content observation data are associated with
identifications.

POSTCONDITIONS.

1\. The observation data are consistent with ICOOS Data Mart standards
policies.

2\. The standardized observation data are ready for storage.

PRIORITY.

Integrate Mixed-Content Data is a Priority 1 function crucial to the
operation of the first instantiation of the ICOOS Data Mart prototype.
It may require augmentation when new interfacing capabilities (such as
OPeNDAP) are added.

NORMAL COURSE.

Watch for indication that mixed-content data are available for
standardization processing. Convert the mixed-content observation data
to standardized-content in accordance with ICOOS Data Mart standards
policies. When standardization is complete, make the data available for
storage.

ALTERNATIVE COURSES.

1\. When the repository is absent, overwrite the previous standardized
data.

ASSUMPTIONS.

None identified.

### Supply Standardized Data (Association)

Incoming realtime observation data that conform to the Data Mart\'s
standards policies are transferred to the data repository.

##### Provide Standardized Data (Supply Standardized Data Role)

Observation data consistent with Data Mart standards policies are made
available for storage upon completion of the standardization process.

##### Store Standardized Data (Record Standardized Data Role)

Observation data consistent with Data Mart standards policies are made
available for storage upon completion of the standardization process.

### Conduct OPeNDAP Data Transfer (Association)

The functionality resident in Manage OPeNDAP Data Transfer carries out
data transfers with OPeNDAP-capable data providers.

##### Provide Data Per OPeNDAP (Acquire Data Role)

When OPeNDAP protocol is in effect, data providers respond to data
requests from the Data Mart by retrieving the requested data, assembling
network-compatible files, and transmitting the data files to the ICOOS
Data Mart.

##### Capture OPeNDAP Data Files (Manage OPeNDAP Data Transfer Role)

Functionality in Manage OPeNDAP Data Transfer issues OPeNDAP-compliant
data queries and receives the resulting files sent by the data provider.

### Manage OPeNDAP Data Transfer (Use Case)

SUMMARY.

This functionality controls the OPeNDAP capability of the ICOOS Data
Mart. The functionality periodically connects with OPeNDAP-compliant
data providers and issues requests. Data files are received and their
observations are extracted. The observations are made available for
standardization. The observations can be actual measurements or
predictions.

USE CASE #4.

Manage OPeNDAP Data Transfer.

ACTORS.

Acquire Data (Level 1), Integrate Mixed-Content Data.

DESCRIPTION.

Check the predefined set of data providers. For each data provider,
establish a communication link to be used at appropriate times
consistent with the providers update cycle. Prepare the
OPeNDAP-compliant data request. Send the OPeNDAP-compliant data request
to the data provider. Receive requested data files. Bring the incoming
files into a work area. Terminate the link when the data transfer is
complete. Find mixed-content observation data within the data files.
Copy out the mixed-content data and apply appropriate identification
tags. Provide the mixed-content data to the Integrate Mixed-Content Data
functionality.

PRECONDITIONS.

1\. The Manage OPeNDAP Data Transfer functionality is idle until
notified that the data channel is to be opened to an OPeNDAP-capable
data provider.

2\. For gridded data, the time window and rectangular geographic
boundaries are known.

3\. For discrete location data, the reporting station and time window
are known.

4\. Specific desired measurement types are known.

5\. Files from the provider contain measurements in mixed-content
formats.

6\. Identities of the data providers are known.

7\. Communication channels between the Data Mart and the data providers
are not established.

8\. The physical means exists for establishing the communication
channels.

9\. The procedures for establishing the communication channels are
known.

10\. The Data Mart can acquire data using OPeNDAP processes.

11\. The observation data are contained in downloaded data files.

12\. Data embedded in the web page code are in mixed-content format.

POSTCONDITIONS.

1\. The mixed-content observation data are identified.

2\. The mixed-content data are accessible for standardization.

3\. The Manage OPeNDAP Data Transfer functionality is ready to handle
OPeNDAP interfacing with a new provider.

4\. The connection to the data provider is closed when the data transfer
is complete.

PRIORITY.

Manage Web Page Data Transfer is a Priority 2 function behind the
Priority 1 Manage Web Page Data Transfer function. Its purpose is to
refine and advance the capabilities of the ICOOS Prototype that are
initially established using the Priority 1 function.

NORMAL COURSE.

Be idle while watching for indication that an OPeNDAP interfacing is
imminent. Determine the identity of the data provider to be contacted.
When the proper time arrives, establish the communication channel with
the data provider. Assemble a data request and sent it to the data
provider. Capture the incoming data file sent by the provider in
response to the request. Find observation data in the file. Copy out the
observation data and associate with identifiers. Make the observation
data available for downstream processes. Clear internal processes.
Return to idle.

ALTERNATIVE COURSES.

1\. When data are missing from the files, flag the missing data.

2\. If the data provider does not respond to OPeNDAP queries, flag the
failure, close the connection, clear internal processes, and return to
idle.

SPECIAL REQUIREMENTS.

1\. No format or data requirements are to be propagated back to the
OPeNDAP data provider.

2\. Associate extracted data with appropriate identification
information.

3\. Provide indication when the processing of OPeNDAP files is
completed.

ASSUMPTIONS.

1\. Error checking occurs in the communication services levels.

2\. The identifications of data providers are known.

3\. Each data provider is uniquely described for identification,
address, interface protocol, contact times, and any other parameter
relevant to establishing the communication channel within the Data Mart
(searching is not required).

4\. The population of data providers is small enough to avoid impacting
time-based access.

### Supply Mixed-Content OPeNDAP Data (Association)

Mixed-content data extracted from the OPeNDAP files are made available
for standardization in units of measure and numerical formats.

##### Extract Data From OPeNDAP Download (Manage OPeNDAP Data Transfer Role)

Mixed-content data are copied out of the OPeNDAP files and associated
with identification information in preparation for standardization.

##### Standardize Extracted OPeNDAP Data (Integrate Mixed-Content Data Role)

The mixed-content OPeNDAP data are made consistent with units and
formats used by the Data Mart.

# ETL Program Level 2 Analysis View

This section focuses on the Level 2 analysis view of the ETL Program
within the ICOOS Data Mart. The analysis view depicts policies and
processes as classes of objects constituting the business model
(requirements) of the ETL Program.

Instantiations of the ETL Program Class periodically interface with data
warehouses to request realtime observation data for stations within the
area of interest (AOI). ETL programs acquire the web pages or OPeNDAP
files provided by the warehouses, read out the observation data , and
put the observations into the Data Mart\'s geospatial database.

Figure 2-2 graphically depicts the Level 2 classes arranged to show how
they relate to one another inside the Level 1 ETL Program Class. The
Level 2 classes of processes shown as component parts of the Level 1 ETL
Program Class are:

> 1\. Timer
>
> 2\. Transformation Manager
>
> 3\. Data Storage Interface
>
> 4\. Relational Data Base Manager
>
> 5\. Working Database
>
> 6\. Loader
>
> 7\. Web Page Scraper
>
> 8\. Base Template
>
> 9\. Scraper Settings
>
> 10\. OPeNDAP Interface
>
> 11\. OPeNDAP Base Template
>
> 12\. OPeNDAP Settings
>
> 13\. Extracted Observation
>
> 14\. Web Page Interface
>
> 15\. OPeNDAP
>
> 16 Geospatial Interface

Items 1 through 6 manage information flows. Items 7, 8, and 9 obtain
observation data from web pages. Items 10, 11, and 12 obtain observation
data from OPeNDAP file downloads. Item 13 describes the principal
article sought by the ETL Program. Items 14 and 15 transport
observations (actual and predicted) to the Data Mart. Item 16 is the
gateway to the user-accessible geospatial database.

The classes are associated with one another through the following
associations:

> 1\. Initiates (an instantiation of the Timer Class starts an
> instantiation of the Transformation Manager Class)
>
> 2\. Coordinates Actions of (an instantiation of the Transformation
> Manager Class coordinates the activities of an instantiation of the
> Data Storage Interface Class)
>
> 3\. Coordinates ETL Activities of (a transformation manager
> coordinates operation of a web page scraper)
>
> 4\. Acquires HTML Web Page From (an instantiation of the Web Page
> Scraper Class obtains a web page from an instantiation of the Data
> Warehouse Class)
>
> 5\. Enables Web Page Parsing by (a base template tells a web page
> scraper where the observations are located on a web page)
>
> 6\. Provides Extracted Observation to (a web page scraper makes its
> output available to a data storage interface)
>
> 7\. Are Fetched by (#1) (scraper settings are retrieved from a Data
> Mart\'s relational database by a data storage interface)
>
> 8\. Guide Operation of (#1) (scraper settings define how a web page
> scraper works with a particular data warehouse)
>
> 9\. Is Created by (#1) (some extracted observations are created by a
> web scraper during web page parsing)
>
> 10\. Exchanges Observation Data with (a data storage interface and a
> relational database manager conduct a two-way data flow between the
> ETL program and relational database)
>
> 11\. Controls Operations of (the relational data base manager
> supervises the flow of data into and out of the working database)
>
> 12\. Coordinates Activation of (the transformation manager starts the
> loader at the appropriate step of the sequence)
>
> 13\. Obtains New Observations Data From (the loader queries the
> relational data base manager to get the latest realtime observations)
>
> 14\. Coordinates ETL OPeNDAP Activities of (a transformation manager
> coordinates the activities of the OPeNDAP Interface process)
>
> 15\. Enables File Parsing by (an OPeNDAP Base Template tells an
> OPeNDAP Interface where the observations are located in a downloaded
> OPeNDAP data file)
>
> 16\. Provides Extracted Observation to (#2) (an OPeNDAP Interface
> makes its output available to a data storage interface)
>
> 17\. Is Created by (#2) (some extracted observations are created by
> the OPeNDAP Interface)
>
> 18\. Guide Operation of (#2) (OPeNDAP Settings define how an OPeNDAP
> Interface works with a particular data warehouse).

![](media/image2.wmf)

[]{#__RefHeading___Toc137958523 .anchor}Figure 2‑2. ICOOS IIS ETL
Program Level 2 Classes.

## Timer (Class)

Members of the Timer Class coordinate Data Mart observations acquisition
activities with warehouse reporting cycles.

Warehouses collect observations from their stations at regular time
intervals, then report them through their station web pages or OPeNDAP
portals. Instantiations of the Timer Class are synchronized with the
warehouse cycles.

### Period (Timer Attribute)

The Period is how often the timer emits a signal to activate the Data
Mart\'s warehouse connection processes. A period is determined from how
often a warehouse\'s data store is populated with new realtime
observations.

### Start Signal (Timer Attribute)

The Start Signal tells members of the Transformation Manager Class to
begin a new cycle of observations acquisition.

## Initiates (Association)

The timer signals the transformation manager when it is time to start
the ETL program processes.

### Issue Start (Timer Role)

The timer issues the start signal when the time has arrived to begin ETL
processing.

### Start ETL Program (Transformation Manager Role)

The transformation manager starts the ETL program processing upon
receipt of the start signal from the timer.

## Transformation Manager (Class)

An instantiation of the Transformation Manager Class coordinates the
internal operation of an ETL program. It supervises the extraction,
transfer, and loading of observations from a web page or downloaded data
file to the Data Mart relational database.

The transformation manager starts the data fetch operation upon receipt
of a signal from the timer. The manager determines which interface
functionality to activate (web page or OPeNDAP). The manager activates
processes in the Data Storage Interface Class to retrieve operational
settings from the Data Mart\'s relational database for loading into the
web page scraper or the OPeNDAP Interface. It then starts scraper or
OPeNDAP processes that load the base template, connect to a warehouse\'s
station\'s URL, acquire the web page or data file, parse the web page or
data file, create extracted observations, check the observation data,
and manage responses to exceptions (such as a failed URL, a nonreporting
station, missing observation data, and quality issues).

### Executive (Transformation Manager Attribute)

The Executive watches for the timer signal, then activates the
performance monitor and sequencers.

### Process Monitor (Transformation Manager Attribute)

The Process Monitor checks for initializations and completions of
processes in the ETL Program. The monitor issues an alert in response to
an unexpected event in the ETL Program.

An example of an unexpected event is the noncompletion of a settings
retrieval from the relational database.

### Scraper Sequencer (Transformation Manager Attribute)

The Scraper Sequencer activates scraper processes in logical order.

### Retrieval Sequencer (Transformation Manager Attribute)

The Retrieval Sequencer coordinates interface functions that retrieve
scraper or OPeNDAP settings from the relational database.

### Storage Sequencer (Transformation Manger Attribute)

The Storage Sequencer coordinates interface functions for storing
extracted observations and associated information in the relational
database.

### OPeNDAP Interface Sequencer (Transformation Manger Attribute)

The OPeNDAP Interface Sequencer activates interfacing processes in
logical order.

## Coordinates Actions of (Association)

The transformation manager controls and monitors functions in the data
storage interface to coordinate data flow activities with other ETL
program actions.

### Control Data Flow (Transformation Manager Role)

The transformation manager issues commands to the data storage interface
to perform data flow operations consistent with the processing state of
the scraper and OPeNDAP interface. Retrievals are performed to acquire
settings, and database insertions are performed after the observations
are extracted.

### Provide Interface (Data Storage Interface Role)

The data storage interface functions carry out data flow operations
between the ETL Program and the Relational Database Manager in response
to commands from the transformation manager.

## Data Storage Interface (Class)

The Data Storage Interface contains proxies that put observation data
into, and retrieve settings from, the Data Mart\'s relational database.
The proxies are activated by the transformation manager, web page
scraper, and OPeNDAP interface. There is an interface proxy for each
table in the relational database.

### Station Table Interface (Data Storage Interface Attribute)

The Station Table Interface contains functions to insert, delete,
update, select, and retrieve data stored in the Station Table in the
Data Mart\'s relational database.

### Location History Table Interface (Data Storage Interface Attribute)

The Location History Table Interface contains functions to insert,
delete, update, select, and retrieve data stored in the Location History
Table in the Data Mart\'s relational database.

### Collection Platform Table Interface (Data Storage Interface Attribute)

The Collection Platform Table Interface contains functions to insert,
delete, update, select, and retrieve data stored in the Collection
Platform Table in the Data Mart\'s relational database.

### Operational Status Table Interface (Data Storage Interface Attribute)

The Operational Status Table Interface contains functions to insert,
delete, update, select, and retrieve data stored in the Operational
Status Table in the Data Mart\'s relational database.

### Collection Sensor Table Interface (Data Storage Interface Attribute)

The Collection Sensor Table Interface contains functions to insert,
delete, update, select, and retrieve data stored in the Collection
Sensor Table in the Data Mart\'s relational database.

### Sensor Table Interface (Data Storage Interface Attribute)

The Sensor Table Interface contains functions to insert, delete, update,
select, and retrieve data stored in the Sensor Table in the Data Mart\'s
relational database.

### Measurement Table Interface (Data Storage Interface Attribute)

The Measurement Table Interface contains functions to insert, delete,
update, select, and retrieve data stored in the Measurement Table in the
Data Mart\'s relational database.

### Station Type Table Interface (Data Storage Interface Attribute)

The Station Type Interface contains functions to insert, delete, update,
select, and retrieve data stored in the Station Type Table in the Data
Mart\'s relational database.

### Benchmark Reference Table Interface (Data Storage Interface Attribute)

The Benchmark Reference Table Interface contains functions to insert,
delete, update, select, and retrieve data stored in the Benchmark
Reference Table in the Data Mart\'s relational database.

### Benchmark Table Interface (Data Storage Interface Attribute)

The Benchmark Table Interface contains functions to insert, delete,
update, select, and retrieve data stored in the Benchmark Table in the
Data Mart\'s relational database.

### Extraction Type Table Interface (Data Storage Interface Attribute)

The Extraction Type Table Interface contains functions to insert,
delete, update, select, and retrieve data stored in the Extraction Type
Table in the Data Mart\'s relational database.

### Participating Station Table Interface (Data Storage Interface Attribute)

The Participating Station Table Interface contains functions to insert,
delete, update, select, and retrieve data stored in the Participating
Station Table in the Data Mart\'s relational database.

### Observation Type Table Interface (Data Storage Interface Attribute)

The Observation Type Table Interface contains functions to insert,
delete, update, select, and retrieve data stored in the Observation Type
Table in the Data Mart\'s relational database.

### Observation Structure Table Interface (Data Storage Interface Attribute)

The Observation Structure Table Interface contains functions to insert,
delete, update, select, and retrieve data stored in the Observation
Structure Table in the Data Mart\'s relational database.

### Observation Attribute Table Interface (Data Storage Interface Attribute)

The Observation Attribute Table Interface contains functions to insert,
delete, update, select, and retrieve data stored in the Observation
Attribute Table in the Data Mart\'s relational database.

### Observed Data Table Interface (Data Storage Interface Attribute)

The Observed Data Table Interface contains functions to insert, delete,
update, select, and retrieve data stored in the Observed Data Table in
the Data Mart\'s relational database.

### Range Edit Table Interface (Data Storage Interface Attribute)

The Range Edit Table Interface contains functions to insert, delete,
update, select, and retrieve data stored in the Range Edit Table in the
Data Mart\'s relational database.

### Discrete Edit Table Interface (Data Storage Interface Attribute)

The Discrete Edit Table Interface contains functions to insert, delete,
update, select, and retrieve data stored in the Discrete Edit Table in
the Data Mart\'s relational database.

### Exclude Edit Table Interface (Data Storage Interface Attribute)

The Exclude Edit Table Interface contains functions to insert, delete,
update, select, and retrieve data stored in the Exclude Edit Table in
the Data Mart\'s relational database.

### OPeNDAP and Predictions Tables Interfaces(Data Storage Interface Attribute)

This is a placeholder entry reserving definitions of interfaces for
predictions tables and OPeNDAP settings data.

## Exchanges Observation Data with (Association)

This is a two-way data flow consisting of requests and data.

### Issue Requests and Data (Data Storage Interface Role)

The data storage interface sends observation data and handling
instructions to the relational database manager for storage or retrieval
in the database.

### Store and Retrieve (Relational Data Base Manager Role)

The relational data base manager stores data in accordance with
instructions from the data storage interface. The database manager
retrieves data in response to queries issued by the data storage
interface.

## Are Fetched by (#1) (Association)

Operational settings for the scraper that are stored in the Data Mart\'s
relational database are retrieved by data storage interface functions
and are made available for use by the scraper.

### Retrieve (Data Storage Interface Role)

The data storage interface functions retrieve the scraper\'s operational
settings from the relational database.

### Wait for Use (Scraper Settings Role)

The scraper\'s operational settings are placed where the scraper can
access them.

## Are Fetched by (#2) (Association)

Operational settings for the OPeNDAP Interface that are stored in the
Data Mart\'s relational database are retrieved by data storage interface
functions and are made available for use by the OPeNDAP Interface.

### Retrieve (Data Storage Interface Role)

The data storage interface functions retrieve the OPeNDAP Interface\'s
operational settings from the relational database.

### Wait for Use (OPeNDAP Settings Role)

The OPeNDAP Interface\'s operational settings are placed where the
OPeNDAP Interface can access them.

## Scraper Settings (Class)

The Scraper Settings consist of parameters that \"dial in\" the desired
function mode of scraper processes and provide traceability to storage
locations in the Data Mart. Scraper settings are stored in the Data
Mart\'s relational database. They are retrieved from the database and
installed before the scraper begins working with a station\'s web page
code.

Some identification parameters in the settings are associated with the
extracted observations during transfer from the scraper to storage.
Examples are the participating station and observation type. Others tell
the scraper how to perform certain quality control functions, like range
checking.

### Extraction Type (Scraper Settings Attribute)

The Extraction Type parameters link an observation with its cycle and
processing type in the Data Mart. The scraper uses the type information
to determine how to process the value parsed from the web page.

### Participating Station (Scraper Settings Attribute)

The Participating Station parameter enables the scraper to find the
station\'s URL. It is associated with extracted observations when they
are turned over to the data storage interface.

### Station Summary (Scraper Settings Attribute)

The Station Summary contains factors that relate a station\'s position
and ID information at the time of the observations to the storage
location in the relational database. It connects the observation to most
of the significant descriptors of a station.

This information also accounts for station relocation due to its
physical movement.

### Observation Type (Scraper Settings Attribute)

The Observation Type is a logical link to the observation\'s name and
description in the Data Mart\'s relational database.

### Observation Structure (Scraper Settings Attribute)

Observation Structure consists of Data Mart internal flags that relate
the observation to its description and tell where it is to be stored in
the relational database.

### Attribute Summary (Scraper Settings Attribute)

The Attribute Summary tells the scraper how many observations to look
for.

### Observation Attribute (Scraper Settings Attribute)

The Observation Attribute relates a particular observation to the
specification of its name, type, units, resolution, precision, etc.
within the Data Mart relational database.

### Range Edit (Scraper Settings Attribute)

The Range Edit parameters enable the scraper to perform quality control
range checking for each observation\'s value.

### Discrete Edit (Scraper Settings Attribute)

The Discrete Edit parameters enable the scraper to check for reporting
of allowable indicators in an observation.

Examples of allowable discrete indicators are found in the observation
reporting wind direction. Two allowed discrete indicators are \"S,\" for
south and \"SSW\" for south-southwest.

### Exclude Edit (Scraper Settings Attribute)

The Exclude Edit settings enable the scraper to recognize when an
observation\'s contents indicate that no value exists.

Example:

Sometimes a station\'s web page observation column contains an entry
like \"999.99,\" or \"-99,\" or \"-\" because no measurement value
exists. The absence could be because the station is not equipped with
the appropriate sensor, or that the sensor is malfunctioning.

### Station URL (Scraper Settings Attribute)

The Station URL enables the scraper to connect to the data warehouse and
acquire the web page for the target station.

## OPeNDAP Settings (Class)

The OPeNDAP Settings consist of parameters that control the desired
function mode of the OPeNDAP Interface and provide traceability to
storage locations in the Data Mart. OPeNDAP settings are stored in the
Data Mart\'s relational database. They are retrieved from the database
and installed before the OPeNDAP Interface begins working with the data
provider and the incoming data.

Some identification parameters in the settings are associated with the
extracted data during transfer from the OPeNDAP Interface instantiation
to storage. Examples are the data source and data type (actual
observation or prediction). Others tell the OPeNDAP Interface
instantiation how to perform certain quality control functions, like
range checking.

### Extraction Type (OPeNDAP Settings Attribute)

The Extraction Type parameters link an observation with its cycle and
processing type in the Data Mart. The OPeNDAP Interface uses the type
information to determine how to process the value parsed from the data
file.

### Participating Data Source (OPeNDAP Settings Attribute)

The Participating Data Source parameter enables the OPeNDAP Interface to
find the data source\'s URL. It is associated with extracted observation
and prediction data when they are turned over to the data storage
interface.

### Data Source Summary (OPeNDAP Settings Attribute)

The Data Source Summary contains factors that relate a data source\'s
position and ID information (at the time of the observation or
prediction) to the storage location in the relational database. It
connects the observation or prediction to most of the significant
descriptors of the data source.

This information also accounts for station relocation due to its
physical movement, and differentiates between physical stations and
models.

### Observation Type (OPeNDAP Settings Attribute)

The Observation Type is a logical link to the observation\'s or
prediction\'s name and description in the Data Mart\'s relational
database. An Observation Type can be \"actual\" (for a realworld
measurement) or \"prediction\" (for a model forecast).

### Observation Structure (OPeNDAP Settings Attribute)

Observation Structure consists of Data Mart internal flags that relate
the observation or prediction to its description and tell where it is to
be stored in the relational database.

### Attribute Summary (OPeNDAP Settings Attribute)

The Attribute Summary tells the OPeNDAP Interface how many observations
or predictions to look for in a downloaded data file.

### Observation Attribute (OPeNDAP Settings Attribute)

The Observation Attribute relates a particular observation or prediction
to the specification of its name, type, units, resolution, precision,
etc. within the Data Mart relational database.

### Range Edit (OPeNDAP Settings Attribute)

The Range Edit parameters enable the OPeNDAP Interface to perform
quality control range checking for each observation\'s or prediction\'s
value.

### Discrete Edit (OPeNDAP Settings Attribute)

The Discrete Edit parameters enable the OPeNDAP Interface to check for
reporting of allowable indicators in an observation or prediction.

Examples of allowable discrete indicators are found in the observation
reporting wind direction. Two allowed discrete indicators are \"S,\" for
south and \"SSW\" for south-southwest.

### Exclude Edit (OPeNDAP Settings Attribute)

The Exclude Edit settings enable the OPeNDAP Interface to recognize when
an observation\'s or prediction\'s contents indicate that no value
exists.

Example:

Sometimes a downloaded OPeNDAP file data column contains an entry like
\"999.99,\" or \"-99,\" or \"-\" because no measurement or predicted
value exists. The absence could be because the station is not equipped
with the appropriate sensor, the sensor is malfunctioning, or the model
is configured to skip the prediction.

### Data Source URL (OPeNDAP Settings Attribute)

The Data Source URL enables the OPeNDAP Interface to connect to the data
warehouse, issue data queries, and acquire the data files for the
targeted domain.

## Guide Operation of (#1) (Association)

The settings are preset controls used by the scraper to define its
operation.

### Support Scraper Access (Scraper Settings Role)

The scraper settings are accessed by the scraper.

### Consult Settings (Web Page Scraper Role)

The scraper refers to the settings to define its functions during
operation.

## Guide Operation of (#2) (Association)

The settings are preset controls used by the OPeNDAP Interface to define
its operation. These may include pro forma data requests for specific
observations in particular geographic areas.

### Support OPeNDAP Access (OPeNDAP Settings Role)

The OPeNDAP settings are accessed by the OPeNDAP Interface.

### (OPeNDAP Interface Role)

The OPeNDAP Interface refers to the settings to define its functions
during operation.

## Base Template (Class)

The Base Template describes the layout of a station\'s web page code. It
is used by the web page scraper to find observations embedded in the
code.

Normally, a single base template applies to all stations accessed
through the same data warehouse. The warehouse usually creates all of
its station\'s web pages to the same format.

### Applicable Agent (Base Template Attribute)

The Applicable Agent is the entity (usually a data warehouse)
responsible for implementing the stations\' web page format.

### Pattern Definition (Base Template Attribute)

The Pattern Definition tells the scraper where to look for observation
data embedded in a station\'s web page code.

## OPeNDAP Base Template (Class)

The OPeNDAP Base Template describes the layout of a warehouse\'s
downloaded data file. It is used by the OPeNDAP Interface to find
observation data in the file.

### Host Agent (OPeNDAP Base Template Attribute)

The Host Agent is the entity (usually a data warehouse) hosting the
OPeNDAP website.

### File Pattern (OPeNDAP Base Template Attribute)

The File Pattern tells the OPeNDAP Interface where to look for
observation data in the downloaded data file.

## Enables Web Page Parsing by (Association)

The base template points to places in the web page code format where
observation data can be found.

### Reveal Pattern (Base Template Role)

The base template contains pattern information that can be read by the
scraper to locate observation data in a station\'s web page code.

### Read Pattern (Web Page Scraper Role)

The scraper uses references in the base template to find the places in
the web page code where the observation data are stored.

## Enables File Parsing by (Association)

The OPeNDAP Base Template tells the OPeNDAP Interface where to find
observation data in the downloaded data file.

### Consult Pattern (OPeNDAP Interface Role)

The OPeNDAP Interface uses the base template to find the observation
data.

### Provide Pattern (OPeNDAP Base Template Role)

The base template contains pattern information that can be read by the
OPeNDAP Interface to locate observation data in the downloaded data
file.

## Coordinates ETL Activities of (Association)

The transformation manager controls and monitors functions in the web
page scraper to coordinate scraper activities with other ETL program
actions.

### Control Scraper (Transformation Manager Role)

The transformation manager issues start and stop commands, and monitors
the operating status of scraper functions.

### Parse Web Page (Web Page Scraper Role)

The web page scraper contains functions that acquire a web page and copy
out the observations embedded in the code. The functions are activated
by the transformation manager. The scraper\'s constituent functions
report their modes and statuses to the transformation manager.

## Coordinates ETL OPeNDAP Activities of (Association)

The Transformation Manager controls and monitors functions in the
OPeNDAP Interface to coordinate its activities with other ETL program
actions.

### Parse Data File (OPeNDAP Interface Role)

The OPeNDAP Interface contains functions that capture an incoming
observation data file and copy out the observations data. The functions
are activated by the transformation manager. The OPeNDAP Interface\'s
constituent functions report their modes and statuses to the
transformation manager.

### Control OPeNDAP Interface (Transformation Manager Role)

The transformation manager issues start and stop commands, and monitors
the operating status of the OPeNDAP Interface functions.

## Web Page Scraper (Class)

The purpose of each instantiation of the Web Page Scraper Class is to
copy the realtime observations from a station\'s web page and make them
available for storage in the Data Mart\'s relational database.

A scraper is controlled by a transformation manager that starts the
processing in response to a timing signal. For each station web page,
the manager configures the scraper with operational settings retrieved
from the relational database. The scraper connects to the station\'s
data warehouse to obtain the web page code containing the station\'s
realtime observations. Using the base template describing the web
page\'s pattern, the scraper parses the web page code to create
extracted observations formatted for efficient handling by Data Mart
processes. The extracted observations are handed over to the data
storage interface for placement into the Data Mart relational database.

### Target Station List (Web Page Scraper Attribute)

The Target Station List defines the web page domain dedicated to a
particular instantiation of the Web Page Scraper Class. The scraper
acquires the web page for each station in the list.

### Station Connector (Web Page Scraper Attribute)

The Station Connector uses a station\'s URL to bring the station\'s web
page code into the Data Mart.

### Web Page Acquisition (Web Page Scraper Attribute)

The Web Page Acquisition process captures the station\'s incoming web
page code into a logic work area where it can be interpreted by other
scraper functions.

### Template Loader (Web Page Scraper Attribute)

The Template Loader finds the base template applicable to the target
station and aims the parsing functions at the realtime observation data
embedded in the web page code.

### Web Page Parser (Web Page Scraper Attribute)

The Web Page Parser segregates the observation data from the web page
code.

### Observation Extractor (Web Page Scraper Attribute)

The Observation Extractor places the parsed observation data into a
format compatible with data handling functions in the Data Mart. The
extractor creates instantiations of the Extracted Observation Class.

### Settings Acquisition (Web Page Scraper Attribute)

The Settings Acquisition process positions the settings parameters for
use by scraper functions.

### Exceptions Handler (Web Page Scraper Attribute)

The Exceptions Handler detects missing web pages, nonreporting stations,
absent data, web page format changes, and other lapses of essential
functions. It sets indications that direct the scraper\'s subsequent
operations.

Subsequent operations include moving on to the next station if a web
page is missing, inserting fillers when observations are missing, and
flagging a station when its web page format is inconsistent with the
base template.

### Data Checker (Web Page Scraper Attribute)

The Data Checker performs quality tests on extracted observations to
detect errors, range limit violations, and statistical anomalies.

## OPeNDAP Interface (Class)

Instantiations of the OPeNDAP Interface Class obtain data files from
OPeNDAP-enabled providers and make their contents available for storage
in the Data Mart\'s relational database. The OPeNDAP Interface Class
deals with two types of observation data: actual and predictions.

An OPeNDAP Interface instantiation is controlled by a transformation
manager that starts the processing in response to a timing signal. For
each OPeNDAP-enabled data provider, the manager configures the OPeNDAP
Interface instantiation with operational settings retrieved from the
relational database. The settings enable connection, query, and data
receipt in accordance with the OPeNDAP protocol.

Using the base template describing the data files\' patterns, the
OPeNDAP Interface extracts actual or predicted observation data in a
format that expedites efficient handling by Data Mart processes. The
extracted data are handed over to the data storage interface for
placement into the Data Mart relational database.

### Target Data Source (OPeNDAP Interface Attribute)

The Target Data Source List defines the web page domain dedicated to a
particular instantiation of the OPeNDAP Interface Class. The OPeNDAP
Interface queries each data provider in the list to obtain actual and
predicted observation data files.

### Data Source Connector (OPeNDAP Interface Attribute)

The Data Source Connector uses a web URL to activate the data
provider\'s OPeNDAP functionality.

### OPeNDAP Query Builder (OPeNDAP Interface Attribute)

The OPeNDAP Query Builder assembles the OPeNDAP data request.

### Template Loader (OPeNDAP Interface Attribute)

The Template Loader finds the base template applicable to the target
data provider and aims the parsing functions at the observation data
embedded in the downloaded data files.

### Data File Parser (OPeNDAP Interface Attribute)

The Data File Parser finds the observation data in the downloaded data
files.

### Data Extractor (OPeNDAP Interface Attribute)

The Observation Extractor places the parsed observation data into a
format compatible with data handling functions in the Data Mart. The
extractor creates instantiations of the Extracted Observation Class.

### Settings Acquisition (OPeNDAP Interface Attribute)

The Settings Acquisition process positions the settings parameters for
use by OPeNDAP Interface functions.

### Exceptions Handler (OPeNDAP Interface Attribute)

The Exceptions Handler detects missing data in downloaded data files,
nonreporting data providers, data file format changes, and other lapses
of essential functions. It sets indications that direct the OPeNDAP
Interface\'s subsequent operations.

Subsequent operations include moving on to the data provider if there is
no OPeNDAP response, inserting fillers when observations are missing,
and flagging a data provider when its downloaded data file is
inconsistent with the base template.

### Data Checker (OPeNDAP Interface Attribute)

The Data Checker performs quality tests on extracted observations to
detect errors, range limit violations, and statistical anomalies.

### File Handler (OPeNDAP Interface Attribute)

The File Handler captures incoming OPeNDAP files and places them in
working areas for parsing and data extraction.

### Query Sender (OPeNDAP Interface Attribute)

The Query Sender sends the OPeNDAP data request to the OPeNDAP-enabled
data provider.

## Is Created by (#1) (Association)

The extracted observation is created by the scraper during web page
parsing.

### Extract Observation (Web Page Scraper Role)

The scraper creates the extracted observation upon copying out the data
in the web page code, as specified by the base template and scraper
settings.

### Contain Observation Data (Extracted Observation Role)

The extracted observation contains the name-value pair extracted from
the web page code.

## Is Created by (#2) (Association)

The extracted observation is created by the OPeNDAP Interface during
data file parsing.

### Read Observation Data (OPeNDAP Interface Role)

The OPeNDAP Interface creates the extracted observation upon copying out
the data in the downloaded file.

### Contain Observation Data (Extracted Observation Role)

The extracted observation contains the name-value pair extracted from
the downloaded data file.

## Extracted Observation (Class)

Extracted Observations are created by the web page scraper and the
OPeNDAP Interface. Each extracted observation is formatted to enable
efficient handling by Data Mart processes.

The ICOOS IIS deals with two basic types of observations: \"actual\" and
\"predicted.\" An extracted observation is either an actual measurement
or an output of a predictive model.

Extracted observations are associated with certain settings (like
participating station and observation type) to facilitate handling and
storage. Extracted observations are passed by the scraper or OPeNDAP
Interface to the data storage interface for handoff to the relational
database where they are stored and accessed for 30 days.

An observation may have different names, depending on the provider. When
there are two or more names for the same observation, the ICOOS IIS
normalizes the different names to a common vernacular.

### Type (Extracted Observation Attribute)

The Type is either \"actual\" or \"prediction.\" The Type designator
indicates whether an extracted observation is the result of an actual
measurement of conditions or is the output of a predictive model.

### Short Name (Extracted Observation Attribute)

The Short Name is the name of the observation. When there are two or
more names for the same observation, the ICOOS IIS normalizes the
different names to a common vernacular.

### Value (Extracted Observation Attribute)

The Value is the quantitative measurement reported by an observation.

For an \"actual\" observation, it is the reading of a measurement
reported by a sensor and recorded by an instantiation of the Data
Collection Package Class (see the Level 1 diagram in the Logical View
folder) when assembling observations from measurements and timing marks.

For a \"prediction,\" it is the calculated quantity output by a
forecasting model.

### Date (Extracted Observation Attribute)

The Date is the year, month, and day of the observation.

For an \"actual\" observation, it is originally recorded by an
instantiation of the Data Collection Package Class when assembling
observations from measurements and timing marks.

For a \"prediction,\" it signifies when in the future the observation
will assume the expected quantity.

### Time (Extracted Observation Attribute)

The Time tells when the observation is made. It consists of hours,
minutes, seconds, and time zone.

For an \"actual\" observation, time is originally recorded by an
instantiation of the Data Collection Package Class when assembling
observations from measurements and timing marks.

For a \"prediction,\" it signifies when in the future the observation
will assume the expected quantity.

## Provides Extracted Observation to (#1) (Association)

The scraper is a source of observation data that gets stored by the data
storage interface.

### Prepare Extracted Observation (Web Page Scraper Role)

The scraper places the extracted observation in a place and format
accessible to data storage interface functions.

### Acquire Extracted Observation (Data Storage Interface Role)

The data storage interface transfers the extracted observation into a
form that can be handled by the relational database manager.

This can involve copying observation data into the right places in
pre-canned database storage commands.

## Provides Extracted Observation to (#2) (Association)

The OPeNDAP Interface is a source of observation data that gets stored
by the data storage interface.

### Prepare Extracted Observation (OPeNDAP Interface Role)

The OPeNDAP Interface places the extracted observation in a place and
format accessible to data storage interface functions.

### Acquire Extracted Observation (Data Storage Interface Role)

The data storage interface transfers the extracted observation into a
form that can be handled by the relational database manager.

This can involve copying observation data into the right places in
pre-canned database storage commands.

## Relational Database Manager (Class)

The relational data base manager is responsible for the operation of the
ETL internal working database.

### Scraper Service (Relational Database Manager Attribute)

The Scraper Service interfaces with the Data Storage Interface function
for storing observations data and retrieving scraper settings.

### Loader Interface (Relational Database Manager Attribute)

The Loader Interface supports the receiving of retrieval requests from
the loader and providing newly acquired observations data to the loader.

### OPeNDAP Interface Service (Relational Database Manager Attribute)

The OPeNDAP Interface Service interfaces with the Data Storage Interface
function for storing observations data and retrieving OPeNDAP Interface
settings.

## Controls Operation of (Association)

Observation data flow into and out of the ETL internal working database
under control of the relational data base manager.

### Manage Observations Data Flows (Relational Database Manager Role)

The relational database manager operates the data storage and retrieval
activities of the ETL internal working database.

### Store Observations Data (Working Database Role)

The working database stores and provides data as required by the
relational database manager.

## Working Database (Class)

The ETL internal working database temporarily stores observation data
collected by the scraper and OPeNDAP Interface. After the scraper or
OPeNDAP Interface completes its run, the data are retrieved by the
loader for association with location and geospatial format descriptors.
The database also contains settings that control operation of the
scraper and OPeNDAP Interface. The working database resides within the
ETL Program and is not visible to outside users.

### Observations Data (Working Database Attribute)

Observations data extracted from data warehouse web pages or OPeNDAP
files are stored until needed by the loader. At this stage, the
observations are considered to be \"business\" data, not \"geospatial\"
data.

### Scraper Data (Working Database Attribute)

Settings needed to control scraper operations are stored in the ETL
internal working database.

### Business Schema (Working Database Attribute)

The Business Schema describes the internal organization of the working
database into data tables and data formats.

### OPeNDAP Data (Working Database Attribute)

Settings needed to control OPeNDAP Interface operations are stored in
the ETL internal working database.

## Loader (Class)

Loader functions create geospatial data by associating the new incoming
observations stored in the ETL internal working database with position
and format information. The loader then provides the geospatial data to
the Data Mart\'s geospatial database manager for storage in the
geospatial database. The transformation manager starts the loader
functions after the scraper has completed its task.

### Query Generator (Loader Attribute)

The Query Generator issues retrieval requests to the ETL internal
working relational database manager. Its purpose is to bring up the
newly acquired observations data for use by the Geographic Positioner.

### Geographic Positioner (Loader Attribute)

The Geographic Positioner converts the newly acquired observations data
into geospatial data by associating the observations with geographic
position and GIS format parameters.

### GIS Output (Loader Attribute)

The GIS Output is the interface between the ETL Program and the Data
Mart\'s geospatial database. GIS Output provides geospatially formatted
realtime data to the geospatial database manager with storage
instructions.

## Coordinates Activation of (Association)

The transformation manager controls and monitors functions that transfer
observation data from the working database to the geospatial database.

### Control Loading (Transformation Manager Role)

Loader functions are run after new observation data are placed into the
ETL internal working database.

### Prepare Geospatial Data (Loader Role)

Loader functions associate data residing in the ETL internal working
database with location and formatting information. The resulting
associated data are loaded into the geospatial database for access by
clients.

## Obtains New Observations Data From (Association)

The loader functions obtain the latest observation data from the ETL
internal working database.

### Issue Retrieval Requests (Loader Role)

Loader functions query the ETL internal working database to obtain the
latest observation data.

### Retrieve Observations Data (Relational Database Manager Role)

Functions in the ETL internal working database provide the latest
observation data in response to queries from the loader functions.

## Geospatial Interface (Class)

The Geospatial Interface enables instantiations of the ETL Program Class
to provide geospatial data to the Data Mart\'s Geospatial Data
Repository for storage and to support data requests from users.

Storage instructions and incoming geospatial data are passed to the
geospatial database manager by the loader functions via the Geospatial
Interface.

# Geospatial Data Repository Level 2 Analysis View

This section focuses on the Level 2 analysis view of the Geospatial Data
Repository within the ICOOS Data Mart. The analysis view depicts
policies and processes as classes of objects constituting the business
model (requirements) of the Geospatial Data Repository.

The Level 2 classes shown in Figure 3-3 are

> 1\. Geospatial Database Manager
>
> 2\. Geospatial Database
>
> 3\. Geospatial Interface
>
> 4\. Web Feature Service

The classes are associated with one another through the following
association:

> 1\. Controls Operations in (an instantiation of the Geospatial
> Database Manager manages the data inflows and outflows of an
> instantiation of the Geospatial Database)

![](media/image3.wmf)

[]{#__RefHeading___Toc137958524 .anchor}Figure 3‑3. ICOOS IIS Geospatial
Data Repository Level 2 Classes.

## Geospatial Database Manager (Class)

The geospatial database manager is responsible for the operation of the
Data Mart\'s geospatial database

### ETL Program Interface (Geospatial Database Manager Attribute)

The ETL Program interface supports the loader by receiving geospatial
data and responding to storage instructions.

### Client Interface (Geospatial Database Manager Attribute)

The Client Interface receives data requests from clients and provides
the requested data for transmittal to clients.

## Geospatial Interface (Class)

This is a shared interface discussed in Paragraph 2.32.

## Controls Operations in (Association)

Geospatial database manager functions supervise operations within the
Data Mart\'s geospatial database.

### Manage Geospatial Data Flows (Geospatial Database Manager Role)

Functions of the geospatial database manager oversee data flows of into
and out of the geospatial database.

### Store Geospatial Data (Geospatial Database Role)

The geospatial database stores and provides geospatial data as required
by the geospatial database manager.

## Geospatial Database (Class)

The geospatial database stores all observation data collected by the ETL
Program. This database is visible to ICOOS users through the geospatial
database manager.

### Geospatial Data (Geospatial Database Attribute)

The Geospatial Data consist of \"realtime\" observations and predicted
observations, and are stored for up to 30 days.

### Geospatial Schema (Geospatial Database Attribute)

The Geospatial Schema describes the internal organization of the
geospatial database into tables and data formats.

## Web Feature Service (Class)

Geospatial data gathered by the ICOOS Data Mart are provided to Data
Users via the Web Feature Service (WFS) interface.

Glossary

  -----------------------------------------------------------------------
  **Acronym or            **Definition**
  Nomenclature**          
  ----------------------- -----------------------------------------------
  AOI                     Area of Interest

  ETL                     Extract, Transfer, and Load

  ICOOS                   Integrated Coastal and Ocean Observations
                          System

  IIS                     Integrated Information System

  NCO                     Network Centric Operations

  NCO/P&T                 Network Centric Operations Programs and
                          Technologies

  NOAA                    National Oceanic and Atmospheric Administration

  NTDE                    National Tidal Data Epoch

  UML                     Unified Modeling Language

  URL                     Uniform Resource Locator

  WA                      Washington (state)

  WFS                     Web Feature Service
  -----------------------------------------------------------------------