# Presentation: 642433

## Global Earth ObservationIntegrated Data Environment (GEO-IDE) - A Short Course -

**Presentation to **

***Presenter name and affiliation***

## NOAA’ Top Ten Challenges1

- 10) Alphabet Soup
- 9) Stove Pipes
- 8) Integration
- 7) Architecture
- 6) Data Sharing

- 5) User Needs
- 4) Maximizing Benefits
- 3) Communication
- 2) Data Management
- 1) Execution

- Vice Admiral Lautenbacher in address to American Meteorological Society Annual Meeting, January 30, 2006

## Data Management - a top priority

- Improving data management is among the highest priority challenges facing NOAA... integrated data management is at the heart of the GEOSS concept

**Notes:**
During he presentation at the AMS, the Vice Admiral also noted that data management is the key to better integrating data and information to better meet societal needs.  This is why data management is at the very center of GEOSS.

## NOAA’s GEO-IDE

**Scope **– NOAA-wide architecture development to integrate legacy systems and guide development of future NOAA environmental data management systems
**Vision **– NOAA’s GEO-IDE is envisioned as a “system of systems” – a framework that provides effective and efficient integration of NOAA’s many quasi-independent systems
**Foundation** – built upon agreed standards, principles and guidelines
**Approach **– evolution of existing systems into a service-oriented architecture ** **
**Result **– a single system of systems (user perspective) to access the data sets needed to address significant societal questions

## Why is improved integration needed?

***Why is improved integration needed?***
***Important societal issues require data from many observing systems***

| Atmospheric Observations |  |  |
| --- | --- | --- |
| Land Surface Observation |  |  |
| Ocean Observations |  |  |
| Space Observations |  |  |
| Data Systems |  | Coordinated, efficient,integrated, interoperable |

**Discipline Specific View**

**Whole System View**

**Current systems are program specific, focused, individually efficient.**
**But incompatible, not integrated, isolated from one another and from wider environmental community**

**Notes:**
Need an integrated, whole system view to provide the information that society needs.

## Today’s Challenges

**Incompatible syntax (formats) and semantics (terminology) among science disciplines within NOAA.  Thousands exist.  Several examples:  **
**Naming standards – Surface Air Temperature**
**Meteorology (WMO) named “Temperature/dry bulb temperature**
**Meteorology (air pollution) named “Boundary layer temperature”**
**Oceanography named “Air Temperature”**
**Location standards (latitude, longitude, elevation)**
**Lat/Lon can be degrees/minutes/seconds or degrees to tenths and hundredths  **
**Latitude E/W, 0-180 positive and negative, or 0-360 running east or west**
**Z used to designate elevation in both atmosphere and ocean but positive is up in the atmosphere and down in the ocean**
**Formats  (>50 formats used within NOAA; translators and standards needed)**
**GRIB, NetCDF, HDF and others used for gridded data**
**BUFR, NetCDF, and **_**many**_** others used for observations  **

## Goals

**Through GEO-IDE NOAA will**
**Identify and address integration gaps in data management systems **
**Create interoperability across existing data management systems**
**Develop and adopt data standards for formats and terminology**
**Integrate measurements, data, and products **
**Examine the need for future data management requirements**
**and will achieve**
**Cost avoidances in NOAA business through improved efficiency and reduced duplication**
**Reduced risks for US IEOS and GEOSS**

## GEO-IDE - an essential component ofenvironmental information management for NOAA

- Integrated observing, data processing and information management systems
- Connected by NOAA’s Integrated Data Environment
- Contributes to
- U.S. Global Earth Observation System (USGEO) and
- International Global Earth Observing System of Systems (GEOSS).

## Scope

- Concerned with environmental and geospatial data and information obtained or generated from worldwide sources to support NOAA's mission (as defined in NOAA Administrative Order 212-15)
- Does not consider administrative support systems such as finance, personnel, acquisition or facilities management
- Includes all aspects of data management, including data acquisition, ingest, data processing, archival and access

## Vision

- “System of systems” – a framework to effectively and efficiently integrate NOAA’s many systems
- Minimize impact on legacy systems
- Utilize standards:
- Adopt, adapt and only reluctantly create
- Emphasis on flexibility
- Work towards a service-oriented architecture

## Approach

- Each NOAA LO/program/project continues to manage its data independently
- Service Oriented Architecture*
- Agreed principles, guidelines and standards

-  Reference: Federal CIO Council, Jan ’06  "Services and Components Based Architectures: A Strategic Guide for Implementing Distributed and Reusable Components and Services in the Federal Government"

## Service-Oriented Architecture

- Under an SOA, capabilities are built one at a time to create “Web Services”
- Web services can be considered as a layer built on top of existing systems in which capabilities to access these systems are made available to applications that require them
- The fabric of the SOA is built upon standards for:
- discovery (e.g. CF, FGDC, ISO, SQL)
- transport (e.g. HTTP, FTP, OPeNDAP)
- use (e.g. netCDF, HTML, etc.)
- Can be tightly coupled (SOAP) or loosely coupled (REST)

## SOA Concept

## SOA (continued)

- Operational Public Access Services: for public access to data, products and information services
- Operational Services: where security, timeliness, and reliability are paramount
- Scientific Services: where efficient and flexible discovery and access to data sets are required
- Commercial value-added services

- Four general classes of web services are anticipated:

## Key Development Strategies

- Maintain and minimize impact on legacy systems
- Evolutionary development through pilot projects
- Coordinate activities through Communities of Interest organized by “Data Types”
- Grids, time-series, moving-sensor multi-dimensional, profiles, trajectories, geospatial framework, point data and metadata

## Communities of interest based upon data typesAn initial list

**Grids **(e.g. model output, gridded data products)
**Moving-sensor multidimensional fields**  (e.g. satellite swaths, side-scan sonar, weather radar)
**Time series** (e.g. fish landings, sun spot activity, climate data, paleo-records)
**Profiles** (e.g. atmospheric soundings, ocean casts, profiling floats)
**Trajectories **(e.g. underway ship measurements, aircraft track data, ocean surface drifters)
**Geospatial Framework Data **(e.g. shorelines, fault lines, marine boundaries, map annotations)
**Point data **(e.g. tsunami or seismic occurrences, geodetic control)
**Metadata** - information needed for the use and interpretation of data

## Standards

- NOAA-wide standards are an essential component
- Adopt, adapt and only as a last resort develop
- Process is proposed to guide nomination, evaluation and adoption of standards

## NOAA GEO-IDE Standards Process 

**Open** – clear means for all interested parties to participate
**Efficient** – as streamlined as possible given other requirements
**Dynamic** – standards updated or retired as technology and practices evolve
**Coordinated** – consult with other organizations that are facing related standards issues (e.g. Federal, GEOSS, WMO, IOOS, industry, etc)
**Technically sound** – decisions based on technical merits and cost/benefits
**Methodical and evolutionary** – any new standards will be harmonized with existing successful standards
**Layered** – adopt broadly accepted industry standards but with discipline-specific profiles, schemas and vocabularies
**Grounded in software practices** – standards will be tested in functioning software before being accepted
**Communication** - outreach and education to ensure NOAA data managers receive adequate training and support in the utilization of standards

## NOAA GEO-IDE Standards Process(continued)

**Submitted Standard** – Standards can be submitted by anyone in NOAA for evaluation to see if they address needs and could be applicable to NOAA
**Proposed NOAA Standard** – Can be provisionally used within NOAA for technical evaluation
**Recommended NOAA Standard** – NOAA data systems should consider supporting the standard wherever applicable for evaluation in real-world NOAA systems
**NOAA Standard** – Approved and mandated where appropriate

- Phased approach of submission, evaluation and adoption

## NOAA GEO-IDE Standards Process(continued)

- GEO-IDE standards are inclusive not exclusive
- not intended to stipulate that any standard can NOT be utilized in a NOAA data system
- Intended to identify standards that should be supported by NOAA information management systems. Examples
- NOAA systems must support standard A or B; or
- NOAA systems exchanging data as part of a service-oriented architecture must support this standard; or
- NOAA system must be able to _ingest_ data using this standard; or
- NOAA system must be able to _output_ data in standard A or B

## Types of Standards Needed

- Standard names and terminology
- Metadata standards (content and format)
- e.g. FGDC and ISO 19115 w/ remote sensing extensions
- Standard formats for delivery of data/products
- WMO, NetCDF, HDF, GeoTIF, JPEG, etc.
- Web Services Standards
- World Wide Web Consortium
- OGC (Features, Coverage, GML)
- OPeNDAP

## Fast-track submission

- An initial set of well known and/or widely used standards are being considered as an initial set of “submitted” standards

## Data Management Principles

**Leadership: **Information is a strategic asset and information management must be a key component of every environmental program
**Stewardship: **Data must be collected, produced, documented and maintained keeping the requirements of _all_ users in mind
**Long-term preservation:** Observations, data products of lasting value, and associated metadata must be preserved
**Requirements-driven: **Providers and users of data and products must play an active role in defining requirements that drive development and evolution of data management systems
**Discovery and access:** Mechanisms to facilitate discovery, delivery, use and interpretation of data and products are essential

## Data Management Principles(continued)

**Standards and practices:** Effective application of standards and best practices contributes to development of systems that are interoperable, efficient, reliable, scalable and adaptable
**Quality:** Data, products and information should be of quality sufficient to meet the requirements of society and to support sound decision making
**Cooperation and coordination**: Each organization can realize the full potential of its data only by participating in a global community of integrated information management systems
**Security:** Data, information, and products must be preserved and protected from unintended or malicious modification, unauthorized use, or inadvertent disclosure

## NOAA Guide on Integrated Information Management

- The primary source for guidance on NOAA environmental information systems
- Available on-line and updated frequently
- Contains or provides pointers to:
- NOAA and other Federally-mandated data management policies
- NOAA-wide standards (in all stages of approval)
- Registry of information management tools and software
- Planning template with guidelines and/or checklist for creating and implementing NOAA project-level data management implementation plans

## Governance Structure and Program Control 

- GEO-IDE Project Office
- NOAA Data Management Integration Team (DMIT)
- Technical implementation teams
- Standards Team
- Training Team
- SOA Team
- Communication Team
- And others...

## Advantages for developers

- Single reference point for NOAA data management guidelines, principles and standards (“The Guide”)
- Catalog of software tools
- More efficient development by enhanced re-use of code and utilization of “services”
- More customers for your data and products

## Advantages for scientistsand other customers

- NOAA data and products are easier to:
- Locate
- Acquire  and load into their favorite software
- Use and understand
- Integrate with other data from NOAA
- Integrate with data from other US-GEO and GEOSS partners

## How you can participate

- Participate in the NOAA standards process:
- Nominate standards
- Apply relevant standards to your work
- Develop and distribute any tools, extensions, or enhancements to standards that are needed to meet your requirements
- Provide evaluation comments to GEO-IDE Project Office
- Contribute to the work of one of the GEO-IDE implementation teams
- Develop proof of concept prototypes to explore what core Web services are needed
- Use core NOAA Web services when developing new systems
- Develop Web Service interfaces to your legacy systems
- Contribute software to be listed in “The Guide”

## Backup Slides