**Near-Term Missions Standards Recommendations**

**December 2, 2002**

**SEEDS Near-Term Mission Standard Study Team**

**Contributors**

> Richard Ullman, NASA/GSFC, Study Team Lead
>
> Jingli Yang, ERT, Study Team
>
> Cheryl Craig, NCAR, Study Team
>
> John Evans, GST, Study Team
>
> Larry Klein, L-3 Analytics, Study Team
>
> Dorian Shuford, ERT, Study Team
>
> Siri Jodha Singh Khalsa, L-3 Analytics, Study Team
>
> Matt Smith, UAH, Study Team

**Table of Contents**

[1.0 Introduction 1](#introduction)

[1.1 SEEDS Goals and Strategy 1](#seeds-goals-and-strategy)

[1.2 The Rationale for Standards 2](#the-rationale-for-standards)

[1.3 Assumptions 4](#assumptions)

[1.4 Methodology 4](#methodology)

[2.0 Near-Term Mission and Heritage Mission Standards
6](#near-term-mission-and-heritage-mission-standards)

[2.1 SEEDS Near-Term Missions 6](#seeds-near-term-missions)

[2.2 Heritage Mission Standards 7](#heritage-mission-standards)

[3.0 Lessons Learned 12](#lessons-learned)

[3.1 Lessons Learned on Implementing and Using NASA EOS Standards
12](#lessons-learned-on-implementing-and-using-nasa-eos-standards)

[3.1.1 Landsat 7 12](#landsat-7)

[3.1.2 TERRA 13](#terra)

[3.1.3 AQUA 14](#aqua)

[3.1.4 AURA 16](#aura)

[3.1.5 QuikSCAT/SeaWinds 17](#quikscatseawinds)

[3.1.6 ACRIM 18](#acrim)

[3.1.7 SeaWiFS 18](#seawifs)

[3.1.8 Jason-1 19](#jason-1)

[3.1.9 AVHRR 19](#avhrr)

[3.2 Lessons Learned on Implementing and Using other Standards
20](#lessons-learned-on-implementing-and-using-other-standards)

[3.2.1 NOAA Standards 20](#noaa-standards)

[3.2.2 The Spatial Data Transfer Standard (SDTS)
21](#the-spatial-data-transfer-standard-sdts)

[4.0 Essential Standards Concepts 25](#essential-standards-concepts)

[4.1 A Comparison with Private, Ad-hoc, Binary Information Transfer
25](#a-comparison-with-private-ad-hoc-binary-information-transfer)

[4.2 Mandatory vs. Optional Elements, Profiles and Extensions
27](#mandatory-vs.-optional-elements-profiles-and-extensions)

[4.3 Abstract vs. Implementation Standards
28](#abstract-vs.-implementation-standards)

[4.4 Content and Format vs. Behavior and Interface
29](#content-and-format-vs.-behavior-and-interface)

[4.5 Web-based Data Service Standards
32](#web-based-data-service-standards)

[5.0 Standards Evaluation 35](#standards-evaluation)

[5.1 Evaluation Criteria 35](#evaluation-criteria)

[5.2 Data Standards Evaluation 36](#data-standards-evaluation)

[5.3 Metadata and Documentation Standards Evaluation
45](#metadata-and-documentation-standards-evaluation)

[5.4 Catalog Interface Standards Evaluation
47](#catalog-interface-standards-evaluation)

[5.5 User Surveys 50](#user-surveys)

[5.5.1 Data Format Standards 50](#data-format-standards)

[5.5.2 Metadata Format Standards 54](#metadata-format-standards)

[6.0 Summary 57](#summary)

[7.0 Conclusions 60](#conclusions)

[7.1 Data Packaging Standards 60](#data-packaging-standards)

[NTMS recommends the following for packaging of near-term mission
standard products.
60](#ntms-recommends-the-following-for-packaging-of-near-term-mission-standard-products.)

[7.1.1 Data Distribution Formats Recommendations
60](#data-distribution-formats-recommendations)

[7.1.2 Data Interchange Formats Recommendations
61](#data-interchange-formats-recommendations)

[7.2 Metadata Standards Recommendations
62](#metadata-standards-recommendations)

[7.3 Documentation Standards Recommendations
63](#documentation-standards-recommendations)

[7.4 Catalog Interface Standards Recommendations
64](#catalog-interface-standards-recommendations)

[7.5 Standard Evolution Process & Other Activities Recommendations
64](#standard-evolution-process-and-other-activities-recommendations)

[Acronym List 66](#acronym-list)

**List of Tables and Figures**

[Figure 1.1.1 Simplified ESE network Data Flow
2](#__RefHeading___Toc26754273)

[Table 2.1.1 SEEDS Near-Term Missions 6](#__RefHeading___Toc26754274)

[Table 2.1.2 SEEDS Near-Term Mission Standards
6](#__RefHeading___Toc26754275)

[Table 2.2.3 SEEDS Heritage Missions Data Management Information
7](#__RefHeading___Toc26754276)

[Table 4.4.1 Viewpoints and Levels of Abstraction
29](#__RefHeading___Toc26754277)

[Table 4.4.2 Criteria For Format Standards
30](#__RefHeading___Toc26754278)

[Table 4.4.3 Criteria For Interface Standards
30](#__RefHeading___Toc26754279)

[Table 4.4.4 Data Models and Software Access Libraries
30](#__RefHeading___Toc26754280)

[Table 5.2.1 Data Standards Interoperability
37](#__RefHeading___Toc26754281)

[Table 5.2.2 Data Standards Availability
38](#__RefHeading___Toc26754282)

[Table 5.2.3 Data Standards Portability 38](#__RefHeading___Toc26754283)

[Table 5.2.4 Data Standards Evolvability
39](#__RefHeading___Toc26754284)

[Table 5.2.5 Data Standards Extensibility
41](#__RefHeading___Toc26754285)

[Table 5.2.6 Data Standards Self-Describing
41](#__RefHeading___Toc26754286)

[Table 5.2.7 Data Standards Tools Support
43](#__RefHeading___Toc26754287)

[Table 5.2.8 Semantic Completeness 44](#__RefHeading___Toc26754288)

[Table 5.2.9 Data Standards Evaluation 45](#__RefHeading___Toc26754289)

[Table 5.3.1 Metadata and Documentation Standards Evaluation
46](#__RefHeading___Toc26754290)

[Table 5.4.1 Catalog Interface Standards Evaluation
48](#__RefHeading___Toc26754291)

[Table 5.5.1 Survey Ratings of Attribute Importance
51](#__RefHeading___Toc26754292)

[Table 5.5.2 Data Standards Survey Evaluation
52](#__RefHeading___Toc26754293)

[Table 5.5.3 Summary of Survey Essay Questions
54](#__RefHeading___Toc26754294)

[Table 5.5.4 Metadata Standards Survey Evaluation
55](#__RefHeading___Toc26754295)

[Table 5.5.5 Summary of Metadata Survey Essay Questions
56](#__RefHeading___Toc26754296)

# Introduction 

## SEEDS Goals and Strategy 

SEEDS, previously called NewDISS, is the [S]{.underline}trategic
[E]{.underline}volution of the [E]{.underline}arth Science Enterprise
[D]{.underline}ata [S]{.underline}ystems meant to serve research and
application needs in the next ten years. Its primary goal is to support
NASA's Earth Science Enterprise (ESE), which, in turn, contributes to
the US Global Change Research Program (USGCRP). As such, SEEDS is driven
principally by the objectives of scientific research, but must also
serve the needs of both scientific research and a wide variety of
practical applications.

Future ESE data systems will consist of a heterogeneous mix of
interdependent components derived from the contributions of numerous
individuals and institutions. These widely varying participants will be
responsible for data management functions including data acquisition and
synthesis; access to data and services; and data stewardship.

\"An important premise underlying the operation of \[the ESE network of
data systems and services\] is that its various parts should have
considerable freedom in the ways in which they implement their functions
and capabilities. Implementation will not be centrally developed, nor
will the pieces developed be centrally managed. However, every part of
\[the ESE network\] should be configured in such a way that data and
information can be readily transferred to any other. This will be
achieved primarily through the adoption of common standards and
practices \[[[]{#d1_1_1 .anchor}1](#r1_1_1)\].\"

Figure 1.1.1 is a simplified data flow diagram of the ESE network of
data systems and services \[[1](#r1_1_1)\]. Five types of data centers,
namely Backbone Processing Centers, Principal Investigator (PI)-managed
Mission Data Centers, Science Data Centers, Applications Data Centers,
and Multimission Data Centers are shown in the diagram. Several data
flows, such as data flows from PI-managed Mission Data Centers to
Multimission Data Centers and vice versa, from Science Data Centers to
Applications Data Centers and vice versa, from Science Data Centers to
Science Data Center, from PI-managed Mission Data Centers to PI-managed
Mission Data Centers, etc. are omitted for simplicity. Four different
types of data flow are identified in the diagram. Internal data flow
refers to data flow inside each data center. L0, or raw spacecraft
instrument data flow, refers to spacecraft or level 0 data flow between
mission operations, PI-managed Data Centers or Multimission Data
Centers, and Backbone or Long-Term Archive Data Centers. Distribution
flow denotes data distribution to end-users. System interchange flow
denotes data exchange between data centers. As suggested by Figure
1.1.1, the ESE network provides a means for opening numerous new
channels for Earth Science satellite data streams to reach the user
community. Such data streams will flow to users both directly from
mission data processing centers as well as via many intermediate
information providers.

![](media/image1.wmf){width="5.997916666666667in"
height="3.683333333333333in"}

[]{#__RefHeading___Toc26754273 .anchor}Figure 1.1.1 Simplified ESE
network Data Flow (Adopted from Figure C-2 \[[**1**](#r1_1_1)\])

The SEEDS Near-Term Missions Standards (NTMS) study group is tasked to
make recommendations for the use of standards by the ESE near-term
missions. These standards are not meant to prescribe the ways that each
near-term mission manages data internally or the L0, or raw spacecraft
instrument data flow. Instead, the recommended standards pertain to the
data distribution to end-users and to the data interchange between the
ESE network of data systems and services components (*i.e.,* between
different data centers as shown in Figure 1.1.1).

## The Rationale for Standards

Standards aid in interoperability between data systems and facilitate
access by users and the software they use. The successful adoption and
use of standards for the ESE network of data systems will reduce the
cost and enhance the efficiency of data system development and
maintenance. Use of standards for the interchange among the ESE data and
service components also makes it easy for data and service providers to
join the ESE network of data systems without negotiating one-to-one
agreements with each potential provider. The standards that the NTMS
study group is addressing include data packaging standards, data catalog
interface standards, metadata standards, and documentation standards, as
defined below.

-   *Data Packaging Standards* define how to package or encode data that
    is stored on a computer or transferred from one system to another.
    Software libraries may be available to facilitate decoding,
    encoding, or manipulating data packaged in a particular way.

-   *Data Catalog Interface Standards* specify data search-and-order
    requests between ESE data and services components, usually over a
    network. These interface standards are defined independently of the
    data's packaging (encoding).

-   *Metadata / Documentation Standards* provide a common lexicon and a
    set of attributes describing data to ensure that users can 1) find
    the data in catalogs, registries, and other indexes; 2) interpret
    the data unambiguously; and 3) apply system services correctly.
    Metadata is usually highly structured and formalized, whereas,
    documentation usually refers to more free-text descriptions. Most
    metadata and documentation standards are content standards
    (format-independent); XML is a popular encoding for metadata.

    -   *Content Standards* for data or metadata define the information
        elements and their intended meaning (semantics), independently
        of how these elements may be encoded in files (their syntax).
        Two or more encodings of the same content standard can be mapped
        (machine-translated) to each other with no loss of information.

For years, various satellite missions and scientific communities have
found ways to use each other's data, but stable, rich standards can
further promote opportunities in research and applications for data
users worldwide. The evolution of these standards over the past 25 years
or so has largely been driven by specific science communities with a
goal of making life easier for themselves. The past 10 years or so has
seen ever wider global scientific communities tied together through the
Internet with a goal of still faster-paced data exchange and hopefully
faster-paced research results. However, the diversity of available data
sources and data standards presents a significant challenge to Earth
science researchers, especially interdisciplinary Earth scientists.

As almost any researcher can attest, a substantial portion of the
resources required to perform an investigation are expended on locating,
obtaining, and then reading and possibly reformatting the necessary
data. Standardization of data formats, metadata, documentation, and
catalog interface standards can lower the threshold on data exchange
between the ESE network of data systems and services components and the
user access to the data products. The Internet offers a compelling
example of the essential role standards play in facilitating data
exchange. Without the underpinnings of the Internet - TCP/IP, HTML,
SMTP, GIF, JPEG, PDF, etc., the explosion of information exchange
brought about by the Internet could never have happened.

## Assumptions

This study focuses on near-term missions that are already in formulation
and is aimed to provide concrete, specific recommendations for the
near-term missions' use. The following assumptions are made to carry out
this study.

1.  The emerging field of Web Services is driving rapid development of
    data format-neutral service interface standards. Examples relevant
    to ESE data include the OpenGIS Web Map Service and Web Coverage
    Service. However, the use of online services is still only emerging
    in practical ESE work; it will take some time before Web Services
    become a part of mainstream data access and distribution.

2.  For the near-term missions, the preferred mode of delivering data
    remains the transfer of discrete files. Therefore the file format
    itself is critical to the interchange standard.

3.  Content data standards (define the information elements and their
    intended meaning (semantics), independently of their syntax) provide
    well-known semantics that can support interoperability through
    translators or cross-reference tables. The leading definition for
    such standards is the Federal Geographic Data Committee (FGDC) that
    has developed *Content Standard for Remote Sensing Swath Data* and
    *Content Standards for Digital Orthoimagery*. However in practice,
    content standards alone may not suffice for transferring complex
    data between different user communities without information loss or
    distortion.

4.  The processes of standards development and adoption are the
    responsibility of the long-term standards study team. The NTMS study
    recommends a first evolutionary step in adoption of standards by
    endorsing specific standards and practices.

## Methodology

This document provides recommendations for the use of standards by the
near-term missions. We analyzed what standards are currently in use in
the near-term heritage missions and other EOS missions, posing questions
such as: What are the lessons learned on implementing and using those
standards currently in use? What are the lessons learned from other
government agencies such as NOAA? What criteria should we use to
evaluate different standards? What feedback do data producers and data
users have on standards? What standards do users think NASA should use
in the future? Once we provide recommendations, how can the
recommendations be implemented for the near-term missions? What
respective activities should be supported in order to facilitate the
adoption of the standards?

This report intends to answer these questions. We studied the published
objectives of the assigned missions and interviewed some key planners in
an attempt to understand the role data systems and data systems
standards were expected to play in those missions and did play in their
direct heritage. We discussed our progress at the first SEEDS Public
Workshop held February 5-8, 2002 in Greenbelt, Maryland. To verify our
general understanding, we also asked for, and received, direct
one-on-one feedback from the near-term missions on our draft survey. We
investigated each of the standards identified by the mission heritage
survey and common standards used in other government agencies and
industry. We researched their technical aspects and surveyed the
opinions held by potential end-users and producers. We developed a
structured survey and individually interviewed many EOSDIS DAAC User
Working Group members and data users and producers in NOAA. We also
conducted a survey of EOS data users and producers at the 2002 NASA
Science Data Processing Workshop. The gathered survey, interview, and
workshop opinions were consolidated.

**[References:]{.underline}**

\[[[]{#r1_1_1 .anchor}1](#d1_1_1)\] A 6 to 10 Year Approach to Data
Systems and Services for NASA's Earth Science Enterprise; Draft Version
1.0; February 2001; Section A.3.

# Near-Term Mission and Heritage Mission Standards

## SEEDS Near-Term Missions

The missions that SEEDS is initially targeted to support include the
following eight near-term missions (Table 2.1.1).

[]{#__RefHeading___Toc26754274 .anchor}Table 2.1.1 SEEDS Near-Term
Missions

  --------------------------------------- --------------- ---------------
  **Mission Name**                        **Phase**       **Anticipated
                                                          Launch Date**

  Landsat Data Continuity Mission (LDCM)  Formulation     2006

  NPOESS Preparatory Project (NPP)        Formulation     2006

  Ocean Surface Topography Measurement    Formulation     2006
  (OSTM)                                                  

  Ocean Vector Winds                      Formulation     2007

  Global Precipitation Measurement (GPM)  Formulation     2007

  Solar Irradiance                        Formulation     2007

  Carbon Cycle Initiative (CCI)           Pre-            2008-2012
                                          Formulation     

  Total Column Ozone                      Pre-            N/A
                                          Formulation     
  --------------------------------------- --------------- ---------------

[See Acronym List if needed](#acronym-list)

A summary of the near-term mission instruments, data formats, and
metadata standards is described in Table 2.1.2. As shown in the table,
LDCM, the first near-term mission, has already decided the data and
metadata standards they plan to use for the mission data products
(specified in the Request For Proposal (RFP) they released October
2001). Our recommendations for the use of data, metadata, and data
interfaces in near-term missions may, or may not, impact the LDCM
mission.

[]{#__RefHeading___Toc26754275 .anchor}Table 2.1.2 SEEDS Near-Term
Mission Standards

+------------------+-----------------------+------------+-------------+
| **Missions**     | **Instrument**        | **Data     | **Metadata  |
|                  |                       | Format**   | Format**    |
+==================+=======================+============+=============+
| LDCM             | Not specified         | 1\. HDF    | 1\. ECS     |
|                  |                       |            |             |
|                  |                       | 2\.        | 2\. FGDC    |
|                  |                       | GeoTIFF    |             |
|                  |                       |            |             |
|                  |                       | 3\. L7     |             |
|                  |                       | Fast       |             |
|                  |                       | Format     |             |
+------------------+-----------------------+------------+-------------+
| NPP              | ATMS                  | N/A        | N/A         |
+------------------+-----------------------+------------+-------------+
|                  | CrIS                  |            |             |
+------------------+-----------------------+------------+-------------+
|                  | VIIRS                 |            |             |
+------------------+-----------------------+------------+-------------+
| OSTM (or         | N/A                   | N/A        | N/A         |
| Jason-2)         |                       |            |             |
+------------------+-----------------------+------------+-------------+
| Ocean Winds      | Seawinds              | N/A        | N/A         |
+------------------+-----------------------+------------+-------------+
| GPM              | Dual Frequency Radar  | N/A        | N/A         |
|                  | (DFR)                 |            |             |
+------------------+-----------------------+------------+-------------+
|                  | Advanced TRMM         |            |             |
|                  | Microwave Imager      |            |             |
|                  | (TMI)                 |            |             |
+------------------+-----------------------+------------+-------------+
|                  | Nadir-viewing         |            |             |
|                  | Microwave Radiometer  |            |             |
+------------------+-----------------------+------------+-------------+
| Solar Irradiance | N/A                   | N/A        | N/A         |
+------------------+-----------------------+------------+-------------+
| CCI Missions:    | A passive             | N/A        | N/A         |
|                  | spectrometer          |            |             |
| 1\. Pathfinder   |                       |            |             |
| CO~2~            |                       |            |             |
|                  |                       |            |             |
| 2\. Ocean Carbon |                       |            |             |
|                  |                       |            |             |
| 3\. Low Density  |                       |            |             |
| Biomass          |                       |            |             |
|                  |                       |            |             |
| 4\. High Density |                       |            |             |
| Biomass          |                       |            |             |
|                  |                       |            |             |
| 5\. Advanced     |                       |            |             |
| Atmospheric      |                       |            |             |
| CO~2~            |                       |            |             |
+------------------+-----------------------+------------+-------------+
|                  |                       |            |             |
+------------------+-----------------------+------------+-------------+
|                  | A rotating scanner    |            |             |
|                  | telescope             |            |             |
+------------------+-----------------------+------------+-------------+
|                  | A hyperspectral       |            |             |
|                  | imager                |            |             |
+------------------+-----------------------+------------+-------------+
|                  | A P-band SAR and an   |            |             |
|                  | imaging laser         |            |             |
|                  | altimeter             |            |             |
+------------------+-----------------------+------------+-------------+
|                  | A pulsed, dual        |            |             |
|                  | frequency, tunable    |            |             |
|                  | laser sounder         |            |             |
+------------------+-----------------------+------------+-------------+
| Total Column     | Some combination of   | N/A        | N/A         |
| Ozone            | OMPS-like, TOMS-like, |            |             |
|                  | SAGE-like and an IR   |            |             |
|                  | limb sounder          |            |             |
+------------------+-----------------------+------------+-------------+

[See Acronym List if needed](#acronym-list)

## Heritage Mission Standards

Data management information for near-term missions and heritage missions
is presented in Table 2.2.3.

[]{#__RefHeading___Toc26754276 .anchor}Table 2.2.3 SEEDS Heritage
Missions Data Management Information

+-------+-------+---------+---------+-------+------+-------+-------+
| Mi    | Her   | H       | Pro     | Ar    | Data | Met   | Ca    |
| ssion | itage | eritage | duction | chive | Fo   | adata | talog |
|       | Mi    | Ins     | Site    | Site  | rmat | F     | Inte  |
|       | ssion | trument |         |       |      | ormat | rface |
+=======+=======+=========+=========+=======+======+=======+=======+
| LDCM  | La    | TM      | EDC     | EDC   | 1\.  | 1\.   | C     |
|       | ndsat |         | DAAC    | DAAC  | HDF  | ECS   | ollec |
|       | 1-7   | ETM+    |         |       | 4    |       | tion: |
|       |       |         |         |       |      | 2\.   | Z3    |
|       |       |         |         |       | 2\.  | FGDC  | 9.50, |
|       |       |         |         |       | Geo  |       |       |
|       |       |         |         |       | TIFF |       | C     |
|       |       |         |         |       |      |       | ustom |
|       |       |         |         |       | 3\.  |       |       |
|       |       |         |         |       | L7   |       | Inven |
|       |       |         |         |       | Fast |       | tory: |
|       |       |         |         |       | Fo   |       | E     |
|       |       |         |         |       | rmat |       | OSDIS |
|       |       |         |         |       |      |       | V0    |
|       |       |         |         |       |      |       |       |
|       |       |         |         |       |      |       | Guide |
|       |       |         |         |       |      |       | Docu  |
|       |       |         |         |       |      |       | ment: |
|       |       |         |         |       |      |       | Z     |
|       |       |         |         |       |      |       | 39.50 |
+-------+-------+---------+---------+-------+------+-------+-------+
| NPP   | Aqua  | AMSU    | GSFC    | GSFC  | HDF  | ECS   | C     |
|       |       |         | DAAC    | DAAC  | -EOS |       | ollec |
|       |       | HSB     |         |       | 4    |       | tion: |
|       |       |         |         |       |      |       | Z3    |
|       |       | AIRS    |         |       |      |       | 9.50, |
|       |       |         |         |       |      |       |       |
|       |       |         |         |       |      |       | C     |
|       |       |         |         |       |      |       | ustom |
|       |       |         |         |       |      |       |       |
|       |       |         |         |       |      |       | Inven |
|       |       |         |         |       |      |       | tory: |
|       |       |         |         |       |      |       | E     |
|       |       |         |         |       |      |       | OSDIS |
|       |       |         |         |       |      |       | V0    |
|       |       |         |         |       |      |       |       |
|       |       |         |         |       |      |       | Guide |
|       |       |         |         |       |      |       | Docu  |
|       |       |         |         |       |      |       | ment: |
|       |       |         |         |       |      |       | Z     |
|       |       |         |         |       |      |       | 39.50 |
+-------+-------+---------+---------+-------+------+-------+-------+
|       | Terra | MODIS   | GSFC    | GSFC  | HDF  | ECS   |       |
|       |       |         | DAAC    | DAAC  | -EOS |       |       |
|       |       |         |         |       | 4    |       |       |
|       |       |         | NSIDC   | NSIDC |      |       |       |
|       |       |         | DAAC    | DAAC  |      |       |       |
|       |       |         |         |       |      |       |       |
|       |       |         | EDC     | EDC   |      |       |       |
|       |       |         | DAAC    | DAAC  |      |       |       |
+-------+-------+---------+---------+-------+------+-------+-------+
| OSTM  | Ja    | Pos     | PO DAAC | PO    | Na   | C     | N/A   |
|       | son-1 | eidon-2 |         | DAAC  | tive | ustom |       |
|       |       | Radar   | AVISO   |       | Bi   |       | (Data |
|       |       | Al      |         | AVISO | nary |       | is    |
|       |       | timeter |         |       |      |       | not   |
|       |       |         |         |       |      |       | avai  |
|       |       |         |         |       |      |       | lable |
|       |       |         |         |       |      |       | to    |
|       |       |         |         |       |      |       | p     |
|       |       |         |         |       |      |       | ublic |
|       |       |         |         |       |      |       | yet)  |
+-------+-------+---------+---------+-------+------+-------+-------+
|       |       | Jason   |         |       |      |       |       |
|       |       | Mi      |         |       |      |       |       |
|       |       | crowave |         |       |      |       |       |
|       |       | Rad     |         |       |      |       |       |
|       |       | iometer |         |       |      |       |       |
+-------+-------+---------+---------+-------+------+-------+-------+
|       | Tope  | Topex   | PO DAAC | PO    | Na   | C     | C     |
|       | x/Pos | Al      |         | DAAC  | tive | ustom | ollec |
|       | eidon | timeter | AVISO   |       | Bi   |       | tion: |
|       |       |         |         | AVISO | nary |       | Z3    |
|       |       |         |         |       | for  |       | 9.50, |
|       |       |         |         |       | low  |       |       |
|       |       |         |         |       | l    |       | C     |
|       |       |         |         |       | evel |       | ustom |
|       |       |         |         |       | prod |       |       |
|       |       |         |         |       | ucts |       | Inven |
|       |       |         |         |       |      |       | tory: |
|       |       |         |         |       | ne   |       | E     |
|       |       |         |         |       | tCDF |       | OSDIS |
|       |       |         |         |       | for  |       | V0    |
|       |       |         |         |       | L    |       |       |
|       |       |         |         |       | evel |       | Guide |
|       |       |         |         |       | 3    |       | Docu  |
|       |       |         |         |       | pro  |       | ment: |
|       |       |         |         |       | duct |       | Z     |
|       |       |         |         |       |      |       | 39.50 |
+-------+-------+---------+---------+-------+------+-------+-------+
|       |       | Topex   |         |       |      |       |       |
|       |       | Mi      |         |       |      |       |       |
|       |       | crowave |         |       |      |       |       |
|       |       | Rad     |         |       |      |       |       |
|       |       | iometer |         |       |      |       |       |
+-------+-------+---------+---------+-------+------+-------+-------+
| Ocean | Ad    | NSCAT   | JPL     | PO    | HDF  | Ad    | C     |
| Winds | eos-1 |         | SeaPAC  | DAAC  | 4    | apted | ollec |
|       |       |         |         |       |      | ECS   | tion: |
|       |       |         |         |       |      |       | Z3    |
|       |       |         |         |       |      |       | 9.50, |
|       |       |         |         |       |      |       |       |
|       |       |         |         |       |      |       | C     |
|       |       |         |         |       |      |       | ustom |
|       |       |         |         |       |      |       |       |
|       |       |         |         |       |      |       | Inven |
|       |       |         |         |       |      |       | tory: |
|       |       |         |         |       |      |       | E     |
|       |       |         |         |       |      |       | OSDIS |
|       |       |         |         |       |      |       | V0    |
|       |       |         |         |       |      |       |       |
|       |       |         |         |       |      |       | Guide |
|       |       |         |         |       |      |       | Docu  |
|       |       |         |         |       |      |       | ment: |
|       |       |         |         |       |      |       | Z     |
|       |       |         |         |       |      |       | 39.50 |
+-------+-------+---------+---------+-------+------+-------+-------+
|       | Qui   | S       | JPL     | PO    | HDF  | Ad    |       |
|       | kscat | eawinds | SeaPAC  | DAAC  | 4    | apted |       |
|       |       |         |         |       |      | ECS   |       |
|       |       |         |         |       | BUFR |       |       |
+-------+-------+---------+---------+-------+------+-------+-------+
|       | Ad    | S       | JPL     | PO    | HDF  | Ad    |       |
|       | eos-2 | eawinds | SeaPAC  | DAAC  | 4    | apted |       |
|       |       |         |         |       |      |       |       |
|       |       |         |         |       | BUFR | ECS   |       |
+-------+-------+---------+---------+-------+------+-------+-------+
| GPM   | TRMM  | TMI     | GSFC    | GSFC  | HDF  | ECS   | C     |
|       |       |         | DAAC    | DAAC  | 4    |       | ollec |
|       |       |         |         |       |      |       | tion: |
|       |       |         |         |       |      |       | Z3    |
|       |       |         |         |       |      |       | 9.50, |
|       |       |         |         |       |      |       |       |
|       |       |         |         |       |      |       | C     |
|       |       |         |         |       |      |       | ustom |
|       |       |         |         |       |      |       |       |
|       |       |         |         |       |      |       | Inven |
|       |       |         |         |       |      |       | tory: |
|       |       |         |         |       |      |       | E     |
|       |       |         |         |       |      |       | OSDIS |
|       |       |         |         |       |      |       | V0    |
|       |       |         |         |       |      |       |       |
|       |       |         |         |       |      |       | Guide |
|       |       |         |         |       |      |       | Docu  |
|       |       |         |         |       |      |       | ment: |
|       |       |         |         |       |      |       | Z     |
|       |       |         |         |       |      |       | 39.50 |
+-------+-------+---------+---------+-------+------+-------+-------+
|       |       | VIIRS   | GSFC    | GSFC  | HDF  | ECS   |       |
|       |       |         | DAAC    | DAAC  | 4    |       |       |
+-------+-------+---------+---------+-------+------+-------+-------+
|       |       | PR      | GSFC    | GSFC  | HDF  | ECS   |       |
|       |       |         | DAAC    | DAAC  | 4    |       |       |
+-------+-------+---------+---------+-------+------+-------+-------+
|       |       | CERES   | LaRC    | LaRC  | HDF  | ECS   |       |
|       |       |         | SIP     | DAAC  | 4    |       |       |
+-------+-------+---------+---------+-------+------+-------+-------+
|       |       | LIS     | GHRC    | GHRC  | HDF  | ECS   |       |
|       |       |         | SIP     |       | 4    |       |       |
+-------+-------+---------+---------+-------+------+-------+-------+
| Solar | SNOE  | XPS     | LASP    | LASP  | A    | C     | C     |
| Irrad |       |         |         |       | SCII | ustom | ollec |
| iance |       |         |         |       |      |       | tion: |
|       |       |         |         |       |      |       | NSSDC |
|       |       |         |         |       |      |       | M     |
|       |       |         |         |       |      |       | aster |
|       |       |         |         |       |      |       | Ca    |
|       |       |         |         |       |      |       | talog |
|       |       |         |         |       |      |       |       |
|       |       |         |         |       |      |       | Inven |
|       |       |         |         |       |      |       | tory: |
|       |       |         |         |       |      |       | C     |
|       |       |         |         |       |      |       | ustom |
|       |       |         |         |       |      |       |       |
|       |       |         |         |       |      |       | Docu  |
|       |       |         |         |       |      |       | ment: |
|       |       |         |         |       |      |       | WAIS  |
+-------+-------+---------+---------+-------+------+-------+-------+
|       | UARS  | S       | UARS    | GSFC  | Na   | N     | C     |
|       | SOL   | OLSTICE | CDHF    | DAAC  | tive | ative | ollec |
|       | STICE |         | and     |       | Bi   | SFDU  | tion: |
|       |       | SIM     | GSFC    |       | nary | f     | Z3    |
|       |       |         |         |       |      | ormat | 9.50, |
|       |       |         |         |       | Fo   |       |       |
|       |       |         |         |       | rmat |       | C     |
|       |       |         |         |       |      |       | ustom |
|       |       |         |         |       |      |       |       |
|       |       |         |         |       |      |       | Inven |
|       |       |         |         |       |      |       | tory: |
|       |       |         |         |       |      |       | E     |
|       |       |         |         |       |      |       | OSDIS |
|       |       |         |         |       |      |       | V0    |
|       |       |         |         |       |      |       |       |
|       |       |         |         |       |      |       | Guide |
|       |       |         |         |       |      |       | Docu  |
|       |       |         |         |       |      |       | ment: |
|       |       |         |         |       |      |       | Z     |
|       |       |         |         |       |      |       | 39.50 |
+-------+-------+---------+---------+-------+------+-------+-------+
|       | ACRIM | TIM     | ACRIM   | LaRC  | HDF  | ECS   |       |
|       | III   |         | III     | DAAC  | 4    |       |       |
|       |       |         | SIPS    |       |      |       |       |
+-------+-------+---------+---------+-------+------+-------+-------+
|       | EOS   | SIM     | LASP    | GSFC  | H    | ECS   |       |
|       | SORCE |         | SORCE   | DAAC  | DF-5 |       |       |
|       |       | S       | SIP     |       |      |       |       |
|       |       | OLSTICE |         |       |      |       |       |
|       |       |         |         |       |      |       |       |
|       |       | XPS     |         |       |      |       |       |
|       |       |         |         |       |      |       |       |
|       |       | TIM     |         |       |      |       |       |
+-------+-------+---------+---------+-------+------+-------+-------+
| CCI   | Se    | SeaWiFS | GSFC    | GSFC  | HDF  | ECS   | C     |
|       | aStar |         | DAAC    | DAAC  |      |       | ollec |
|       |       |         |         |       | FF   |       | tion: |
|       |       |         |         |       |      |       | Z3    |
|       |       |         |         |       |      |       | 9.50, |
|       |       |         |         |       |      |       |       |
|       |       |         |         |       |      |       | C     |
|       |       |         |         |       |      |       | ustom |
|       |       |         |         |       |      |       |       |
|       |       |         |         |       |      |       | Inven |
|       |       |         |         |       |      |       | tory: |
|       |       |         |         |       |      |       | E     |
|       |       |         |         |       |      |       | OSDIS |
|       |       |         |         |       |      |       | V0    |
|       |       |         |         |       |      |       |       |
|       |       |         |         |       |      |       | Guide |
|       |       |         |         |       |      |       | Docu  |
|       |       |         |         |       |      |       | ment: |
|       |       |         |         |       |      |       | Z     |
|       |       |         |         |       |      |       | 39.50 |
+-------+-------+---------+---------+-------+------+-------+-------+
|       | Terra | MODIS   | GSFC    | GSFC  | HDF  | ECS   |       |
|       |       |         | DAAC    | DAAC  | -EOS |       |       |
|       |       |         |         |       | 4    |       |       |
+-------+-------+---------+---------+-------+------+-------+-------+
|       | Nim   | CZCS    | GSFC    | GSFC  | HDF  | N     |       |
|       | bus-7 |         | DAAC    | DAAC  |      | ative |       |
|       |       |         |         |       | DSP  | f     |       |
|       |       |         |         |       |      | ormat |       |
|       |       |         |         |       | CRTT |       |       |
+-------+-------+---------+---------+-------+------+-------+-------+
|       | VCL   | MBLA    | R       | EDC   | Unk  | Un    | Un    |
|       |       |         | aytheon | DAAC  | nown | known | known |
|       |       |         | ITSS    |       |      |       |       |
+-------+-------+---------+---------+-------+------+-------+-------+
| Total | Nim   | TOMS    | GSFC    | GSFC  | H    | ECS   | C     |
| C     | bus-7 |         | DAAC    | DAAC  | DF-4 |       | ollec |
| olumn |       |         |         |       |      |       | tion: |
| Ozone | Met   |         |         |       |      |       | Z3    |
|       | eor-4 |         |         |       |      |       | 9.50, |
|       |       |         |         |       |      |       |       |
|       | ADOES |         |         |       |      |       | C     |
|       |       |         |         |       |      |       | ustom |
|       | Earth |         |         |       |      |       |       |
|       | Probe |         |         |       |      |       | Inven |
|       |       |         |         |       |      |       | tory: |
|       | Qui   |         |         |       |      |       | E     |
|       | kTOMS |         |         |       |      |       | OSDIS |
|       |       |         |         |       |      |       | V0,   |
|       |       |         |         |       |      |       | C     |
|       |       |         |         |       |      |       | ustom |
|       |       |         |         |       |      |       |       |
|       |       |         |         |       |      |       | Guide |
|       |       |         |         |       |      |       | Docu  |
|       |       |         |         |       |      |       | ment: |
|       |       |         |         |       |      |       | Z     |
|       |       |         |         |       |      |       | 39.50 |
+-------+-------+---------+---------+-------+------+-------+-------+
|       | AURA  | OMI     | GSFC    | GSFC  | H    | ECS   | N/A   |
|       |       |         | DAAC    | DAAC  | DF-4 |       |       |
|       |       |         |         |       | for  |       | (Data |
|       |       |         |         |       | L    |       | is    |
|       |       |         |         |       | evel |       | not   |
|       |       |         |         |       | 0    |       | avai  |
|       |       |         |         |       | and  |       | lable |
|       |       |         |         |       | 1    |       | yet)  |
|       |       |         |         |       |      |       |       |
|       |       |         |         |       | HDF  |       |       |
|       |       |         |         |       | -EOS |       |       |
|       |       |         |         |       | 5    |       |       |
|       |       |         |         |       | for  |       |       |
|       |       |         |         |       | L    |       |       |
|       |       |         |         |       | evel |       |       |
|       |       |         |         |       | 2 up |       |       |
+-------+-------+---------+---------+-------+------+-------+-------+

[See Acronym List if needed](#acronym-list)

Several observations can be made from Table 2.2.3:

1.  Most of the heritage missions use the Hierarchical Data Format
    (HDF 4) or HDF-EOS (Earth Observing System) data formats and the
    EOSDIS Core System (ECS) metadata format for archiving and
    distribution. Note that the HDF 5 is a newer library meant to
    eventually replace HDF 4. HDF-EOS, used by Terra and Aqua missions,
    is based on HDF 4. HDF-EOS 5, to be used by the Aura mission, is
    based on HDF 5. Heritage missions that do not use the HDF or HDF-EOS
    data formats and the ECS metadata format for product distribution
    are the Jason-1, Topex/Poseidon, and the Upper Atmospheric Research
    Satellite (UARS) missions. The Jason-1 and Topex/Poseidon missions
    are heritage missions to the Ocean Surface Topography Mission. UARS
    is a heritage mission to the Solar Irradiance mission.

2.  Several heritage missions distribute their data products in multiple
    data and metadata formats. For example, Landsat missions distribute
    their data products in three different data formats, namely HDF,
    GeoTIFF, and Fast Format, and two metadata formats, ECS and FGDC.
    SeaWinds distributes their data products in HDF and BUFR (Binary
    Universal Format For Representation of data) format. The HDF format
    is for distributing research data products by the NASA Jet
    Propulsion Laboratory (JPL) Distributed Active Archive Center
    (DAAC), while BUFR format is used to distribute operational data
    products by NOAA NESDIS (National Environmental Satellite Data and
    Information Service).

3.  Data distribution formats for heritage missions consist of HDF,
    HDF-EOS, netCDF, GeoTIFF, Fast Format, BUFR, Binary, and ASCII.
    Metadata distribution formats for heritage missions include ECS,
    FGDC, and custom formats.

4.  The same catalog interface standards are used across all of the
    heritage missions except, SNOE, a solar physics mission. Catalog
    interfaces used for collection metadata include the NASA Global
    Change Master Directory and Mercury. The GCMD supports the ISO/ANSI
    Z39.50 standard as well as customized search capabilities. The
    Mercury is a less-structured search system for Earth science data
    that is based in Z39.50 and internet standards such as http and ftp.
    The catalog interface standard used for inventory data search and
    order is the EOSDIS Version 0 protocol. TOMS data can also be
    accessed from the TOMS instrument team website using internet
    standard ftp. The Guide document search is based on the Z39.50
    standard. Since XSP data is archived and distributed from the
    National Space Science Data Center (NSSDC), both the collection and
    inventory catalog searches are based on the NSSDC custom interfaces
    and standards. The XSP documentation search is based on the Wide
    Area Information System (WAIS) access.

# Lessons Learned

This chapter presents lessons learned from past experiences with data
and metadata standards used for NASA SEEDS heritage missions and NOAA
missions. Some of the lessons learned pertain to past experiences with
developing or implementing the standards, and others are related to past
experiences with using the standards.

## Lessons Learned on Implementing and Using NASA EOS Standards

### 3.1.1 Landsat 7 {#landsat-7 .unnumbered}

Landsat 7 data products are archived in the HDF format but distributed
in three different formats: GeoTIFF, Landsat 7 Fast Format, and HDF.
Based on statistics collected by the EDC DAAC \[Earth Resources
Observation System (EROS) Data Center (EDC) Distributed Active Archive
Center (DAAC)\] User Services from January 1, 2001, to September 30,
2001, most of the users ordered L-7 data either in Fast Format (46%) or
in GeoTIFF (42%). Only 12% of the users ordered L-7 data in HDF format.
Of the users who ordered data in HDF format, most were from
international ground stations and the data product they ordered was
Level 0R. HDF is the only format available for Level 0R. These
statistics indicate that:

-   User communities welcome multiple distribution data formats.
    Statistics have shown that users order Landsat 7 data in all three
    available formats with the majority (88%) of the users choosing
    GeoTIFF or Fast Format. This indicates that for well-developed
    satellite mission user communities such as the Landsat data user
    community, multiple data distribution formats are needed. Different
    users choose different data formats in their applications.

-   Heritage mission data distribution formats play an important role.
    The reason the majority of the Landsat 7 users choose GeoTIFF or
    Fast Format may be because the Landsat 7 heritage mission Landsat 5
    data products are distributed in Fast Format or GeoTIFF format.
    Thus, users were already familiar with those two formats. It seems
    natural that users should choose to use a format they are already
    familiar with rather than switching to a new data format, such as
    HDF.

-   GeoTIFF data format is gaining popularity among Geographic
    Information System (GIS) users. Landsat Thematic Mapper (TM) data
    (Landsat 4-5) products have been distributed in Fast Format
    since 1984. EDC DAAC began distributing Landsat 5 TM data products
    in GeoTIFF in recent years. However, based on the statistics
    collected from January 1 to September 30, 2001, almost half (42%) of
    the users order Landsat 7 data products in GeoTIFF format. As
    GeoTIFF format is becoming a popular data format in the GIS user
    community, EDC DAAC is considering distributing other land remote
    sensing data, such as ASTER (Advanced Spaceborne Thermal Emission
    And Reflection Radiometer) data products, in GeoTIFF format in
    addition to the HDF format.

### 3.1.2 TERRA {#terra .unnumbered}

The flagship in NASA\'s Earth Observing System (EOS), Terra launched on
December 18, 1999 and began collecting science data on February 24,
2000. There are five instruments onboard Terra, namely MODIS, ASTER,
MISR, CERES, and MOPITT (see [Acronym List](#acronym-list)). The data
products from Terra, consisting of a great variety of ocean, atmosphere,
and land data sets, are archived and distributed in HDF-EOS format as
required by the EOS project. Terra metadata conforms to the ECS data
model.

In the early 1990\'s, NASA\'s Earth Science Data Information Systems
(ESDIS) began evaluating data format standards in preparation for the
launches of the EOS satellites. In 1993, after careful consideration of
over a dozen different formats, ESDIS chose the Hierarchical Data Format
(HDF) for EOS standard data products. During the ECS design phase, it
was realized that while HDF was a good format to use for storing data,
further standardization would be advantageous. HDF provided little
convention for associating spatial and temporal information with the
science data itself. To enable additional standardization, the HDF-EOS
data format was developed. This format adds mechanisms for storing
geo-referencing and temporal information, data organization, and
metadata storage.

Terra instrument teams and users have had several problems with
implementing and using the HDF-EOS standard and the ECS data model.

-   The HDF-EOS Grid and Swath provided a natural structure for the bulk
    of data derived from instruments on Terra and other EOS missions;
    however, there was no convention for storing individual data values.
    For example, in the case of one producer, real numbers are stored in
    14 bits and 2 additional bits are used for a special purpose rather
    than using all 16 bits to store the number. The HDF-EOS library can
    access these data; however, translation and other application tools
    can have problems. If processing is to be performed on individual
    words or bits, errors can occur if the user is not cognizant of the
    storage method.

-   There was no convention for packaging both HDF-EOS and HDF objects
    in the same file. All MODIS (Moderate-Resolution Imaging
    Spectroradiometer) Level 2 and 3 products are different. Even though
    they use HDF-EOS structures to store their primary data, many and
    varied vanilla HDF objects are included in MODIS standard products.
    MODIS also uses global and local text attributes to store non-ECS
    metadata rather than dumping it all into the ArchiveMetadata
    attributes as the HDF-EOS design calls for. This implies that
    software beyond the HDF-EOS library is required to access the
    additional attributes.

-   Even though HDF-EOS provides a standard for packaging geolocation
    information, there was no detailed standard for actually calculating
    this information. For example, some ASTER products are geolocated
    using a geoid (geodetic coordinates) while others are geolocated
    using an ellipsoid (geocentric coordinates). This is not a priori
    obvious to data users.

-   HDF-EOS has a steep learning curve. Once that hurdle is overcome,
    platform independence and common packaging provide convenience in
    access. However, scientists who are used to flat binary format
    complain about the complexity of HDF-EOS.

-   It was a mistake to try to have one HDF-EOS profile to fit all
    disciplines. In Terra MODIS case, this leads to unproductive
    wrangling, an overly broad profile, and poor fit for some (maybe
    all) disciplines. The lesson learned is to develop strong discipline
    specific profiles and worry about crossing disciplines later.

-   An important lesson learned from Terra is not to impose immature
    standards such as HDF-EOS. All the following are needed in no less
    than launch time minus three years:

    -   Need an expert base before products are defined.

    -   Need tools to verify proper implementation.

    -   Need experienced help desk support (and more) and to help with
        implementation.

-   There have been many mismatches between ESDT (Earth Sciences Data
    Type) and metadata output from MODIS production. This has led to a
    large number of product ingest failures at the DAACs. Quality
    control on the production end is lacking, and it can be traced to
    the poor versioning on the MODIS processing system end. There would
    be no problem if the MODIS processing team acquired their Metadata
    Configuration Files (MCFs) from installed descriptors at the DAACs.
    In reality, they modify the MCF locally and then send the changes to
    ECS. As a result, there can be mismatches between the DAACs
    installed ESDT and what MODIS is using. This problem has all but
    disappeared since the MODIS processing team is now using only the
    official MCFs.

### 3.1.3 AQUA {#aqua .unnumbered}

AQUA is a NASA Earth Science satellite mission mainly designed to study
Earth's water cycle. AQUA was formerly named EOS PM, signifying its
afternoon equatorial crossing time, as opposed to the morning equatorial
crossing time for TERRA. Aqua will carry six instruments in a
near-polar, low-Earth orbit. The six instruments are the Atmospheric
Infrared Sounder (AIRS), the Advanced Microwave Sounding Unit (AMSU-A),
the Humidity Sounder of Brazil (HSB), the Advanced Microwave Scanning
Radiometer for EOS (AMSR-E), the Moderate-Resolution Imaging
Spectroradiometer (MODIS), and the Clouds and the Earth\'s Radiant
Energy System (CERES). The MODIS and CERES instruments are the same as
those onboard TERRA launched in 2000. The AQUA mission launched in May
2002.

The data format and metadata standards for the AQUA instrument data are
the same as those for TERRA, namely the HDF-EOS and the ECS data model,
respectively. Lessons learned from the AIRS instrument team (Evan
Manning, AIRS principal developer) and the AMSR-E instrument team (Dawn
Conway, University of Alabama in Huntsville, Lead Software Engineer for
the AMSR-E Science Team) on implementing the data and metadata standards
are summarized below.

1.  In general, using the HDF-EOS standards requires a fair amount of
    "buy-in" and has a steep learning curve. Instrument team developers
    adapted, but casual users had more trouble. For example, it was
    relatively easy for an instrument programmer to produce the HDF-EOS
    files using the simple API. A lot of end-users, however, are
    reluctant to accept or "buy into" HDF-EOS because it is new. Both
    the AIRS and the AMSR-E teams found that HDF-EOS is very easy to
    use.

2.  The HDF-EOS format has adequately supported AIRS and AMSR-E
    requirements, but:

-   The HDF-EOS should explicitly support field annotations. Without a
    standard, some developers will add their own annotation to internal
    HDF objects.

-   The field/attribute distinction is not clear. It seems that a swath
    attribute is anything that does not have a dimension that is a
    geolocation dimension. HDF-EOS Swath thinks it\'s anything with less
    than 2 dimensions.

3.  The documentation for the HDF-EOS is clearly inadequate. It could
    really use some good sample programs. For example, provide examples
    that actually do something non-trivial, such as check for error
    conditions.

4.  While AMSR-E Lead Science Computing Facility (SCF) found that
    implementation of the required ECS metadata was simple and
    straightforward; the AIRS team encountered several problems
    implementing the ECS data model. In fact, the AMSR-E team found the
    Science Data Processing (SDP) toolkit unnecessary to complete their
    tasks. It was noted, however, that the ECS keywords should better
    relate to keywords used in the GCMD (Global Change Master
    Directory). Problems that the AIRS team encountered are:

-   The ECS tools for implementing the ECS metadata standards are not
    easy to use. There are some really tricky parts, like setting
    "hdfattrname" to \"coremetadata.0\" or \"coremetadata\" depending on
    whether it is embedded metadata or not. The interface is generally
    confusing.

-   The amount of lead-time for adding an ECS Product Specific Attribute
    or changing attribute valids, etc. is too long.

-   Documentation for the ECS data model is not adequate.

-   The AIRS team supported ESDIS's (led by Bob Lutz) attempts to add
    new valids for ScienceQualityFlag. The failure of those attempts
    makes it hard for AIRS to support data access as they would prefer
    to.

5.  On a general development note, both teams discovered the importance
    of regular, consistent communications (telecons, meetings, etc.)
    between the SCF, SIPS (Science Investigator-lead Processing System),
    DAAC, and ECS.

### 3.1.4 AURA {#aura .unnumbered}

Aura is a NASA mission to study the Earth\'s ozone, air quality, and
climate. This mission is designed exclusively to conduct research on the
composition, chemistry, and dynamics of the Earth\'s upper and lower
atmosphere by employing multiple instruments on a single satellite.
Aura\'s chemistry measurements will follow-up on measurements that began
with NASA\'s UARS and will continue the record of satellite ozone data
collected from the TOMS (Total Ozone Mapping Spectrometer) missions. The
satellite will be launched in January 2004 and will operate for five or
more years. The Aura data products will be distributed in HDF-EOS5
format. Aura metadata will conform to the ECS data model.

The HDF file format was designed to be a very flexible format. It is
able to store many different types of scientific data in a variety of
ways. While this flexibility is an asset to customized data storage, it
is not ideal when one is trying to ease sharing of data. As there is so
much flexibility, two different developers storing the exact same data
can store the data in dramatically different ways. To constrain HDF for
use in the EOS community, HDF-EOS was developed.

While HDF-EOS constrains HDF with its POINT, GRID, and SWATH interfaces,
it is still possible to create two files that are completely different
and require dramatically different readers. Areas of potential mismatch
include:

-   Organization of data fields and attributes

-   Dimension names

-   Geolocation names and dimension ordering

-   Data field names and dimension ordering

-   Units for data fields

-   Attribute names, values, and units

When the Aura Data System Working Group (DSWG) reviewed the proposed
structure of the Level 2 data files from each instrument, it was
discovered that each instrument\'s data files were, at times, quite
different. DSWG agreed that with a little work, it was possible to adopt
a uniform set of file format guidelines and that it was advantageous to
do so. One of the main advantages of this standard is to allow users the
ability to use the same set of tools and I/O routines for any of the
Level 2 data from instruments within Aura. At the time of this writing,
the \"HDF-EOS Aura File Format Guidelines\" has been adopted by all of
the EOS Aura instrument teams. The guidelines contain detailed, specific
information on how to store data. All of the items listed above are
specifically addressed. As the launch of Aura has not yet occurred at
the time of this writing, the outcome of this endeavor has not been
determined, but it is hopeful that by adopting a uniform set of strict
guidelines that the benefits will be many. The current guidelines can be
found at:

<http://www.eos.ucar.edu/hirdls/HDFEOS_Aura_File_Format_Guidelines.doc>
(Microsoft Word version)

<http://www.eos.ucar.edu/hirdls/HDFEOS_Aura_File_Format_Guidelines.pdf>
(Adobe Acrobat format)

### 3.1.5 QuikSCAT/SeaWinds {#quikscatseawinds .unnumbered}

The SeaWinds instrument on the QuickScat satellite is a specialized
microwave radar that measures near-surface wind speed and direction
under all weather conditions and cloud cover. It was launched in 1999 as
a follow-on mission to the NASA scatterometer (NSCAT) that flew on the
Japanese ADEOS-1 (Advanced Earth Observing Satellite) platform during
1996-1997; and the Seasat-A scatterometer system (SASS), which flew in
1978.

A unique feature of the QuikSCAT/SeaWinds mission is that SeaWinds data
are processed, archived, and distributed at both NASA JPL and NOAA
NESDIS. SeaWinds data are downloaded from QuikSCAT once every orbit (101
minutes). The stream passes on from the receiving ground station to the
Central Standard Autonomous File Server (C-SAFS) at Goddard Space Flight
Center (GSFC). The data are then forwarded to both JPL and NOAA. JPL
uses these data to produce its science-level wind product, while NOAA
uses an altered version of JPL's processing to produce its own [Near
Real Time (NRT) wind
product](http://www.soc.soton.ac.uk/JRD/MET/WGASF/workshop/PDF/32connor.doc.pdf).
This dichotomy can be summarized as follows:

-   While the processing software used at NASA JPL and at NOAA NESDIS is
    the same, data products produced at JPL are research products (with
    higher accuracy) used for research and in the application community,
    while data products from NOAA are near real-time products (within 3
    hours of observation) targeted for operational users such as the
    National Weather Services (NWS).

-   The SeaWinds products distributed by JPL are in HDF format while
    data products distributed by NOAA NESDIS are in BUFR format. This is
    because many operational and modeling users use the WMO (World
    Meteorological Organization) data standards, BUFR and GRiB (GRidded
    Binary). NOAA is required to provide data to their operational users
    in BUFR/GRiB format. For the future, the current plan is to move the
    NRT processing from NOAA to the Physical Oceanography (PO) DAAC at
    JPL, starting with the ADEOS-II mission in 2002.

### 3.1.6 ACRIM  {#acrim .unnumbered}

For ACRIM, using HDF-EOS was required; however, since mapping the
terrain of the Earth was not necessary (ACRIM is solar pointing), the
EOS part did not apply. ACRIM was actually using something akin to a
subset of HDF. Because ACRIM used HDF in a limited fashion, enough tools
were available, but it still required the team to learn almost
everything about HDF in order to determine what functions they actually
needed. Overall, HDF was relatively easy to implement. Some lessons
learned indicate that the following would have been helpful in the
implementation of HDF:

-   An instruction manual -- "What would have been helpful is a manual
    with step-by-step instructions; it could have been a quicker
    implementation."

-   Help desk -- "Having someone who could spend a little time over the
    phone would have been very helpful."

-   Rectifying the problems with creating HDF files with REAL and
    INTEGER values.

\[Frank Boecherer, ACRIM Science Computing Facility, Personal
Communication, June 2002\]

### 3.1.7 SeaWiFS  {#seawifs .unnumbered}

Ten years ago, when SeaWiFS was in development, HDF had some
capabilities that were not supported at that time. In the beginning, HDF
was largely an image format; it only supported a limited number of data
sets, and it had floating point numbers only. The SeaWiFS team
identified these deficiencies early on; documented and issued reports;
then received responses from National Center for Supercomputing
Applications (NCSA). As a result, HDF was made more friendly and easier
to use. In addition, the parallel development of HDF for use with IDL
allowed users to write their own HDF tools. The main thing that was
learned through the experience of implementing HDF into the SeaWiFS
project was that good user support is essential. The group at NCSA
responded to all of their needs at the time. "That was the thing that
made it work -- user support, help desk." \[Fred Patt, SAIC Project
Manager, Personal Communication, June 2002\]

SeaDAS (SeaWiFS Data Analysis System) is a comprehensive image analysis
package for the processing, display, analysis, and quality control of
all SeaWiFS data products, ADEOS / OCTS (Advanced Earth Observing
Satellite / Ocean Color and Temperature Scanner, Japan), MOS (Modular
Optoelectronic Scanner, Germany), CZCS (Coastal Zone Color Scanner,
NASA), and Ancillary data (Meteorological, Ozone). HDF facilitated the
development of this powerful tool. The versatility of HDF also allows
individuals to develop their own uses within the SeaDAS system. HDF was
mandated for the SeaWiFS project because EOS was still under
development, and SeaWiFS was to pave the way for future missions. One
lesson learned is: allow time to develop tools (or preferably use
existing tools) to facilitate ease of use. \[Jim Acker, DAAC User
Support, Personal Communication, June 2002\]

### 3.1.8 Jason-1  {#jason-1 .unnumbered}

Jason-1 chose to use its own ad-hoc, format (binary) as the primary data
product for historical reasons (continuity). The main advantage of using
binary is that it is fast and simple. Once given the read program, it is
self-contained. A disadvantage to binary is that each data set requires
its own read program.

Initially, one of the problems with HDF was that software for reading
the format was not widely available, and it did not work on many
important computer classes. A second problem, in the past, was that
installing the HDF libraries required major system administration
knowledge. Also, the initial jump into HDF is difficult and requires a
lot of "handholding", but only for first-time users. However, the beauty
of HDF is uniformity across mission data sets.

From these ideas, the main lessons drawn are:

-   Before declaring a format \"STD\", make sure its supporting software
    installs properly and runs on the main machines intended.

-   Understand which classes of users will be EXCLUDED by the new format
    (for example, the simple binary format of Topex can be read on even
    a windows 95 computer, but HDF will not install there). It is
    acceptable to exclude classes of users CONSCIOUSLY, but not because
    of oversight.

-   Do not underestimate the "handholding" that will be needed to help
    users install, then run, the new software. HDF, etc. are not \'read
    programs,' they compare to major operating systems or major
    commercial packages (IDL, Matlab, Mathematica, etc) in their
    complexity and their installation can be as complex.

\[Victor Zlotnicki, Jet Propulsion Laboratory, Personal Communication,
June 2002\]

### 3.1.9 AVHRR  {#avhrr .unnumbered}

AVHRR data format was based on TIROS data for continuity (level 1B,
native binary). However, about 2-years ago, NOAA began offering AMSU
data in HDF-EOS along with the BUFR and 1B products. The response to
HDF-EOS was great. Almost all of the climate scientists are now using
the HDF-EOS format by their own choice. In the future, NOAA hopes to
offer AVHRR as an HDF-EOS product, due to customer demand. \[Ingrid
Guch, National Environmental Satellite, Data and Information Services
(NESDIS), Personal Communication, June 2002\]

The HDF format has already been chosen for the reprocessing of all AVHRR
data for JPL. It was known that the data files would need to be
compressed, but the problem was, if just a small part of a big data set
was needed, the entire file would have to be decompressed and then the
small subset would have to be extracted. With HDF, a chunking process
exists (also called tiling). This compresses the data in such a way that
it allows storage of data sets in chunks that can be decompressed
separately. Thus, HDF-4 was chosen for the reprocessing of the AVHRR
data. \[Peter Cornillon, University of Rhode Island, Oceanography
Department, Personal Communication, June 2002\]

## Lessons Learned on Implementing and Using other Standards

### 3.2.1 NOAA Standards {#noaa-standards .unnumbered}

The National Oceanic and Atmospheric Administration\'s (NOAA\'s)
National Environmental Satellite, Data, and Information Service (NESDIS)
operates NOAA's environmental (weather) satellites and manages the
processing and distribution of the data and images these satellites
produce daily. NOAA's operational weather satellite system is composed
of two types of satellites: Geostationary Operational Environmental
Satellites (GOES) for \"now-casting\" and short-range warning and
Polar-Orbiting Environmental Satellites (POES) for longer-term
forecasting. Both types of satellites are necessary for providing a
complete global weather monitoring system. The primary customer is
NOAA\'s National Weather Service (NWS), which uses satellite data to
create forecasts for the public, television, radio, and weather advisory
services.

NOAA NESDIS does not use consistent data and metadata formats for their
POES and GOES satellite data archive and distribution. The POES and GOES
data are processed by the Information Processing Division (IPD) of the
NESDIS Office of Satellite Data Processing and Distribution
([OSDPD](http://psbsgi1.nesdis.noaa.gov:8080/)). The
[IPD](http://www.osdpd.noaa.gov/OSDPD/OSDPD.html) is responsible for
ingest, processing, and dissemination of environmental satellite data.
The GOES data are distributed in McIDAS formats. The POES weather and
climate data products are distributed in various different data formats
including flat binary file, Level 1b, GIF, ASCII, BUFR, GRiB, HDF-EOS,
netCDF, and McIDAS \[[[]{#d3_2_1 .anchor}1](#r3_2_1)\].

-   In general, NOAA NESDIS uses multiple distribution data formats to
    satisfy different user communities' needs \[Ingrid Guch, NOAA
    NESDIS, personal communication\]. The National Weather Service or
    the modeling community (US and international) uses the WMO data
    standards, BUFR and GRiB. These users have been relying on NOAA to
    format the data in BUFR and GRiB (as opposed to them taking the data
    and running their own converter). The BUFR/GRiB formats are very
    complex, though, and not generally used by the people outside the
    modeling community.

-   The imaging, climate, and scientific community as well as the NOAA
    NESDIS maintenance personnel greatly prefer the HDF-EOS data (ease
    in visualization, combining datasets, using commercial software,
    etc.). The netCDF format has the same benefit.

-   Other experienced users (education, academic, etc.) seem to prefer a
    binary or ASCII flat file so they can easily manipulate it and add
    GIS or whatever extensions they like.

-   Browsing users (education, some academic folks, etc.) prefer the
    option of ASCII, spreadsheet, and GIF.

-   For satellite data (sensor counts with navigation and calibration
    appended but not applied), users seem satisfied with the current
    packed binary file (Level 1b format). The internal NESDIS
    maintenance personnel have been using an unpacked binary file (Level
    1b star) for ease of use in real-time processing. However, this
    requires recreation of the \"unpacked\" file from archived metadata
    and the 1b if reprocessing is necessary (problems occurred in the
    real-time processing).

Long-term environmental satellite data products are archived and
distributed at the NOAA National Climatic Data Center
([NCDC](http://lwf.ncdc.noaa.gov/oa/ncdc.html)). Archive formats used in
NCDC are different for different data products. Many products are
archived in a custom format and others are in HDF-EOS, Level 1b, ASCII,
or JPEG \[Kathy Kidwell, NOAA NCDC, personal communication, 2002\]. Data
distribution formats are the same as the archive formats in NCDC.
Lessons learned on NOAA data standards are summarized below:

-   Since NOAA is an operational agency and its main customer is the
    NWS, NOAA NESDIS is required to distribute their satellite data in
    BUFR/GRiB format to the NWS or the modeling users, although there
    are many problems with the BUFR/GRiB format \[Ingrid Guch, NOAA
    NESDIS, personal communication; 2002\].

-   NOAA NCDC has many legacy systems and they have problems translating
    data to/from BUFR/GRiB format \[Geoffery Goodrum, NOAA NCDC,
    personal communication, 2002\].

-   The NOAA NESDIS staff have had a positive experience with the
    HDF-EOS data format \[[[]{#d3_2_2 .anchor}2](#r3_2_2)\] and their
    users, mainly imaging, climate, and scientific communities, like the
    HDF-EOS format because of the flexibility, tools, and vendor support
    \[[[]{#d3_2_3 .anchor}3](#r3_2_3)\].

### 3.2.2 The Spatial Data Transfer Standard (SDTS) {#the-spatial-data-transfer-standard-sdts .unnumbered}

The Spatial Data Transfer Standard became a Federal Information
Processing Standard (FIPS 173) in 1992, after a 10-year development
effort. It was to serve as the national spatial data transfer mechanism
for all U. S. Federal agencies, and to be available for use by state and
local government entities, the private sector, and research
organizations. SDTS specifies exchange format constructs, addressing
structure, and content, for spatially-referenced vector and raster data,
to facilitate data transfer between dissimilar spatial database
systems.\[[[]{#d3_2_4 .anchor}4](#r3_2_4)\] The Spatial Data Transfer
Standard (SDTS) doesn't prescribe a single data model; rather it
provides a set of rules intended to represent virtually any data model.

However, SDTS fell short of its ambitious goals; and the marketplace was
slow to accept and support it. Arctur *et al.* \[[[]{#d3_2_5
.anchor}5](#r3_2_5)\] list a number of reasons for this:

> *Complexity* - SDTS was driven primarily by large national-level data
> producers and their needs (very large databases, complex
> interdependencies, high precision, flexible models, extensive
> metadata, collaborative updates, etc.). These needs far exceeded those
> of casual "desktop GIS" users and of most commercial, regional, or
> local GIS projects, and they stretch even today's GIS technology to
> its limits. Many people in the GIS community found SDTS to be overly
> complex, few understood its intended purpose, and thus few chose it
> when other, more established formats were available.\[[[]{#d3_2_6
> .anchor}6](#r3_2_6)\] (Arctur *et al.* \[[5](#r3_2_5)\] suggest that
> as GIS users become more sophisticated, they may demand more of their
> technology (including data models and formats), and be more able and
> willing to cope with the implied complexity.)
>
> *Slow development of the standard in a fast-changing market* - In the
> decade that elapsed between the first work on SDTS and its final
> adoption as a standard, the GIS industry grew significantly, and
> several vendor-specific exchange formats came into widespread use,
> which satisfied many users' immediate needs, and thus limited the
> community's interest in using SDTS (which many perceived as yet
> another format). Even though the standard was mandated for all federal
> agencies, most data suppliers, responding to user demand, offered
> alternative data encodings -- and only the most curious and
> experimental users chose SDTS.
>
> *Limited vendor support -* SDTS got caught in a "chicken-and-egg"
> situation with GIS vendors: in order to build market demand for
> SDTS-aware software, data providers needed to produce large volumes of
> SDTS data. But they needed to use commercial GIS products to build
> these data; so they had to persuade vendors to produce SDTS products
> in the absence of customer demand. A few vendors did include STDS
> conversion tools in their products (*e.g.,* ESRI's Arc/Info,
> Laser-Scan's Gothic); however different products interpreted SDTS
> ambiguities differently (see below), so they would often fail to
> translate unexpected STDS constructs introduced by another vendor's
> product.
>
> *Slow development of practical profile*s - SDTS was a very general
> standard: any practical use of it required users to agree on a
> particular profile. But due to the complexity of SDTS, and the limited
> educational material (such as usage examples) available to the
> geospatial community, it took another four years to complete the first
> usable profile of SDTS (the Topological Vector Profile). The lack of
> interest in, and understanding of, SDTS among the GIS community also
> reduced the demand for useful profiles, and the community's enthusiasm
> for working on them. In the end, this first profile proved to be both
> limiting (encoding fairly mundane examples required awkward
> workarounds) and unnecessarily complex (it required arc/node/polygon
> topology, which was unnecessary or even meaningless for many
> commonly-used cases). \[[[]{#d3_2_7 .anchor}7](#r3_2_7)\]
>
> *Harmonization delays -* Subsequent efforts to define other SDTS
> profiles (the Raster Profile and Transportation Network Profile) were
> almost complete when they became mired in attempts to harmonize them
> with similar standards being developed in NIMA, NATO, and the European
> Union. This resulted in further delays to their development. (Arctur
> *et al.* \[[5](#r3_2_5)\] suggest that early harmonization is easier,
> and that profiles should not be developed so quickly as to overlook
> other, related standards.)
>
> *Ambiguity* in the data model (*e.g.*, the cardinality of
> relationships) and the data semantics (*e.g.*, the meaning of
> relationships among entities) of SDTS and its profiles limited the
> utility of SDTS for reliable information transfer. (Arctur
> \[[[]{#d3_2_8 .anchor}8](#r3_2_8)\] likens an SDTS profile to a game
> in which teams agree on the size of the ball and the shape of the
> field, but not on the rules of play.) SDTS was supposed to be very
> general, and to make datasets *self-describing*; that is, the data
> model could be determined from the dataset contents. But this proved
> an elusive goal; and thus many even of those who were willing to be
> "SDTS pioneers" ultimately concluded that its practical value was
> limited.

In addition, during and after the development of SDTS, new,
unanticipated technical expectations arose, which demanded significant
technical (re)design and international coordination, and further
weakened the community's support for SDTS:

-   a standard means of representing subtiles within a dataset;

-   support for permanent, universally unique object identifiers across
    all datasets;

-   support for value-added extensions and incremental updates by users;

-   support for tracking changes and historical lineage of features and
    spatial primitives;

-   harmonizing the metadata content with emerging international
    standards; and

-   harmonizing repository organization with emerging OpenGIS software
    interfaces.

Some of these issues might have been anticipated in the design of SDTS,
while others stemmed from the increasing sophistication of GIS products
and their users over the years.

The need for harmonization with OpenGIS led to OpenGIS' work on
interface specifications for access to geospatial data (features,
coverages, identifiers, etc.). Since the late 1990s, OpenGIS has been
the locus of much subsequent work in this area. It focused first on
accessing geospatial data (*e.g.*, Simple Features Access for SQL, COM,
and CORBA), then on encoding geospatial features in XML (Geography
Markup Language (GML)) for transfer between clients and servers.

In summary, the SDTS experience illustrates the importance of keeping
pace with technology and market trends and emerging expectations, even
after capturing initial requirements. It shows the role of timing: a
standard may be "ahead of its time" (arriving before people are ready to
understand them or accept more complexity) or "overcome by events"
(arriving after people are used to making do without flexible, general,
or vendor-independent solutions). Paradoxically perhaps, SDTS was both!

The SDTS experience also underscores the need to balance advanced needs
with more basic ones; the importance of good documentation and usage
examples; the challenge of "priming the pump" among vendors in advance
of market demand; the benefits and risks of harmonizing with related
standards; and the futility of mandating a standard that fails to meet a
need.

**[References:]{.underline}**

\[[[]{#r3_2_1 .anchor}1](#d3_2_1)\] NESDIS Satellite Product Overview
Display,
<http://osdacces.nesdis.noaa.gov:8081/satprod/products/prod_frameset.cfm?prodid=-1>

\[[[]{#r3_2_2 .anchor}2](#d3_2_2)\] Huan Meng, Doug Moor, Limin Zhao,
Ralph Ferraro, HDF-EOS at NOAA/NESDIS; Presentation; HDF-EOS Workshop,
2000, Landover, MD:
<http://hdfeos.gsfc.nasa.gov/hdfeos/WSfour/meng/hdfeos4.ppt>

\[[[]{#r3_2_3 .anchor}3](#d3_2_3)\] Andrew S. Jones and Thomas H. Vonder
Haar; A Dynamic Parallel Data-Computing Environment for Cross-sensor
Satellite Data Merger and Scientific Analysis; Accepted for publication
by Journal of Atmospheric and Oceanic Technology; March 2002.

\[[[]{#r3_2_4 .anchor}4](#d3_2_4)\] Fegeas, Robin (1995). Q3.3: What is
this SDTS thing and is it available via ftp? In *GIS-L /
comp.infosystems.gis Frequently Asked Questions and General Info List*
(Lisa Nyman, ed.). <http://www.prenhall.com/startgis/faq.html>.

\[[[]{#r3_2_5 .anchor}5](#d3_2_5)\] Arctur, D., Hair, D., Timson, G.,
Martin, E., and Fegeas, R. (1998) Issues and Prospects for the Next
Generation of the Spatial Data Transfer Standard (SDTS). *International
Journal of Geographical Information Science* 12(4): 403-425.

\[[[]{#r3_2_6 .anchor}6](#d3_2_6)\] Hastings et al. (1996) *The Spatial
Data Transfer Standard: Closing the Loop?* - Panel Discussion at
GIS/LIS\--Denver, Colorado\--November 19, 1996.
<http://www.ngdc.noaa.gov/seg/tools/sdts/gislis_main.html>

\[[[]{#r3_2_7 .anchor}7](#d3_2_7)\] Kelley, C., and Gosinski, T., 1994:
Spatial Data Transfer Standard: do you fit the profile? *GIS World*,
August 1994, pp. 48-50.
<http://wwwsgi.ursus.maine.edu/gisweb/spatdb/urisa/ur94076.html>

\[[[]{#r3_2_8 .anchor}8](#d3_2_8)\] Arctur, David K., 1996: Spatial Data
Transfer Standard: A GIS Vendor\'s Perspective. Online article.
<http://web.archive.org/web/19990222153931/www.lsl.co.uk/~arctur/portfolio/sdts.html>

# Essential Standards Concepts

Before evaluating individual data or metadata standards, it may be
useful to review several key concepts crucial to understanding and
comparing standards.

-   A comparison with private, ad-hoc, binary information transfer

```{=html}
<!-- -->
```
-   Mandatory vs. optional elements of a standard; profiles and
    extensions

-   Abstract vs. implementation standards

-   Content and format standards vs. behavior and interface standards

## A Comparison with Private, Ad-hoc, Binary Information Transfer

Webster's dictionary defines a standard as: "That which is established
as a rule or model by authority, custom, or general consent." Thus,
standards exist only within a community of people sharing certain usage
patterns ("custom") or organizational structures (formal "authority" or
informal "general consent"). The emphasis in this study is on standards
accepted by a fairly broad set of users, publicly documented, and either
stable (unchanging) or changeable only by a consensus among these users.

Another aspect of standards is that they govern only a part of the
information transfer process. For instance, GeoTIFF codifies the
georeferencing of an image, but is silent on the meaning of its pixel
values. Whatever a standard does not specify is left to the private
(often implicit) understanding of each user community or to ad-hoc
ancillary information (such as a README file or a telephone message
describing data details). So, at one extreme, complex and rigid
standards specify every aspect of information transfer, and at the other
extreme, private agreements or ad-hoc communications leave everything
implicit or unstructured. Most standards fall somewhere in between; they
govern a certain piece of the information transfer process to let a
certain set of users communicate or work together, but users must also
rely on other standards, private agreements, or ad-hoc qualifiers.

Many earth science data users favor a \"raw binary\" data format that is
both simple and comprehensive. In fact, "raw binary" doesn\'t actually
mean mysterious data files that one must guess at -- but, rather, a
simple format (often some kind of raster grid) used by a small set of
colleagues with little attention to documentation or stability. Thus,
"raw binary" denotes not a single format, but as many different formats
as there are workgroups. Each such format has a syntax and semantics
invented just for that data set or data series, usually without a lot of
attention to other related formats, and with many details left implicit
or provided separately in a mission report or some other
natural-language document.

Such a format may serve many people\'s immediate needs, for several
reasons.

-   A given science team often works with only one kind of data and so
    gets used to the one syntax for that data (*e.g.*, keeps reusing the
    same parser) and the one set of semantics (*e.g*., pins a page from
    the mission report to the cubicle wall).

-   Science teams traditionally put a low emphasis on making their data
    accessible to others outside of their immediate colleagues. They may
    feel that they have done their job by distributing a simple
    \"README\" file with the data.

-   ESE data is commonly encoded as raster grids for which one can make
    easy guesses as to syntax (band-sequential, etc.).

-   Most importantly, perhaps, the semantics of much ESE data (platform
    orientation, sensor model, calibration information, interpretation
    algorithms) are so complex that bundling them with the data is often
    difficult; so they tend to remain in a mission report document -- or
    in people\'s heads. (For example, each MODIS L1b granule has dozens
    of ancillary data items required for proper interpretation along
    with several grids of data error and reliability estimates.)

However, the use of raw binary data relies much more on private
agreements among colleagues than on documented, consensus standards. It
has many of the properties opposite to those of standards, as listed
above in the \"Rationale for Standards\" paragraph. It limits the
ability of science teams to move beyond traditional work methods towards
more effective interdisciplinary research, collaborative work, and
applications. The essential points are:

-   Data in a standard formats (should) convey something about the data
    that its users need to know; whereas, users of binary data must rely
    on inside knowledge or educated guesses to read and interpret the
    data.

-   Data in a standard format may be used outside of an \"inner circle\"
    of colleagues, but only holders of the necessary private information
    can use raw binary data.

-   Standard data formats limit the need for pair-wise translators to
    and from every possible format, whereas, each raw binary format
    needs a different translator.

-   Standard data formats, by fixing the syntax and semantics of
    information, allow the possibility of machine-to-machine
    communication between different systems (that is, interoperability).
    In contrast, raw binary data requires human inspection an
    intervention, thus, hindering (preventing) system interoperability.

-   Standard data formats facilitate unambiguous transfer of information
    between users of different systems working with different datasets.
    This is more difficult with raw binary data, which often loses all
    but raw pixel values in translation.

In summary, the use of private agreements does not constitute a
standard, and so "raw binary" data formats cannot be compared alongside
open consensus standards.

Of course, members of a community may choose to turn a private, internal
convention into a standard for a wider community by documenting and
publishing their shared syntax and semantics and by sticking to what
they document (that is, submitting any changes to a public consensus
process or formal authority). Most standards are born this way when a
usage community publishes its internal conventions to facilitate
collaboration with others.

## Mandatory vs. Optional Elements, Profiles and Extensions 

Many standards have a set of mandatory elements to ensure basic
interoperability plus a set of optional elements to serve a diversity of
users and uses. This provides a "base" standard from which a particular
community of users may define a *profile* (a more specific standard) to
support richer communication among themselves, or more fine-grained
control of each other's services. A profile is a standard derived from a
base standard by adding restrictions: it may require (or exclude) an
element that is optional in the base standard; it may limit the valid
entries under a heading; it may fix the cardinality of a repeating
element; and so on. But, the profile cannot contradict the base
standard; anything mandatory in the base standard remains mandatory in
the profile. Thus, any product that complies with the profile will
comply with the base standard. \[[[]{#d4_1_1 .anchor}1](#r4_1_1),
[[]{#d4_1_2 .anchor}2](#r4_1_2)\]

One profile presented here is HDF-EOS, an EOS-specific adaptation of the
very general Hierarchical Data Format. Of the many different file
structures that are possible with HDF, HDF-EOS defines three (Point,
Grid, and Swath), each with spatial and temporal details alongside
scientific data. In another example, FGDC's Metadata Content Standard
has allowed several community-specific profiles to be defined, and in
fact, the International Organization for Standardization's (ISO)
Metadata standard was designed primarily for profiles. It defines
several hundred elements of which fewer than 20 are required; the
remaining elements are shared vocabulary (*i.e.,* a dictionary) for
building profiles. \[[[]{#d4_1_3 .anchor}3](#r4_1_3)\]

Related to profiles is the notion of *extensions*. These are elements
added to the base standard by consensus among a certain community of
users. As with profiles, extensions do not contradict the base standard
-- what's mandatory remains mandatory; products that fit the extended
standard have everything needed by the base standard, and more.
Nonetheless, by adding more loosely controlled, loosely defined elements
to a standard, extensions may complicate the interoperability and
maintenance of the standard.

For example, the earth imagery user community has extended the FGDC
metadata content standard to more fully describe remotely sensed data by
adding metadata elements such as the sensor model and the orbital
platform, both of which the base standard doesn't provide \[[[]{#d4_1_4
.anchor}4](#r4_1_4)\].

## Abstract vs. Implementation Standards

Standards and specifications for information systems are defined
primarily at two different "levels of abstraction;" implementation
specifications and abstract models \[[[]{#d4_1_5 .anchor}5](#r4_1_5)\].

-   *Implementation specifications* tell software developers *how* to
    express information or requests within particular distributed
    computing environments (such as XML, Java, or the World Wide Web).
    Such standards define data formats, access protocols, object models,
    naming conventions, etc., in terms that are directly usable within
    the targeted computing environment.

    -   *Implementation specifications* are the more immediately useful
        standards when they apply to one's chosen computing context. The
        data-format standards are implementation specifications, as are
        the eXtensible Markup Language (XML) encodings of FGDC, ISO, and
        other metadata standards.

-   *Abstract models* specify *what* information or requests are valid
    or required in principle, irrespective of individual computing
    environments. They define the essential concepts, vocabulary, and
    generic structure (type hierarchy) of computational services and
    information transfer. Although not directly usable to build data or
    software, these models set the stage for creating implementation
    specifications and for extending existing ones to new environments.

    -   *Abstract models* provide well-known semantics that can support
        interoperability through translators or cross-reference tables.
        For instance, thanks to FGDC's Content standard, Z39.50's GEO
        profile can "normalize" any FGDC compliant metadata (regardless
        of actual record formats or field names) for external access --
        that is, map its internal data elements to the GEO field names
        for external access.

    -   In general, consensus-based abstract models of data are often
        termed "content standards." They define the information elements
        and their intended meaning (semantics) independently of their
        syntax -- that is, independent of how these elements may be
        encoded in files on disk or along a communications link. In
        principle, content standards allow different parties to
        communicate meaningfully by mapping their data element names to
        those of the content standard even when they use different
        formats for their data. This works well for fairly simple data
        structures such as the "parameter=value" pairs of many metadata
        files and catalog records. However, with more complex syntax or
        semantics, translating the abstract concepts of the content
        standard into the terms of a particular format often becomes an
        interpretation task requiring judgment calls, assumptions, and
        ambiguity. So in practice, content standards alone may not
        suffice for transferring complex data between different user
        communities without information loss or distortion.

## Content and Format vs. Behavior and Interface 

Table 4.4.1 shows that at each level of abstraction certain standards
define the *interfaces* that allow different systems to work together or
the expected *behavior* of software systems. This is the computation
viewpoint, whose accent is on invoking services effectively and
unambiguously. Other standards define the *content* of geospatial
information or its *encoding* (or packaging) for accurate transfer
between different processing systems. This is the information viewpoint,
which emphasizes efficient, lossless communication \[[5](#r4_1_5)\].

[]{#__RefHeading___Toc26754277 .anchor}Table 4.4.1 Viewpoints and Levels
of Abstraction

  ---------------------- ------------------------ ------------------------
                         **Service Invocation     **Information Transfer
                         (computation             (information
                         viewpoint)**             viewpoint)**

  Implementation\        Interface                Encoding (format)
  specifications                                  
  (\"how\")                                       

  Abstract models\       Behavior                 Content
  (\"what\")                                      
  ---------------------- ------------------------ ------------------------

For distributed computing, both of these viewpoints are crucial and
intertwined. For instance, information content isn't useful without
services to transmit and use it. Conversely, invoking a service
effectively requires that its underlying information be available and
its meaning clear. However, the two viewpoints are also separable: we
may agree on how to represent information regardless of what services
carry it; conversely, we may define how to invoke a service
independently of how we package the information needed or conveyed by
the service.

In a given context, either the computation view (implemented as
interfaces) or the information view (implemented as formats) may take
precedence. Tables 4.4.2 and 4.4.3 below show a few guidelines for
prioritizing standards definition or adoption in certain contexts. In
general, however, deciding which view to emphasize in a given setting is
not straightforward.

[]{#__RefHeading___Toc26754278 .anchor}Table 4.4.2 Criteria For Format
Standards

  ----------------------------------- -----------------------------------
  **Worry about a data format         **Don't worry about a data format
  standard when ...**                 standard when ...**

  Users of different formats need to  There's no reason for users of
  share or communicate data with each different formats ever to share
  other.                              information.

  Each user group (or each user) uses A user consensus already exists on
  a different format.                 one or a few non-proprietary data
                                      formats.

  Available formats fail to convey    A practical, reasonably simple data
  all the information needed for      format conveys all of the
  proper use. (Thus users have to     information users need.
  rely on implicit knowledge or       
  ad-hoc notes to use the data.)      
  ----------------------------------- -----------------------------------

[]{#__RefHeading___Toc26754279 .anchor}Table 4.4.3 Criteria For
Interface Standards

  ----------------------------------- -----------------------------------
  **Worry about a service interface   **Don't worry about a service
  standard when ...**                 interface standard (*i.e.*, rely on
                                      FTP/FedEx) when ...**

  Most users want the output of a few Most users need direct access to
  well-known processing operations,   raw data (as archived) for ad-hoc
  such as subsetting, filtering,      processing and analysis.
  transformations, etc.               

  The intended applications are       Most use of the data requires all
  streamed or interactive -- they     of it (full size and detail) to be
  only use parts of the available     present simultaneously.
  data at a given moment.             

  No one reasonably simple format     Users have not begun to map their
  will ever meet everyone's needs. (A workflow to online database
  service allows users to request the transactions or Web services.
  data they need in a format that     
  fits it.)                           
  ----------------------------------- -----------------------------------

Among the data standards reviewed in this report, GeoTIFF, Landsat Fast
Format, and BUFR/GRiB are clearly file format standards; they specify an
encoding and are silent on what access interface to use. HDF, HDF-EOS,
and netCDF provide a software library to facilitate reading and writing
data files, but they too are file format standards; they don't specify a
format-neutral interface to a service. Table 4.4.4 compares the data
models and software access libraries for a variety of data packaging
standards.

[]{#__RefHeading___Toc26754280 .anchor}Table 4.4.4 Data Models and
Software Access Libraries

+--------+---------------------+--------------------+-----------------+
| **Data | **Logical Model**   | **Physical Model** | **Software      |
| Fo     |                     |                    | Access          |
| rmat** |                     |                    | Libraries**     |
+========+=====================+====================+=================+
| *      | -   Disk format,    | -   XDR-based      | -   C, C++,     |
| *HDF** |     hierarchical,   |                    |     FORTRAN,    |
|        |     and similar to  | -   Storage layout |     Java        |
|        |     Unix file       |     is contiguous  |                 |
|        |     systems         |     (serial) or    |                 |
|        |                     |     chunked        |                 |
|        | ```{=html}          |     (direct        |                 |
|        | <!-- -->            |     access)        |                 |
|        | ```                 |                    |                 |
|        | -                   | -   Datasets       |                 |
|        |    Self-description |     consist of     |                 |
|        |     provided in     |     header         |                 |
|        |     global and      |     attributes &   |                 |
|        |     local           |     data           |                 |
|        |     (individual     |                    |                 |
|        |     objects)        | -   M              |                 |
|        |                     | achine-independent |                 |
|        |    attributesHeader |                    |                 |
|        |     describes disk  |                    |                 |
|        |     structure with  |                    |                 |
|        |     metadata &      |                    |                 |
|        |     pointers        |                    |                 |
|        |                     |                    |                 |
|        | -   Usable for      |                    |                 |
|        |     general         |                    |                 |
|        |     scientific data |                    |                 |
|        |     storage; HFD4   |                    |                 |
|        |     data model      |                    |                 |
|        |     contains:       |                    |                 |
|        |     arrays, tables, |                    |                 |
|        |     raster image    |                    |                 |
|        |     and text        |                    |                 |
|        |     objects. HDF5   |                    |                 |
|        |     data model has  |                    |                 |
|        |     HDF4-type       |                    |                 |
|        |     objects         |                    |                 |
|        |     imbedded within |                    |                 |
|        |     arrays and text |                    |                 |
|        |     attribute       |                    |                 |
|        |     objects.        |                    |                 |
|        |                     |                    |                 |
|        | -   Will support    |                    |                 |
|        |     extended        |                    |                 |
|        |     (multiple       |                    |                 |
|        |     machine) files  |                    |                 |
+--------+---------------------+--------------------+-----------------+
| **HDF  | -   HDF-based:      | -   (Same as HDF)  | -   C, C++,     |
| -EOS** |     Versions 4 and  |                    |     FORTRAN,    |
|        |     5               | -   XDR-based      |     Java        |
|        |                     |                    |                 |
|        | -   Provides        | -   Storage layout |                 |
|        |     standard for    |     is contiguous  |                 |
|        |     geolocation     |     (serial) or    |                 |
|        |     data map to     |     chunked        |                 |
|        |     science data .  |     (direct        |                 |
|        |                     |     access)        |                 |
|        | -   Point           |                    |                 |
|        |     Structure:      | -   Datasets       |                 |
|        |     model for       |     consist of     |                 |
|        |     sparce,         |     header         |                 |
|        |     randomly        |     attributes &   |                 |
|        |     geolocated data |     data           |                 |
|        |                     |                    |                 |
|        | -   Swath           | -   M              |                 |
|        |     Structure:      | achine-independent |                 |
|        |     model for data  |                    |                 |
|        |     best organized  | -   Disk format is |                 |
|        |     by time,        |     available to   |                 |
|        |     latitude or     |     user           |                 |
|        |     track parameter |                    |                 |
|        |                     |                    |                 |
|        | -   Grid Structure: |                    |                 |
|        |     model for data  |                    |                 |
|        |     organized       |                    |                 |
|        |     spatially and   |                    |                 |
|        |     projected.      |                    |                 |
+--------+---------------------+--------------------+-----------------+
| **ne   | -   Self-describing | -   XDR-based      | -   C, FORTRAN, |
| tCDF** |                     |                    |     Java, Perl, |
|        | -   Usable for      | -   Storage layout |     Python,     |
|        |     general         |     direct         |     Ruby.       |
|        |     scientific data |     access\--      |     Tcl/Tk      |
|        |     storage         |     indexed        |                 |
|        |                     |                    |                 |
|        |                     | -   Datasets       |                 |
|        |                     |     consist of     |                 |
|        |                     |     header & data  |                 |
|        |                     |                    |                 |
|        |                     | -   M              |                 |
|        |                     | achine-independent |                 |
|        |                     |                    |                 |
|        |                     | -   Disk format is |                 |
|        |                     |     hidden         |                 |
+--------+---------------------+--------------------+-----------------+
| **Geo  | -   TIFF-based,     | -   Storage layout | -   C, Perl,    |
| TIFF** |     with            |     allows random  |     Python,     |
|        |     geolocation     |     access to      |     Java        |
|        |     tags            |     pixels by      |                 |
|        |                     |     band, strip,   |                 |
|        | -   Raster image    |     or tile        |                 |
|        |     data only       |                    |                 |
|        |                     |                    |                 |
|        | -   Multiple images |                    |                 |
|        |     can be stored   |                    |                 |
|        |     in a single     |                    |                 |
|        |     file.           |                    |                 |
|        |                     |                    |                 |
|        | -   Version 2 will  |                    |                 |
|        |     support         |                    |                 |
|        |     extended files  |                    |                 |
+--------+---------------------+--------------------+-----------------+
| **     | -   Tailored to     | -   Storage layout | -   FORTRAN 77  |
| BUFR** |     atmospheric     |     is serial      |                 |
|        |     data -- point   |                    |                 |
|        |     data            | -   Dataset        |                 |
|        |                     |     consists of    |                 |
|        | -   Based on        |     header + data  |                 |
|        |     sequential,     |                    |                 |
|        |     tape format     |                    |                 |
+--------+---------------------+--------------------+-----------------+
| **     | -   Tailored to     | -   Storage layout | -               |
| GRiB** |     atmospheric     |     appears to be  |    Command-line |
|        |     data -- gridded |     serial --      |     translators |
|        |     data            |     "messages"     |     to ASCII or |
|        |                     |                    |     IEEE binary |
|        | -   Based on        | -   Dataset        |                 |
|        |     sequential,     |     consists of    |                 |
|        |     tape format     |     header + data  |                 |
+--------+---------------------+--------------------+-----------------+
| **Fast | -   Multi-band      | -   Separate       | -   Users write |
| Fo     |     image data      |     header and     |     their own   |
| rmat** |                     |     data files     |     software    |
|        |                     |                    |     based on    |
|        |                     | -   Direct access  |     examples    |
|        |                     |     to individual  |                 |
|        |                     |     bands          |                 |
+--------+---------------------+--------------------+-----------------+
| **Bi   | -   Data model      | -   Different for  | -   Custom      |
| nary** |     chosen by user. |     every product  |     software    |
|        |                     |                    |                 |
|        | -   Record, data    | -   Machine        | -   Users must  |
|        |     types           |     dependent      |     write their |
|        |     determined by   |                    |     own         |
|        |     specific        |                    |                 |
|        |     platform.       |                    |                 |
+--------+---------------------+--------------------+-----------------+

[See Acronym List if needed](#acronym-list)

## Web-based Data Service Standards 

The World Wide Web is driving rapid development of format-neutral
service interface standards. Examples particularly relevant to ESE data
include the OpenGIS Web Coverage Service \[[[]{#d4_1_6
.anchor}6](#r4_1_6)\] and Web Map Service \[[[]{#d4_1_7
.anchor}7](#r4_1_7)\] and the Distributed Oceanographic Data System
(DODS) \[[[]{#d4_1_8 .anchor}8](#r4_1_8)\].

The OpenGIS Consortium (OGC) Web Coverage Service (WCS) is likely to
become an OGC specification in early 2003. It will provide access to
images, imagery collections, and other systematic "fields" of values or
measurements -- usually arrayed on a 2D or 3D spatial grid. It fully
describes the data's spatial location and its semantic content and
allows clients to request subsets in space or along any of the data
dimensions using a syntax based on either Uniform Resource Locators
(URLs) or structured XML messages. The EOSDIS Core System (ECS) Synergy
effort intends to provide WCS access to its large online data holdings
("data pools"); and the GLOBE educational project ("global learning and
observations to benefit the environment:") has begun experimenting with
WCS and WMS (next).

The OGC Web Map Service (WMS) provides access to rendered maps and
pictures using a simple, spatial query syntax and common graphics
formats (PNG, JPEG, etc.). Since its inception in early 2000, this
interface has seen widespread implementation by many vendors,
laboratories, and open-source efforts.

The Distributed Oceanographic Data Service provides format-neutral
access to scientific datasets; its query syntax allows for "slicing" or
"sampling" a dataset along any of its variable values. DODS originated
at MIT and the University of Rhode Island (URI) in the mid-1990s; since
then, it has seen a fair bit of implementation in the oceanographic
community and among NASA DAACs. Recently, URI and NASA-DAACs have built
"gateways" from DODS to WMS and WCS; and URI has begun defining two
distinct successors to DODS: an "Open Source Project for a Network Data
Access Protocol" (OPeNDAP) (tools for generic infrastructure protocols)
and a "National Virtual Ocean Data System (NVODS)" (to supply
oceanographic data and applications). \[[[]{#d4_1_9
.anchor}9](#r4_1_9)\]

(Notable Web-based services in the ESE environment include the
University of Maryland's MOCHA project ("Middleware based on a
code-shipping architecture") \[[[]{#d4_1_10 .anchor}10](#r4_1_10)\]; the
Tropical Rainforest Information Center (TRFIC) at Michigan State
University \[[[]{#d4_1_11 .anchor}11](#r4_1_11)\]; EOS-Webster at the
University of New Hampshire \[[[]{#d4_1_12 .anchor}12](#r4_1_12)\]; and
many others. However, these are not service interface standards but,
rather, particular implementations of distributed systems. Although they
provide a useful benefit to their users, they are not linked by a
well-defined, published service interface standard; instead, they rely
on tightly coupled components or on unpublished or proprietary
interfaces.)

Finally, a number of vendors in the world of e-commerce have championed
the notion of "Web Services" \[[[]{#d4_1_13 .anchor}13](#r4_1_13)\]
consisting of the Web Services Description Language (WSDL)
\[[[]{#d4_1_14 .anchor}14](#r4_1_14)\]; Simple Object Access Protocol
(SOAP) \[[[]{#d4_1_15 .anchor}15](#r4_1_15)\]; and Universal
Description, Discovery, and Integration (UDDI) \[[[]{#d4_1_16
.anchor}16](#r4_1_16)\]. These industry specifications have gained broad
visibility and offer a lot of promise for Web-based data access;
however, the dust is far from settling on this very active area of
technology development.

In generally, the use of Web-based services is still only emerging in
practical ESE work. The primary mechanism for information interchange in
the ESE context remains the transfer of discrete files; it will take
some time before Web-based services become a part of mainstream data
access and distribution in ESE. Accordingly, this document treats format
and content standards only for the near-term missions.

**[References:]{.underline}**

\[[[]{#r4_1_1 .anchor}1](#d4_1_1)\] ISO TC211 (2001), "Geographic
Information -- Profiles"
<http://www.isotc211.org/protdoc/211n1134/211n1134.pdf>

\[[[]{#r4_1_2 .anchor}2](#d4_1_2)\] Federal Geographic Data Committee
(1998), Content Standard for Digital Geospatial Metadata:
<http://www.fgdc.gov/metadata/csdgm/> : Appendix D; Appendix E.

\[[[]{#r4_1_3 .anchor}3](#d4_1_3)\] Simon Cox (2001), Summary of some
geospatial metadata standards:
<http://www.ned.dem.csiro.au/research/visualisation/metadata/geospatial/>

\[[[]{#r4_1_4 .anchor}4](#d4_1_4)\] NASA (2001), Digital Earth Reference
Model v0.5: <http://www.digitalearth.gov/derm/v05/>

\[[[]{#r4_1_5 .anchor}5](#d4_1_5)\] FGDC (2001), Content Standard for
Digital Geospatial Metadata: Extensions for Remote Sensing Metadata:
<http://www.fgdc.gov/standards/status/csdgm_rs_ex.html>

\[[[]{#r4_1_6 .anchor}6](#d4_1_6)\] Evans, John D. (2001), OGC Web
Coverage Server (WCS), Discussion Paper #01-018:
<http://www.opengis.org/techno/discussions/01-018.pdf>

\[[[]{#r4_1_7 .anchor}7](#d4_1_7)\] de La Beaujardière, Jeff (2001), OGC
Web Map Server Interface Implementation Specification, version 1.1.1:
<http://www.opengis.org/techno/specs/01-068r3.pdf>

\[[[]{#r4_1_8 .anchor}8](#d4_1_8)\] Distributed Oceanographic Data
System (DODS): http://www.unidata.ucar.edu/packages/dods/

\[[[]{#r4_1_9 .anchor}9](#d4_1_9)\] Cornillon, Peter (2002), "DODS:
OPeNDAP providing plug-and-play interoperability in a distributed data
system," presentation at the 9th Assembly Meeting of the ESIP
Federation, University of Maryland, College Park, May 15-17, 2002.
<http://www.esipfed.org/business/library/meetings/9th_fed_meeting/ppt/DODS.PPT>

\[[[]{#r4_1_10 .anchor}10](#d4_1_10)\] The MOCHA project:
Self-Extensible Middleware Architecture.
<http://www.cs.umd.edu/projects/mocha/>

\[[[]{#r4_1_11 .anchor}11](#d4_1_11)\] [Tropical Rain Forest Information
Center (TRFIC)](http://www.bsrsi.msu.edu/trfic/).
<http://www.bsrsi.msu.edu/trfic/>

\[[[]{#r4_1_12 .anchor}12](#d4_1_12)\] EOS-WEBSTER: Earth Science Data
from the University of New Hampshire. <http://eos-webster.sr.unh.edu/>

\[[[]{#r4_1_13 .anchor}13](#d4_1_13)\] World Wide Web Consortium (W3C),
2002: Web Services home page. <http://www.w3.org/2002/ws/>

\[[[]{#r4_1_14 .anchor}14](#d4_1_14)\] World Wide Web Consortium (W3C),
2002: Web Services Description Language (WSDL) Version 1.2: W3C Working
Draft 9 July 2002. <http://www.w3.org/TR/2002/WD-wsdl12-20020709/>

\[[[]{#r4_1_15 .anchor}15](#d4_1_15)\] World Wide Web Consortium (W3C),
2002: SOAP Version 1.2 Part 1: Messaging Framework.
<http://www.w3.org/TR/soap12-part1/>.

\[[[]{#r4_1_16 .anchor}16](#d4_1_16)\] Universal Description, Discovery,
and Integration (UDDI) of Business for the Web. http://www.uddi.org

# Standards Evaluation

In order to objectively assess the data and metadata standards
identified in Chapter 2 for the SEEDS near-term missions, an analysis is
carried out to evaluate the standards according to many features or
criteria. Furthermore, a user opinion interview/survey is conducted to
gather user community's feedback on using the standards.

## Evaluation Criteria

Many features or criteria can be used to evaluate the data and metadata
standards identified. The intention of this study is not to identify one
all-purpose standard but, rather, to identify appropriate use of the
standards. For example, some standards are more suitable for
transmission and archiving while others for analysis. For transmission
and archiving, the most important features standards should have are
semantic completeness, portability, self-description, extensibility,
interoperability, etc \[[[]{#d5_1_1 .anchor}1](#r5_1_1)\]. For analysis,
standards should have features such as ease of use, analysis tools
support, etc. Many of these features and others are defined below.

**1. Interoperability** -- Tools exist to translate to other standard
formats with no information loss.

Is there a defined relationship or semantic equivalence between this
standard and other standards? *i.e.,* can the standard be broken into
elements that have the same content as elements for other standards?

Is the definition sufficiently precise to allow development of a
translation algorithm between standards?

What translation tools (well known) have been developed?

**2. Availability** -- Source code for writing and reading data in the
format is widely and publicly available.

Is the source code for writing and reading data widely and publicly
available?

Is the software for reading and writing well documented?

Are the search and order methods for data using the format well
understood and established?

**3. Portability** -- Data in this standard can be used on a variety of
platforms or in a variety of applications (vendor support).

Is the format sufficiently well defined so that data can be ported to
new commonly used platforms with minimal effort?

Is the format sufficiently well implemented that new applications can
access the implementation with minimal effort?

Can the standard be implemented on one platform and installed and tested
on other platforms with minimal modification of source code? *i.e.*,
machine dependent code is minimized.

**4. Evolvability** -- A clear process for maintaining and evolving the
standard exists.

Is there a methodology for adding new features to the standard?

Is there a software development process?

Is there a standard for documentation?

Is there an open process for evolution?

**5. Extensibility** -- Support for extensions and profiles exists.

Does the standard allow extensions or profiles to be developed?

Are there extensions or profiles developed for the standard?

**6. Self-describing** - Files contain data descriptions along with the
data.

Can data in this format be read without a separate document detailing
file contents?

Can the data be described internally to facilitate development of
applications?

Does the format contain information to allow geospatial, temporal,
and/or spectral subsetting?

**7. Tools Support** -- Software tools are available to support the
standard.

Does the standard have freeware support?

Does the standard have COTS (Commercial Off-The-Shelf) software support?

**8. Completeness --** The capacity to carry semantic descriptive
elements of the data explicitly and unambiguously. Higher levels of
completeness can reduce the user\'s dependency on outside information,
implicit knowledge, or guesswork when interpreting and applying the
data.

Can the format carry everything users need to use the data correctly?
*i.e.*, can the format convey the data\'s precise spatial location, its
units of measure, the observation parameters (*e.g.,* spectral bands),
accuracy estimates (error bars), and other elements needed to understand
the data and apply it?

## Data Standards Evaluation 

Using the standards evaluation criteria defined above, Tables 5.2.1
through 5.2.8 analyze and compare data standards in use in heritage
missions and other ESE missions.

[]{#__RefHeading___Toc26754281 .anchor}Table 5.2.1 Data Standards
Interoperability

+-----------+------------------+-----------------+--------------------+
| **Data    | **Evaluation     |                 |                    |
| S         | Questions**      |                 |                    |
| tandard** |                  |                 |                    |
+===========+==================+=================+====================+
|           | **Is there a     | **Can           | **What Translation |
|           | defined          | translation     | Tools (well known) |
|           | relationship or  | algorithms be   | developed?**       |
|           | semantic         | developed       |                    |
|           | equivalence      | easily?**       |                    |
|           | between the      |                 |                    |
|           | standard and     |                 |                    |
|           | other            |                 |                    |
|           | standards?**     |                 |                    |
+-----------+------------------+-----------------+--------------------+
| HDF       | Yes. Since HDF   | Yes, HDF has a  | GIF \<-\> HDF5     |
|           | can contain      | well-documented |                    |
|           | general          | software API.   | HDF4 \<-\> HDF5    |
|           | scientific data, |                 |                    |
|           | it encompasses   |                 | Ensight6 -\> HDF5  |
|           | all the other    |                 |                    |
|           | standards.       |                 |                    |
+-----------+------------------+-----------------+--------------------+
| HDF-EOS   | Yes. As a        | Yes, Point,     | GIF \<-\> HDF-EOS5 |
|           | superset of HDF, | Grid Swath      |                    |
|           | it also          | add-on          | HDF4 \<-\>         |
|           | encompasses the  | structures are  | HDF-EOS5           |
|           | other standards. | w               |                    |
|           |                  | ell-documented. | Ensight6 -\>       |
|           |                  |                 | HDF-EOS5           |
|           |                  |                 |                    |
|           |                  |                 | HDF-EOS -\>        |
|           |                  |                 | GeoTIFF            |
|           |                  |                 |                    |
|           |                  |                 | HDF-EOS Swath -\>  |
|           |                  |                 | HDF-EOS Grid       |
+-----------+------------------+-----------------+--------------------+
| GeoTIFF   | Yes, for         | Yes. Public     | Lots of converters |
|           | image-based      | domain API      | for TIFF; also     |
|           | standards; no,   | library         | GeoTIFF tag read & |
|           | for non-image    | partially       | write              |
|           | standards.       | documented.     |                    |
|           |                  |                 | Specialized        |
|           |                  |                 | converters for L7, |
|           |                  |                 | MODIS, MISR, ASTER |
+-----------+------------------+-----------------+--------------------+
| Fast      | No               | No. No API or   | No                 |
| Format    |                  | library exists. |                    |
+-----------+------------------+-----------------+--------------------+
| Native    | Depends on the   | Depends on the  | No, You have to    |
| Binary    | standard. Most   | standard, but   | write your own     |
|           | are specific to  | usually not,    | translation tool   |
|           | the application. | unless specific |                    |
|           |                  | efforts are     |                    |
|           |                  | made to         |                    |
|           |                  | document and    |                    |
|           |                  | publish an API. |                    |
+-----------+------------------+-----------------+--------------------+
| netCDF    | Yes. Since       | Yes. Net CDF    | -\> HDF            |
|           | netCDF can       | has a           |                    |
|           | contain general  | well-documented | -\> Matlab5        |
|           | scientific data, | API.            |                    |
|           | it encompasses   |                 |                    |
|           | all the other    |                 |                    |
|           | standards.       |                 |                    |
+-----------+------------------+-----------------+--------------------+
| BUFR/GRiB | Yes --           | Yes             | BUFR -\> CDF       |
|           | translation of   |                 |                    |
|           | meteorological   |                 |                    |
|           | parameters to    |                 |                    |
|           | other formats is |                 |                    |
|           | possible, with   |                 |                    |
|           | no loss of       |                 |                    |
|           | content. No for  |                 |                    |
|           | no               |                 |                    |
|           | n-meteorological |                 |                    |
|           | standards.       |                 |                    |
+-----------+------------------+-----------------+--------------------+

[See Acronym List if needed](#acronym-list)

[]{#__RefHeading___Toc26754282 .anchor}Table 5.2.2 Data Standards
Availability

  -------------------------------------------------------------------------
  **Data       **Evaluation                           
  Standard**   Questions**                            
  ------------ ----------------- -------------------- ---------------------
               Source code for   **Read/write         **Format well
               writing and       software well        described to
               reading data      documented?**        facilitate
               widely available?                      application
                                                      development?**

  HDF          Yes               Yes                  C, C++, Fortran, and
                                                      Java interfaces
                                                      exist. Applications
                                                      must use one of these
                                                      interfaces to access
                                                      the data

  HDF-EOS      Yes               Yes                  C, C++, Fortran, and
                                                      Java interfaces
                                                      exist. Applications
                                                      must use one of these
                                                      interfaces to access
                                                      the data

  GeoTIFF      Open source       User interface well  TIFF format well
               libraries; many   documented           documented. COTS
               COTS and freeware                      venders sometimes use
               applications                           variations of the
               available                              standard.

  Fast Format  No                No                   No

  Native       Not always        Not always           Not always
  Binary                                              

  netCDF       Yes (C, C++,      Yes                  Yes
               FORTRAN, Perl)                         

  BUFR/GRiB    There are few     Not always           Not always
               slightly                               
               different read                         
               and write                              
               software from                          
               different                              
               organizations or                       
               countries                              
  -------------------------------------------------------------------------

[See Acronym List if needed](#acronym-list)

[]{#__RefHeading___Toc26754283 .anchor}Table 5.2.3 Data Standards
Portability

  --------------------------------------------------------------------------
  **Data       **Evaluation                             
  Standard**   Questions**                              
  ------------ ------------------- -------------------- --------------------
               **Portable among    **Format is          **Standard can be
               commonly used       sufficiently well    implemented on one
               platforms?**        implemented that new platform and
                                   applications can     installed and tested
                                   access the           on other platforms
                                   implementation with  with minimal
                                   minimal effort?**    modification of
                                                        source code?**

  HDF          Precompiled HDF     Yes                  Yes
               libraries for a                          
               variety of popular                       
               platforms such as                        
               AIX, Cray                                
               HP,SGI,Sun, Linux                        
               and Windows.                             

  HDF-EOS      Precompiled HDF-EOS Yes                  Yes
               libraries for a                          
               variety of popular                       
               platforms such as                        
               AIX, HP, SGI, Sun,                       
               Windows, and Linux.                      

  GeoTIFF      Works on common OSs Need some knowledge  Yes
               (Linux, Unix,       of the specs. Need   
               Windows). Designed  understanding of     
               to be portable, but geotags to develop   
               need some knowledge applications.        
               of specs.                                

  Fast Format  Yes                 No                   Yes

  Native       Usually not         No                   No
  Binary                                                

  netCDF       All major OS's:     Yes                  Yes
               Winx, Unix, Linux,                       
               MacOS                                    

  BUFR/GRiB    YES                 A generalized        YES
                                   application would    
                                   require in depth     
                                   knowledge of all     
                                   variants, which is   
                                   not easy to obtain   
  --------------------------------------------------------------------------

[See Acronym List if needed](#acronym-list)

[]{#__RefHeading___Toc26754284 .anchor}Table 5.2.4 Data Standards
Evolvability

  --------------------------------------------------------------------------------------------------------------------------------
  **Data       **Evaluation                                     
  Standard**   Questions**                                      
  ------------ --------------- ------------- ------------------ ------------------------------------------------------------------
               **Is there a    **Is there a  **Is there a       **Is there an open process for evolution?**
               methodology for software      standard for       
               adding new      development   documentation?**   
               features?**     process?**                       

  HDF          NCSA is a       Yes, HDF      Yes, HDF library   Yes, HDF group allow input from outside users
               currently       library is    follows an         
               active and      funded and    internally defined 
               outside funded  developing    standard for their 
               group whose     software.     documentation.     
               purpose is                                       
               devoted to the                                   
               HDF project.                                     
               They manage                                      
               development                                      
               schedules and                                    
               are open to                                      
               suggestions                                      
               from users.                                      
               They are funded                                  
               from a variety                                   
               of sources.                                      

  HDF-EOS      Support is a    Yes, HDF-EOS  Yes, HDF-EOS       Yes, HDF-EOS group allow input from outside users
               contract from   library is    library follows an 
               NASA. They      funded and    internally defined 
               respond to      developing    standard for their 
               suggestions     software.     documentation.     
               from users. It                                   
               is NASA\'s                                       
               decision on how                                  
               long to support                                  
               the contract                                     
               and whether to                                   
               supply money                                     
               for development                                  
               as well as                                       
               maintenance.                                     

  GeoTIFF      Maintained by   Yes           Yes                OpenGIS, but no formal process. Work on the GeoTIFF v2.0 spec has
               JPL. No formal                                   been [slow
               process,                                         recently](http://www.remotesensing.org/geotiff/faq.html#Status),
               *i.e.,*                                          with some [recent efforts](http://www.geotiff.org/)
               Standards                                        
               committee. The                                   
               standard can be                                  
               modified by                                      
               others.                                          

  Fast Format  No              No            No                 No

  Native       No              No            No                 No
  Binary                                                        

  netCDF       Yes, through    Yes           Yes                Yes, informally through Unidata
               Unidata                                          

  BUFR/GRiB    YES             NO            Appears so, WMO    The WMO CBS approves changes to the format and maintains a
                                             issues Tech. Docs. software registry
                                             on these formats   
  --------------------------------------------------------------------------------------------------------------------------------

[See Acronym List if needed](#acronym-list)

[]{#__RefHeading___Toc26754285 .anchor}Table 5.2.5 Data Standards
Extensibility

  -------------------------------------------------------------------------
  **Data       **Evaluation Questions**   
  Standard**                              
  ------------ -------------------------- ---------------------------------
               **Does the standard allow  **Are there extensions or
               extensions or profiles to  profiles developed for the
               be developed?**            standard?**

  HDF          Yes                        HDF-EOS is a profile which was
                                          developed.

  HDF-EOS      This is a profile of HDF   No

  GeoTIFF      Yes. New projections can   None that are not part of
               be added. Multiple-band    unofficial list of projections
               GeoTIFFs allowed. GeoTIFF  
               2.0 will allow external    
               files.                     

  Fast Format  No                         No

  Native       No                         No
  Binary                                  

  netCDF       Yes                        Yes, *e.g.,* MINC: (Medical Image
                                          netCDF)

  BUFR/GRiB    Yes                        Not sure
  -------------------------------------------------------------------------

[See Acronym List if needed](#acronym-list)

[]{#__RefHeading___Toc26754286 .anchor}Table 5.2.6 Data Standards
Self-Describing

  -------------------------------------------------------------------------
  **Data       **Evaluation                            
  Standard**   Questions**                             
  ------------ ------------------ -------------------- --------------------
               **Is data able to  **Can the data be    **Does the format
               be stored so that  described internally contain information
               it can be read     to facilitate        to allow
               without a separate development of       subsetting?**
               document detailing applications?**      
               file contents? .**                      

  HDF          Data can be stored Data can be          Yes, information can
               so that it is      described with       be supplied to allow
               self-describing.   enough detail to     subsetting, but
               There are no       allow applications   there is not a
               restrictions in    to process data      requirement to do so
               the standard       appropriately. For   in a consistent way.
               though to prevent  instance, scale      Subsetting by
               developers from    factors may be       selecting selected
               using names such   included but it is   data fields can
               as Variable1.      developer dependent  easily be done on
                                  on how to do this.   any HDF file.
                                  As a result, generic 
                                  applications are     
                                  limited in their     
                                  scope. Applications  
                                  developed for a      
                                  specific data set    
                                  can be very precise. 

  HDF-EOS      Data can be stored Data can be          Because of the
               so that it is      described with       profile, subsetting
               self-describing.   enough detail to     along certain
               There are no       allow applications   geolocation fields
               restrictions in    to process data      can be done.
               the standard       appropriately. For   Individual
               though to prevent  instance, scale      developers can break
               developers from    factors may be       this process by not
               using names such   included but it is   following the
               as Variable1.      developer dependent  profile (there is no
                                  on how to do this.   internal checking
                                  As a result, generic done).
                                  applications are     
                                  limited in their     
                                  scope. Applications  
                                  developed for a      
                                  specific data set    
                                  can be very precise. 

  GeoTIFF      Geotags and image  Information can be   Yes
               specs.are in an    extracted at a pixel 
               ASCII header. Need level. Geolocation   
               library to access  and image info. is   
               contents.          available through    
                                  the interface.       

  Fast Format  No                 No                   No

  Native       No                 No                   No
  Binary                                               

  netCDF       Yes                Yes, it was designed Not inherently.
                                  to be                However,
                                  self-describing. CDL community-defined
                                  (schema-like) files  netCDF conventions
                                  are used to create   can be used to write
                                  files initially, but code that allows
                                  are not needed       subsetting
                                  thereafter.          

  BUFR/GRiB    No, you need       YES, but codes are   YES, but need the
               tables to          used to describe     tables
               interpret the data projections,         
                                  geophysical          
                                  parameters, etc., so 
                                  need to know these   
                                  codes to interpret   
                                  the data             
  -------------------------------------------------------------------------

[See Acronym List if needed](#acronym-list)

[]{#__RefHeading___Toc26754287 .anchor}Table 5.2.7 Data Standards Tools
Support

+----------+----------------------------+------------------------------+
| **Data   | **Evaluation Questions**   |                              |
| St       |                            |                              |
| andard** |                            |                              |
+==========+============================+==============================+
|          | Does the standard have     | **Does the standard have     |
|          | freeware support?          | COTS support?**              |
+----------+----------------------------+------------------------------+
| HDF      | Yes, NCSA tools            | Yes, PCI                     |
|          |                            |                              |
|          | ImageMagick                | ENVI                         |
|          |                            |                              |
|          | HDFLook                    | ER Mapper                    |
|          |                            |                              |
|          | HDFExplorer                | ERDAS-Imagine                |
|          |                            |                              |
|          | WIM                        | HDF Explorer                 |
|          |                            |                              |
|          | H5View                     | IDL                          |
|          |                            |                              |
|          |                            | ImageMagick                  |
|          |                            |                              |
|          |                            | MATLAB                       |
|          |                            |                              |
|          |                            | Mathematica                  |
|          |                            |                              |
|          |                            | NCL                          |
|          |                            |                              |
|          |                            | Noesys                       |
|          |                            |                              |
|          |                            | PV-Wave                      |
|          |                            |                              |
|          |                            | WIM                          |
+----------+----------------------------+------------------------------+
| HDF-EOS  | Yes, EOSView               | Slow:                        |
|          |                            |                              |
|          | HE5View                    | ENVI                         |
|          |                            |                              |
|          | Webwinds                   | IDL                          |
|          |                            |                              |
|          |                            | MATLAB                       |
|          |                            |                              |
|          |                            | Noesys                       |
+----------+----------------------------+------------------------------+
| GeoTIFF  | Open-source GIS tools      | Widespread:                  |
|          | (GRASS) and libraries for  |                              |
|          | C (libgeotiff), Java       | PCI-Geomatica                |
|          | (JAI), Python, etc.        |                              |
|          |                            | RSI-ENVI                     |
|          |                            |                              |
|          |                            | ESRI-ArcView                 |
|          |                            |                              |
|          |                            | SoftDesk-AutoCAD             |
|          |                            |                              |
|          |                            | ER Mapper                    |
|          |                            |                              |
|          |                            | ERDAS-Imagine                |
|          |                            |                              |
|          |                            | Laser-Scan                   |
|          |                            |                              |
|          |                            | MapInfo                      |
|          |                            |                              |
|          |                            | MicroImages                  |
|          |                            |                              |
|          |                            | Intergraph-GeoMedia          |
|          |                            |                              |
|          |                            | ENVI/IDL[^1]                 |
+----------+----------------------------+------------------------------+
| Fast     | GRASS, GDAL,               | Moderate:                    |
| Format   |                            |                              |
|          | OSSIM                      | [PCI](h                      |
|          |                            | ttp://www.pcigeomatics.com/) |
|          | (Simple format, so manual  |                              |
|          | import is common)          | ENVI                         |
|          |                            |                              |
|          |                            | [ER                          |
|          |                            | Mappe                        |
|          |                            | r](http://www.ermapper.com/) |
|          |                            |                              |
|          |                            | [ERDAS](ht                   |
|          |                            | tp://www.erdas.com/)-Imagine |
|          |                            |                              |
|          |                            | MicroImages                  |
|          |                            |                              |
|          |                            | (Simple format, so manual    |
|          |                            | import is common)            |
+----------+----------------------------+------------------------------+
| Native   | No                         | No                           |
| Binary   |                            |                              |
+----------+----------------------------+------------------------------+
| netCDF   | DODS                       | AVS                          |
|          |                            |                              |
|          | GMT                        | Environmental workbench      |
|          |                            |                              |
|          | Linkwinds                  | IDL interface                |
|          |                            |                              |
|          | GrADS                      | IRIS Explorer Module         |
|          |                            |                              |
|          | VisAD                      | MATLAB                       |
|          |                            |                              |
|          | etc.                       | NCAR graphics                |
|          |                            |                              |
|          |                            | Noesys                       |
|          |                            |                              |
|          |                            | PPLUS                        |
|          |                            |                              |
|          |                            | PV-Wave                      |
|          |                            |                              |
|          |                            | Silver Dicer                 |
|          |                            |                              |
|          |                            | WXP                          |
+----------+----------------------------+------------------------------+
| B        | Bufkit, GrADS              | Limited                      |
| UFR/GRiB |                            |                              |
+----------+----------------------------+------------------------------+

[See Acronym List if needed](#acronym-list)

[]{#__RefHeading___Toc26754288 .anchor}Table 5.2.8 Semantic Completeness

  -------------------------------------------------------------------------
  **Data       **Completeness        
  Standard**   Questions**           
  ------------ --------------------- --------------------------------------
               **Can the format      **Can the format convey the units of
               convey the data\'s    measure, the observation parameters
               precise spatial       (*e.g.,* spectral bands), accuracy
               location?**           estimates (error bars), and other
                                     elements needed to understand the
                                     data?**

  HDF          Yes                   Yes

  HDF-EOS      Yes                   Yes

  GeoTIFF      Yes (basically)       No

  Fast Format  Yes (basically)       No

  Native       N/A                   N/A
  Binary                             

  netCDF       Yes                   Yes

  BUFR/GRiB    Yes with              Yes with qualifications
               qualifications        
  -------------------------------------------------------------------------

[See Acronym List if needed](#acronym-list)

Based on the analysis of data standards using the eight criteria defined
and the evaluation questions, each standard was giving a rating (low,
medium and high) for each criterion. This rating is based on the answers
to the evaluation questions. If a data standard can satisfy all the
evaluation questions, a high rating is giving. If a data standard cannot
satisfy any of the evaluation questions, a low rating is giving. If a
data standard can satisfy some of the evaluation questions, a medium
rating is giving. Table 5.2.9 summarizes the results.

[]{#__RefHeading___Toc26754289 .anchor}Table 5.2.9 Data Standards
Evaluation

  ------------------------------------------------------------------------------------------------------------------------------------------------
  Criteria               Data Packaging                                                                                          
                         Standards                                                                                               
  ---------------------- --------------- ----------------- ----------------- --------------- ----------------- ----------------- -----------------
                         HDF             HDF-EOS           GeoTIFF           netCDF          Fast Format       **BUFR/ GRiB**    **Native Binary**

  **Interoperability**   [High]{.mark}   [High]{.mark}     [High]{.mark}     [High]{.mark}   [High]{.mark}     [Medium]{.mark}   [Medium]{.mark}

  **Availability**       [High]{.mark}   [High]{.mark}     [High]{.mark}     [High]{.mark}   [Low]{.mark}      [Low]{.mark}      [Medium]{.mark}

  **Portability**        [High]{.mark}   [High]{.mark}     [High]{.mark}     [High]{.mark}   [High]{.mark}     [Medium]{.mark}   [Medium]{.mark}

  **Evolvability**       [High]{.mark}   [High]{.mark}     [Medium]{.mark}   [High]{.mark}   [Low]{.mark}      [Medium]{.mark}   [Low]{.mark}

  **Extensibility**      [High]{.mark}   [Medium]{.mark}   [High]{.mark}     [High]{.mark}   [Low]{.mark}      [Medium]{.mark}   [Low]{.mark}

  **Self-describing**    [High]{.mark}   [High]{.mark}     [Medium]{.mark}   [High]{.mark}   [Low]{.mark}      [Low]{.mark}      [Low]{.mark}

  **Tools Support**      [High]{.mark}   [High]{.mark}     [High]{.mark}     [High]{.mark}   [Medium]{.mark}   [Medium]{.mark}   [Low]{.mark}

  **Completeness**       [High]{.mark}   [High]{.mark}     [Low]{.mark}      [High]{.mark}   [Low]{.mark}      [Medium]{.mark}   [N/A]{.mark}
  ------------------------------------------------------------------------------------------------------------------------------------------------

Table 5.2.9 shows that:

-   HDF and netCDF receive high ratings for all eight criteria. This
    suggests that HDF and netCDF are good candidates as transmission or
    archive standards.

-   Many standards, including HDF, HDF-EOS, GeoTIFF, and netCDF receive
    high ratings for Tools Support. This suggests that these standards
    can be used as analysis standards. Different user communities may
    prefer one standard over the others based on their familiarity with
    the standard and the simplicity and ease of use of the standards.

## Metadata and Documentation Standards Evaluation

This section analyzes five metadata standards and one documentation
standard used in heritage missions and related user communities:

-   The Federal Geographic Data Committee's Content Standard for Digital
    Geospatial Metadata;

-   ISO's draft standard on geographic metadata (ISO 19115);

-   The EOSDIS (Earth Observing System Data and Information System) Core
    System Core Metadata Standard;

-   The Directory Interchange Format (DIF) of the Global Change Master
    Directory (GCMD);

-   Dublin Core Metadata Elements ;

-   The EOSDIS Data Gateway (or Information Management System (IMS) V0)
    Guide.

The above metadata and documentation standards are analyzed according to
the same criteria used for data format standards. Table 5.3.1 shows the
analysis results.

[]{#__RefHeading___Toc26754290 .anchor}Table 5.3.1 Metadata and
Documentation Standards Evaluation

+----------+---------+--------+----------+-----------+-------+--------+
| Criteria | M       |        |          |           |       |        |
|          | etadata |        |          |           |       |        |
|          | Format  |        |          |           |       |        |
+==========+=========+========+==========+===========+=======+========+
|          | ISO     | FGDC   | ECS Data | GCMD DIF  | D     | Guide  |
|          | 19115   | C      | Model    |           | ublin |        |
|          |         | ontent |          |           | Core  |        |
|          |         | St     |          |           |       |        |
|          |         | andard |          |           |       |        |
+----------+---------+--------+----------+-----------+-------+--------+
| Interope | High:   | High:  | High:    | High:     | High: | M      |
| rability | FGDC    | GILS,  | GCMD,    |           |       | edium: |
|          | CSDGM,  | ECS,   | FGDC     | -\>ISO    | FGDC, |        |
|          | GC      | CIP    | co       |           | GILS, | -      |
|          | MD-DIF, | (via   | mpatible | -\>FGDC   | GCMD  | \>FGDC |
|          | etc. to | GEO    |          |           | DIF   |        |
|          | be      | p      |          | -\>Dublin |       |        |
|          | re      | rofile |          | Core      |       |        |
|          | defined | of     |          |           |       |        |
|          | as      | Z39.50 |          | -\>ANZLIC |       |        |
|          | p       | API).  |          |           |       |        |
|          | rofiles |        |          |           |       |        |
|          | of      | To be  |          |           |       |        |
|          | ISO     | reco   |          |           |       |        |
|          | -19115. | nciled |          |           |       |        |
|          |         | w/ ISO |          |           |       |        |
|          |         | 19115. |          |           |       |        |
+----------+---------+--------+----------+-----------+-------+--------+
| Avai     | Low:    | High:  | High:    | High:     | High  | High:  |
| lability | draft   | st     | SDP and  |           |       | Guide  |
|          | on      | andard | MDT      | DIFs &    |       | te     |
|          | private | is on  | Toolkits | DIF       |       | mplate |
|          | ISO     | FGDC   | on       | authoring |       | ava    |
|          | W       | w      | HDF-EOS  | tool      |       | ilable |
|          | ebsite; | ebsite | website  | available |       | on web |
|          | final   |        |          | on web    |       |        |
|          | std.    |        |          |           |       |        |
|          | will    |        |          |           |       |        |
|          | cost    |        |          |           |       |        |
+----------+---------+--------+----------+-----------+-------+--------+
| Por      | High    | High:  | High:    | High: can | High  | High:  |
| tability | content | c      | P        | be        |       | can be |
|          | w/ xml  | ontent | ortable, | created   |       | c      |
|          | schema  | w/ xml | can be   | w/ text   |       | reated |
|          |         | schema | created  | editor    |       | w/     |
|          |         |        | w/ text  |           |       | text   |
|          |         |        | editor   |           |       | editor |
+----------+---------+--------+----------+-----------+-------+--------+
| Exten    | High    | Mo     | High:    | High: Can | High: | High:  |
| sibility | (Built  | derate | Can be   | be        | Can   | Can be |
|          | for     | (exte  | extended | extended  | be    | ex     |
|          | p       | nsions |          |           | ext   | tended |
|          | rofiles | exist) |          |           | ended |        |
|          | &       |        |          |           |       |        |
|          | exte    |        |          |           |       |        |
|          | nsions) |        |          |           |       |        |
+----------+---------+--------+----------+-----------+-------+--------+
| Evol     | High    | Mo     | Medium:  | High:     | Mod   | High:  |
| vability | (many   | derate | Will be  | GCMD      | erate | EDG    |
|          | orgs.   |        | s        | group     |       | s      |
|          | m       |        | upported | maintain  |       | ci/ops |
|          | aintain |        | by       | the DIFs  |       | ma     |
|          | it)     |        | E        |           |       | intain |
|          |         |        | SDIS/EMD |           |       | the    |
|          |         |        |          |           |       | Guide  |
+----------+---------+--------+----------+-----------+-------+--------+
| Self-de  | M       | Mo     | Medium:  | High:     | Mod   | High:  |
| scribing | oderate | derate | Some     |           | erate |        |
|          |         |        | at       | C         |       | Guide  |
|          |         |        | tributes | ollection |       | te     |
|          |         |        | require  | level     |       | mplate |
|          |         |        | docum    | metadata  |       | is     |
|          |         |        | entation | self-de   |       | sel    |
|          |         |        | to       | scribing, |       | f-desc |
|          |         |        | un       | no        |       | ribing |
|          |         |        | derstand | granule   |       |        |
|          |         |        |          | metadata  |       |        |
+----------+---------+--------+----------+-----------+-------+--------+
| Tools    | Low:    | High:  | High:    | High:     | High: | M      |
| Support  | E       |        | SDP      |           |       | edium: |
|          | merging | fgdc   | toolkit  | Authoring | Many: |        |
|          | tran    | meta + |          | tool      | DCMI  | Con    |
|          | slators | xtme   | Metadata |           | Met   | verter |
|          | to/from |        | works    | DIF to    | adata |        |
|          | FGDC;   | SMMS   |          | XML       | Regi  | Card   |
|          | etc.    |        | QAMUT    |           | stry, | c      |
|          |         | Metama |          | Open      | DC-as | reator |
|          | No      | nager, | EDG      | access    | sist, |        |
|          | vendor  | Me     |          | API       | N     | Test   |
|          | support | taStar | T        |           | ordic | bed    |
|          | yet     |        | erraWhom | Science   | DC    |        |
|          |         | mp,    |          | keywords  | met   | Ve     |
|          |         | cns    | ECHO     | interface | adata | rifier |
|          |         |        |          |           | cre   |        |
|          |         | Many   | HDF      | No vendor | ator, | No     |
|          |         | vendor | (EOS)    | support   | DC    | vendor |
|          |         | s      |          |           | -dot, | s      |
|          |         | upport | UFM      |           | Ma    | upport |
|          |         |        |          |           | ntis, |        |
|          |         |        | No       |           | EdNA  |        |
|          |         |        | vendor   |           | Met   |        |
|          |         |        | support  |           | adata |        |
|          |         |        |          |           | w     |        |
|          |         |        |          |           | izard |        |
+----------+---------+--------+----------+-----------+-------+--------+
| Comp     | High:   | High:  | High:    | Low: Only | High: | N/A    |
| leteness | carry   | carry  | carry    | semantics | carry | (not   |
|          | s       | se     | semantic | de        | sem   | appli  |
|          | emantic | mantic | des      | scriptive | antic | cable) |
|          | desc    | descr  | criptive | elements  | d     |        |
|          | riptive | iptive | elements | for       | escri |        |
|          | e       | el     | of the   | c         | ptive |        |
|          | lements | ements | metadata | ollection | ele   |        |
|          | of the  | of the |          | metadata  | ments |        |
|          | m       | me     |          |           | of    |        |
|          | etadata | tadata |          |           | the   |        |
|          |         |        |          |           | met   |        |
|          |         |        |          |           | adata |        |
+----------+---------+--------+----------+-----------+-------+--------+

[See Acronym List if needed](#acronym-list)

As shown in Table 5.3.1, all five metadata standards and the
documentation standard receive similar ratings for most of the criteria
used. 1This is because the metadata standards analyzed, FGDC CSDSM, ISO
19115, and ECS data model, are all based on each other. ISO 19115 was
originally based on FGDC CSDSM version 1.0 and current FGDC "is
consistent with the emerging ISO draft standards". FGDC will adopt ISO
19115 standard when it becomes final in 2002. The ECS data model is FGDC
compliant, and FGDC Remote Sensing Extensions adopted many of the
attributes in the ECS data model. Since ISO 19115 is an international
metadata standard, it seems natural for FGDC and ECS data model to adopt
ISO 19115 when it becomes final. ISO 19115 receives a low rating for
"Availability" and "Tools Support" because the final ISO19115 standard
has not been published.

GCMD is a metadata (collection level only) standard for on-line catalog
access. Therefore, it is different from the ISO19115, FGDC CSDGM, and
ECS data model. GCMD has been widely used in NASA Earth Sciences and
GCMD DIF has been cross-mapped to ISO 19115, FGDC CSDSM, and Dublin Core
elements. GCMD receives a low rating for "Completeness" because it has
only semantic descriptive elements for collection metadata.

Guide document standard is interoperable with the FGDC metadata
clearinghouse via Z39.50 API. However, Guide document standard is more
suitable for Earth Science data sets.

## Catalog Interface Standards Evaluation

This section analyzes five catalog interface standards used in heritage
missions and related user communities:

-   The EOSDIS Version 0 protocol;

-   The GILS profile of the ANSI/ISO Z39.50 protocol;

-   The GEO Profile of Z39.50 by the Federal Geographic Data Committee
    (FGDC);

-   The CIP Profile of Z39.50 by the Committee on Earth Observation
    Satellites (CEOS);

-   The OGC Catalog Interface Specification by the OpenGIS Consortium;

The above catalog interface standards are analyzed according to the same
criteria used for data and metadata standards. However, "Portability" is
not used in the evaluation because interface standards specify
communication procedures and structures between a client and server(s),
which usually run on different platforms. So it is inherent that catalog
interface standards are portable across different platforms.
"Self-describing" is also removed from the evaluation because interface
standards are protocol specifications between a client and server(s) and
self-description does not apply to interface standards. Table 5.4.1
shows the analysis results.

[]{#__RefHeading___Toc26754291 .anchor}Table 5.4.1 Catalog Interface
Standards Evaluation

+--------+----------+----------+-------------+------------+---------+
| Cr     | Catalog  |          |             |            |         |
| iteria | I        |          |             |            |         |
|        | nterface |          |             |            |         |
|        | S        |          |             |            |         |
|        | tandards |          |             |            |         |
+========+==========+==========+=============+============+=========+
|        | EOSDIS   | Z39.50   | Z39.50      | CIP        | OGC     |
|        | V0       |          |             |            |         |
|        |          | GILS     | GEO         |            |         |
+--------+----------+----------+-------------+------------+---------+
| Inte   | Low:     | High:    | High:       | High:      | High:   |
| ropera | Merely   | Inter    |             | Z39.50     | Web     |
| bility | tested   | operable | In          | portions   | profile |
|        | for      | with     | teroperable | are        | is      |
|        | Interope | GEO, CIP | with GILS,  | int        | intero  |
|        | rability | & OGC    | CIP, and    | eroperable | perable |
|        | with CIP |          | OGC         | with GILS, | with    |
|        |          |          |             | GEO and    | GILS,   |
|        |          |          |             | OGC        | GEO,    |
|        |          |          |             |            | and CIP |
+--------+----------+----------+-------------+------------+---------+
| Availa | M        | High:    | High:       | High:      | High:   |
| bility | oderate: |          |             |            |         |
|        | Standard | Standard | Standard    | Standard   | S       |
|        | is       | can be   | can be      | can be     | tandard |
|        | co       | do       | accessed on | downloaded | is on   |
|        | ntrolled | wnloaded | www.bluean  |            | w       |
|        | but      | from the | geltech.com | From       | ww.open |
|        | a        | website  |             | ww         | gis.org |
|        | vailable | of       |             | w.ceos.org |         |
|        | from     | Library  |             |            |         |
|        | NASA     | of       |             |            |         |
|        | ESDIS    | Congress |             |            |         |
+--------+----------+----------+-------------+------------+---------+
| E      | Can be   | Can be   | Can be      | Can be     | Can be  |
| xtensi | extended | extended | extended    | extended   | e       |
| bility | using    | using    | using       | using      | xtended |
|        | the EDG  | Z39.50   | Z39.50      | Z39.50     | using   |
|        | Data     | extended | extended    | extended   | Z39.50  |
|        | Di       | services | services    | services   | e       |
|        | ctionary |          |             |            | xtended |
|        |          |          |             |            | s       |
|        |          |          |             |            | ervices |
+--------+----------+----------+-------------+------------+---------+
| Evolva | Process  | Process  | Process     | Process    | Process |
| bility | exists   | exists   | exists to   | exists to  | exists  |
|        | to       | to       | evolve the  | evolve the | to      |
|        | evolve   | evolve   | standard.   | standard.  | evolve  |
|        | the      | the      | USGS        | CEOS       | the     |
|        | s        | s        | maintains   | maintains  | st      |
|        | tandard. | tandard. | the         | the        | andard. |
|        | N        | Library  | standards   | standards  | Open    |
|        | ASA/GSFC | of       |             |            | GIS     |
|        | ESDIS    | Congress |             |            | Con     |
|        | m        | m        |             |            | sortium |
|        | aintains | aintains |             |            | ma      |
|        | the      | the      |             |            | intains |
|        | standard | standard |             |            | the     |
|        |          |          |             |            | st      |
|        |          |          |             |            | andards |
+--------+----------+----------+-------------+------------+---------+
| Tools  | Low: No  | High:    | Moderate:   | Moderate:  | Mo      |
| S      | co       | Many     | Some        | Some       | derate: |
| upport | mmercial | freeware | freeware    | freeware   | Some    |
|        | or free  | and      | and         | and        | f       |
|        | software | co       | commercial  | commercial | reeware |
|        | tool     | mmercial | tools such  | tools such | and     |
|        | support  | tools    | as Isite    | as Isite   | com     |
|        |          | such as  |             |            | mercial |
|        |          | Isite    | MetaManager | M          | tools   |
|        |          |          |             | etaManager | such as |
|        |          | Met      | MetaStar    |            | Isite   |
|        |          | aManager |             | MetaStar   |         |
|        |          |          |             |            | Meta    |
|        |          | MetaStar |             |            | Manager |
|        |          |          |             |            |         |
|        |          |          |             |            | M       |
|        |          |          |             |            | etaStar |
|        |          |          |             |            |         |
|        |          |          |             |            | Ear     |
|        |          |          |             |            | thscape |
|        |          |          |             |            |         |
|        |          |          |             |            | WebAPI  |
|        |          |          |             |            |         |
|        |          |          |             |            | Service |
|        |          |          |             |            | Manager |
+--------+----------+----------+-------------+------------+---------+
| Comple | Defines  | Defines  | Defines     | Defines    | Defines |
| teness | search,  | search   | search and  | search,    | search  |
|        | re       | and      | retrieval   | retrieval, | &       |
|        | trieval, | r        | functions,  | order, and | re      |
|        | order,   | etrieval | information | browse     | trieval |
|        | and      | fu       | elements,   | functions, | fun     |
|        | browse   | nctions, | and allowed | i          | ctions, |
|        | fu       | inf      | values.     | nformation | and     |
|        | nctions, | ormation | Meets FGDC  | elements,  | opt     |
|        | inf      | e        | re          | and        | ionally |
|        | ormation | lements, | quirements. | allowed    | data    |
|        | e        | and      |             | values.    | access  |
|        | lements, | allowed  |             |            | and     |
|        | and      | values.  |             |            | man     |
|        | allowed  | Meets    |             |            | agement |
|        | values.  | the      |             |            | fun     |
|        | Meets    | requ     |             |            | ctions. |
|        | NASA     | irements |             |            | Is data |
|        | EOSDIS   | of the   |             |            | model   |
|        | requi    | library  |             |            | neutral |
|        | rements. | and      |             |            | (no     |
|        |          | museum   |             |            | specif  |
|        |          | com      |             |            | ication |
|        |          | munities |             |            | for     |
|        |          |          |             |            | info    |
|        |          |          |             |            | rmation |
|        |          |          |             |            | e       |
|        |          |          |             |            | lements |
|        |          |          |             |            | and     |
|        |          |          |             |            | allowed |
|        |          |          |             |            | values) |
+--------+----------+----------+-------------+------------+---------+
| Im     | NASA     | Widely   | Used in     | Used in    | Several |
| plemen | EOSDIS   | used in  | several     | ESA INFEO  | com     |
| tation | uses the | federal  | operational | system     | mercial |
| Status | V0       | and      | geospatial  | since      | v       |
|        | standard | state    | information | 2001.      | endors, |
|        | in       | library, | systems,    |            | such as |
|        | o        | museum,  | such as     |            | GeoDan  |
|        | peration | sci      | FGDC        |            | and     |
|        | since    | entific, | Cle         |            | Co      |
|        | 1994.    | c        | aringhouse, |            | nTerra, |
|        |          | hemical, | CEONet,     |            | have    |
|        |          | bio      | MEL, since  |            | impl    |
|        |          | logical, | mid 1990s   |            | emented |
|        |          | and      |             |            | OGC in  |
|        |          | other    |             |            | their   |
|        |          | com      |             |            | com     |
|        |          | munities |             |            | mercial |
|        |          | since    |             |            | p       |
|        |          | 1990's.  |             |            | roducts |
|        |          |          |             |            | since   |
|        |          |          |             |            | the     |
|        |          |          |             |            | late    |
|        |          |          |             |            | 1990s.  |
+--------+----------+----------+-------------+------------+---------+

[See Acronym List if needed](#acronym-list)

As shown in Table 5.4.1, the GILS, GEO, CIP profiles of Z39.50 and the
OGC catalog standards receive similar ratings for most of the criteria
used. This is mainly because GILS, GEO, CIP profiles and the OGC Web
profile are all based on the Z39.50 protocol \[[[]{#d5_4_2
.anchor}2](#r5_4_2)\] that defines search and present operations. The
search operation includes a query and requests the creation of a result
set. The present operation returns subsets of a result set. Z39.50 is a
stateful protocol. The base data model of Z39.50 is called bib-1 and is
the main set of attributes appropriate to the bibliography world. GILS
is developed on the basis of Z30.50 bib1 attributes, but in addition,
contains extra attributes and references. GEO profile builds on bib-1
and GILS by extending the metadata model to include data elements from
the Content Standard for Digital Geospatial Metadata maintained by the
FGDC. CIP is both a profile and an extension of Z39.50. CIP is a profile
of Z39.50 using messages, attributes, and elements in the standards as
well as defining CIP-specific attributes and elements. CIP is an
extension of Z39.50 through the addition of collections as a data
structure and the addition of an ordering facility. Searches can be
targeted at Collections, Product or Guide inventories. Retrieval of
Browse products is supported. The OGC Catalog Interface standard has a
general model and several profiles, *e.g.,* CORBA profile, Web Profile. 
The OGC catalog interface Web profile is compliant to Z39.50. It
contains constructs for the mandatory functions of search and retrieval,
and for the optional functions of data access (order) using CIP order
mechanism. The Z39.50 messages have been transformed into XML based
messages for easier implementation. OGC catalog interface standard is
data model independent \[[[]{#d5_4_3 .anchor}3](#r5_4_3)\]. CIP is
aligned with GEO and GILS profiles \[[[]{#d5_4_4 .anchor}4](#r5_4_4)\].

The EOSDIS V0 system was originally developed as a prototype system for
the EOSDIS. It gradually evolved into an operational system since 1994.
Although it has low interoperability and low tool support, it is the
only operational catalog interface standard used for inventory data by
NASA Data Active Archive Center (DAAC) Alliance. It has recently been
successfully tested for interoperability with the CIP profile.

## User Surveys 

In conjunction with the analyses of standards described above, we also
conducted a user opinion survey on the data and metadata standards used
in heritage and other missions to gain feedback from the user community.
Not all of the criteria used in the standards analysis above were used
in the survey because the survey was conducted before the criteria
selection was refined. Also, there are several criteria used in the
survey but not in the analysis. These criteria are more subjective than
the more refined criteria used in the standards analysis and are defined
below.

-   *Ease of use* for producers.

-   *Ease of use* for consumers.

-   *Acceptability* - Format is acceptable to a broad cross-section of
    likely users of the products.

-   *Suitability* - Has the proper descriptive power or precision for
    the task.

-   *Survivability* - The ability to be used by the community for many
    years.

There were a total of 45 survey respondents. Twenty surveys were
returned from attendees of the NASA Science Data Processing Workshop,
February 2002. Twenty-five surveys were collected from EOS User Working
Group members at GSFC, LaRC, JPL, EDC, NSIDC, and ORNL DAACs (See
Acronym List) and from other users. The survey results and some relevant
statistics are summarized below.

### 5.5.1 Data Format Standards {#data-format-standards .unnumbered}

Respondents were asked the question: "What weight should NASA give to
the following criteria in evaluating a standard?" They were then asked
to rate defined criteria with respect to "what weight NASA should apply
when evaluating a standard," using a scale from one to six, where one
was the lowest and six was the highest. Users gave "Ease of Use for
Consumers" the highest rating (5.6) while "Evolvability" and "Ease of
Use for Producers" received the lowest ratings. The statistics are
summarized in Table 5.5.1.

[]{#__RefHeading___Toc26754292 .anchor}Table 5.5.1 Survey Ratings of
Attribute Importance

  --------------- -------------- ------------ ------------------- ------------------- ------------------ ----------------- ------------------ ----------------- ----------------------
  **Statistic**   **Criteria**                                                                                                                                  

                  **Ease of Use  **Ease of    **Survivability**   **Acceptability**   **Availability**   **Portability**   **Evolvability**   **Suitability**   **Interoperability**
                  For Producer** Use For                                                                                                                        
                                 Consumer**                                                                                                                     

  **Average**     **4.3**        **5.6**      **5.3**             **5.1**             **5.1**            **5.4**           **4.6**            **4.6**           **4.8**

  **Mode**        **4**          **6**        **6**               **5**               **6**              **6**             **4**              **5**             **6**

  **Sample Size** **41**         **41**       **42**              **42**              **41**             **42**            **41**             **39**            **41**
  --------------- -------------- ------------ ------------------- ------------------- ------------------ ----------------- ------------------ ----------------- ----------------------

Survey respondents were then asked to rate different data format
standards using the set of criteria from the previous question. They
were also ranked on a scale from one to six, where one was the lowest
and six was the highest.

The survey of data standards used in the heritage missions indicated
that users are most familiar with the HDF data standard. Thirty-five of
the 45 total respondents were familiar with and rated HDF, while only
one respondent rated Fast Format and only three rated BUFR. The sample
sizes of the responses concerning Fast Format and BUFR were so small
that these two formats were deleted from the results. Overall, the
number of respondents is small (total respondents is 45, however, the
sample size for a particular standard and particular criteria ranges
between 5 and 35 as shown in Table 5.5.2); thus, drawing decisive
conclusions from the survey is difficult. For example, approximately
one-third of the people surveyed were not familiar with more than three
data standards, and in many cases they gave the standard they were most
familiar with the highest rating. However, the survey still shows some
interesting findings.

-   Binary format received the highest rating for most of the criteria
    used including Interoperability, Acceptability, Availability,
    Survivability, Evolvability, and Ease of Use. However, Binary was
    rated the lowest for Portability.

-   HDF and netCDF were rated highest for Suitability and Portability.
    However, they were rated lowest for Ease of Use. This is due to the
    steep learning curve as indicated by the people surveyed.

-   GeoTIFF was rated the second highest for Ease of Use. It was also
    rated high for Interoperability and Acceptability. However, it was
    rated the lowest for Suitability, Availability and Evolvability.

A summary of the statistics is shown in Table 5.5.2.

[]{#__RefHeading___Toc26754293 .anchor}Table 5.5.2 Data Standards Survey
Evaluation

  ---------------------------------------------------------------------------------
  Criteria           Type of      Data                                    
                     Statistic    Format                                  
  ------------------ ------------ --------- --------- --------- --------- ---------
                                  HDF       HDF-EOS   netCDF    GeoTIFF   Binary

  Interoperability   Average      **4.3**   **3.6**   **3.6**   **4.5**   **4.9**

                     Mode         **6**     **4**     **3**     **5**     **5**

                     Sample Size  **28**    **24**    **10**    **8**     **12**

  Acceptability      Average      **4.4**   **3.9**   **4.1**   **4.3**   **4.8**

                     Mode         **6**     **3**     **4, 6**  **6**     **5, 6**

                     Sample Size  **34**    **27**    **15**    **9**     **13**

  Availability       Average      **4.7**   **4.2**   **4.9**   **3.4**   **5.1**

                     Mode         **6**     **6**     **6**     **3, 4**  **5**

                     Sample Size  **35**    **15**    **15**    **9**     **12**

  Portability        Average      **4.8**   **4.5**   **5.1**   **4.7**   **4.6**

                     Mode         **6**     **6**     **6**     **6**     **5**

                     Sample Size  **34**    **28**    **12**    **11**    **12**

  Evolvability       Average      **4.4**   **4.1**   **3.8**   **3.7**   **4.7**

                     Mode         **6**     **3, 6**  **6**     **4**     **6**

                     Sample Size  **20**    **20**    **5**     **6**     **9**

  Suitability        Average      **4.9**   **5.1**   **4.6**   **3.9**   **4.1**

                     Mode         **5**     **5**     **6**     **4**     **5**

                     Sample Size  **28**    **25**    **9**     **9**     **9**

  Ease of use for    Average      **3.5**   **3.6**   **3.6**   **4.8**   **5.0**
  Consumer                                                                

                     Mode         **4**     **3**     **2**     **4, 6**  **5**

                     Sample Size  **31**    **27**    **12**    **10**    **13**

  Survivability      Average      **4.7**   **4.5**   **4.7**   **4.7**   **5.5**

                     Mode         **5**     **4, 5**  **5, 6**  **4**     **6**

                     Sample Size  **29**    **24**    **12**    **10**    **13**
  ---------------------------------------------------------------------------------

Table 5.5.3 summarizes the results of four essay type questions from the
opinion survey. An interesting result is that the HDF format received
the most responses for all of the questions. About 60% of people
surveyed listed that they had success with HDF and about 30% of the
people listed HDF as a standard they foresee being used in the future.
However, about 25% of the people surveyed listed that HDF is an
impediment to their work. HDF-EOS, netCDF, and binary also received high
ratings as a standard with which respondents had success (about 43% had
success with HDF-EOS, 27% had success with Binary, and 23% had success
with netCDF).

A detailed analysis of the results indicates that most of the
respondents who had success with HDF are data producers from the
computer/data manipulation areas (75% of them had success), atmospheric
scientists (69%), and oceanographers (50%).

Respondents who had problems with HDF are atmospheric scientists (25% of
them had problems), computer/data manipulation specialists (25%), and
environmental scientists (20%). The most common complaint was that HDF
was too complicated, making it difficult to learn (7 out of the 12
respondents), and that there was a lack of available tools (5 out of the
12 respondents). One atmospheric scientist seemed to capture this idea
fairly well: "HDF, HDF-EOS, and netCDF are all initially more difficult,
because they are more complex. But it isn\'t really an impediment as
long as you have the right tools." Other impediments included huge file
size, slow conversion from HDF4 to HDF5, and performing compression in
HDF libraries. Similar complaints were listed for HDF-EOS, but also lack
of support by RSI and cryptic failure messages were mentioned.

For other data formats, one atmospheric scientist said that GeoTIFF
showed a dependency on machines. Another atmospheric scientist called
BUFR, "ancient and primitive." The lone comment from an oceanographer
for the GRiB format labeled it as difficult to use (worse than HDF). And
one data producer said that binary was platform dependent and hard to
verify.

Respondents who recommended HDF as a future standard are data producers
(33% of them recommended), atmospheric scientists (25%), and 17% of
oceanographers. Of the 21 respondents that commented on formats they
foresee emerging in the future, eight (four atmospheric scientist, three
data producers, and one environmental scientist) foresaw some form of
HDF (HDF4, HDF5, or HDF-EOS) for the future because it is a powerful and
versatile format. Four respondents (two data producers and two
atmospheric scientists) felt some form of HDF was inevitable/mandated
for the future.

Concerning other data formats, one oceanographer foresees and recommends
GeoTIFF because it's easy to use, one atmospheric scientist says that
ASCII is obvious for small data sets, and two respondents (an
atmospheric scientist and an oceanographer) foresee/recommend netCDF
because IDL supports it or because younger scientists are adopting it so
it's growing in popularity. An oceanographer and an atmospheric
scientist said that they foresee binary in the future because it has
many tools and is widely available and because it is easy to use. Two
scientists (atmospheric and environmental) recommend binary because it
is simple, while one data producer recommends binary because many models
require binary inputs.

Other comments on what the respondents foresee/recommend include: 1)
atmospheric scientist -- "Simple storage layer coupled with
sophisticated connectivity layer. Let data producers have more freedom
to use most appropriate format but force connectivity rules," 2)
environmental scientist -- "Something more easily accessible across
platforms, software packages," 3) data producer -- "Provide free,
portable tools for ingesting and reformatting HDF and HDF-EOS. End-users
will want to easily convert to formats such as ERDAS, ENVI/IDL, PCI, and
Arcgrids."

[]{#__RefHeading___Toc26754294 .anchor}Table 5.5.3 Summary of Survey
Essay Questions

  -------------------------------------------------------------------------
  Question        Data                                            
                  Format                                          
  --------------- --------- ------------- ----------- ----------- ---------
                  HDF       HDF-EOS       netCDF      GeoTIFF     Binary

  Data Format     **27**    **19**        **10**      **2**       **12**
  Used                                                            
  Successfully                                                    

  Data Format was **12**    **9**         **1**       **1**       **1**
  Impediment                                                      

  Data Format     **13**    **8**         **4**       **1**       **4**
  Foresee                                                         
  Emerging                                                        

  Data Format     **13**    **6**         **1**       **2**       **3**
  Recommend                                                       
  -------------------------------------------------------------------------

### 5.5.2 Metadata Format Standards {#metadata-format-standards .unnumbered}

For the metadata section, respondents of the user questionnaire were
given a series of questions identical to those asked in the data format
section of the survey. Of the 45 total surveys collected, only 16 people
responded to the metadata questions. Thirteen of those were returned
from attendees of the NASA Science Data Processing Workshop, February
2002. Three of those surveys were from EOS User Working Group members.

As in the data format portion of the survey, respondents were given a
series of eight questions with a set of defined criteria as applied to
particular metadata formats. They were asked to rate each metadata
format (see headings in Table 5.5.4) with respect to the criteria using
a scale from one to six, where one was the lowest and six was the
highest.

The survey of metadata standards used in the heritage missions indicates
that users are most familiar with the ECS Data Model, with as many as 11
responses out of the 16 total metadata responses, while only one
respondent rated ISO and only two rated GCMD. The sample sizes of the
responses concerning ISO and GCMD were so small that these two metadata
formats were deleted from the results. Overall, the number of
respondents is extremely small (as mentioned above) with the sample size
for a particular standard and particular criteria ranging from 3 to 11
as shown in Table 5.5.4; thus, drawing decisive conclusions form the
survey is difficult.

From the limited samples shown in Table 5.5.4, the FGDC content metadata
and ECS data model received comparable ratings for most of the criteria.
The FGDC content metadata received a little higher ratings than the ECS
data model on Acceptability, Availability, Evolvability, and
Survivability, while the ECS data model received a little higher ratings
than the FGDC metadata model on Portability and Interoperability.

[]{#__RefHeading___Toc26754295 .anchor}Table 5.5.4 Metadata Standards
Survey Evaluation

  ------------------------------------------------------------------------
  Criteria              Statistics         Metadata Format  
  --------------------- ------------------ ---------------- --------------
                                           FGDC             ECS

  Interoperability      Average            **3.0**          **3.4**

                        Sample Size        **3**            **7**

  Acceptability         Average            **4.0**          **3.8**

                        Sample Size        **5**            **10**

  Availability          Average            **4.0**          **3.5**

                        Sample Size        **4**            **11**

  Portability           Average            **3.0**          **4.0**

                        Sample Size        **4**            **10**

  Evolvability          Average            **5.0**          **3.7**

                        Sample Size        **2**            **9**

  Suitability           Average            **4.4**          **4.5**

                        Sample Size        **5**            **11**

  Ease of use for       Average            **3.4**          **3.5**
  Consumer                                                  

                        Sample Size        **5**            **10**

  Survivability         Average            **5.6**          **4.4**

                        Sample Size        **5**            **9**
  ------------------------------------------------------------------------

Table 5.5.5 summarizes the results of four essay type questions from the
opinion survey. Sample sizes are very small as only 11 respondents
answered the first question, 4 respondents answered the second and third
questions, and 3 people answered the fourth question. The most
significant observation is that 8 (73%) of the 11 respondents listed
that they had success with the ECS Data Model. Only 27% claimed success
using the FGDC Content Standard. 27% of the 11 respondents said that the
ECS Data Model was an impediment, while 9% claimed the FGDC content
standard was an impediment to their research. A detailed analysis of the
results indicates that the respondents who claimed success with the ECS
data model are data producers (5) and oceanographers (2). Some of the
data producers (3) who claimed success with the ECS data model also
indicated that they had problems with the ECS data model. Most of the
complaints related to the ECS data model are that it is too complex, not
flexible, not consistently applied, and that there is not enough tool
support. Respondents who claimed success with the FGDC content standard
include one data producer, one environmental scientist, and one
atmospheric scientist. The environmental scientist who had used FGDC
successfully also indicated that FGDC is an impediment in that there are
too few tools and little portability, documentation, and
interoperability. In terms of metadata that the respondents foresee in
the future, one suggests adopting ISO 19115 to replace the current FGDC
and adopting the FGDC extensions for remote sensing based on the ECS
data model. One respondent recommends XML standard descriptions and
defining XML DTD/schema for all the specific applications. One
respondent suggests refining the ECS data model, dropping most of the
groups/classes, and attaching metadata to files.

[]{#__RefHeading___Toc26754296 .anchor}Table 5.5.5 Summary of Metadata
Survey Essay Questions

  ------------------------------------------------------------------------
  Question                      Number of Times    
                                Metadata Format    
                                Was Listed         
  ----------------------------- ------------------ -----------------------
                                FGDC               ECS

  Data Format Used Successfully **3**              **7**

  Data Format was Impediment    **1**              **3**

  Data Format Foresee Emerging  **1**              **2**

  Data Format Recommentd        **1**              **2**
  ------------------------------------------------------------------------

**[References:]{.underline}**

\[[[]{#r5_1_1 .anchor}1](#d5_1_1)\] Jim Frew, 1998, How to Think about
Data Formats,
<http://spso.gsfc.nasa.gov/diss/Meetings/19981005/frew_pres_9810.html>.

\[[[]{#r5_4_2 .anchor}2](#d5_4_2)\] Information Retrieval (Z39.50):
Application Service Definition and Protocol Specification, Z39.50
Maintenance Agency, 1995

\[[[]{#r5_4_3 .anchor}3](#d5_4_3)\] OpenGIS -- Catalog Interface
Implementation Specification (version 1.0), OpenGIS Project Document
99-051s

\[[[]{#r5_4_4 .anchor}4](#d5_4_4)\] CIP Alignment with other Z39.50
profiles, 1997, European Commission

# Summary

We have surveyed the standards for data and metadata that are in use in
heritage missions or under consideration by the missions expected to be
in formulation in the near-term. Lessons learned from heritage missions,
some of the NOAA missions, and STDS have been reviewed. In particular,
data and metadata standards in use in heritage missions and other EOS
missions have been analyzed using a suite of criteria. Simple statistics
and results from our user interview/survey to gather data producers' and
data users' feedback on data and metadata standards are presented.

The highlights of lessons learned from heritage missions and other
standards are summarized below.

-   Multiple data distribution formats are used for some heritage
    missions, such as Landsat-7 and QuikSCAT/SeaWinds, to satisfy the
    diverse requirements from the user communities. Many NOAA POES
    missions use multiple data distributions formats to give users the
    flexibility to select the best data formats for their applications.
    This adds to the workload of NOAA agencies, such as NOAA NESDIS and
    NOAA NCDC, requiring them to develop and maintain different data
    translation tools in order to support different requirements from
    their user communities.

-   Many different versions of HDF-EOS have been implemented for Terra
    data products, thus, creating problems for data interchange between
    mission instrument teams and users because different readers may be
    needed to read different implementations of the HDF-EOS data format.
    New EOS missions have realized this problem. For example, the Aura
    mission instrument teams have decided to adopt a uniform set of
    HDF-EOS file format guidelines so that data products from any Aura
    instrument are easily interchanged, *i.e*., the same set of tools
    and I/O routines can be used for all of the Aura data products.

-   An important lesson learned from several missions, including
    Jason-1, SeaWiFS, SeaWinds, and ACRIM, is to provide good user
    support and experienced help desk for HDF-EOS implementation and
    usage. Many missions indicated that the "handholding" should not be
    underestimated.

-   The Spatial Data Transfer Standard (SDTS), the national spatial data
    transfer mechanism for all U.S. Federal agencies, fell short of its
    ambitious goals and the marketplace was slow to accept and support
    it. The SDTS experience illustrates the importance of keeping pace
    with technology and market trends and emerging expectations, even
    after capturing initial requirements.

We devised eight standards criteria in order to objectively evaluate
data and metadata standards. The results from the analysis of data
format standards and metadata standards using the eight evaluation
criteria are summarized below.

-   HDF, and netCDF received high ratings for the evaluation criteria
    such as Interoperability, Availability, Evolvability, Portability,
    Extensibility, Tools Support, Completeness, and Self-describing.
    Many standards, including HDF, HDF-EOS, GeoTIFF, and netCDF received
    high ratings for Tools Support.

-   BUFR/GriB and Fast Format in general were rated low to medium for
    many evaluation criteria, mainly Self-describing, Availability, and
    Completeness.

-   Native binary received low ratings for Evolvability, Extensibility,
    Self-describing, and Tools Support (Chapter 5). Based on our
    analysis, native binary does not constitute a data standard and it
    cannot be compared alongside open consensus standards.

-   The GILS, GEO, and CIP profiles of Z39.50 and the OGC catalog
    standards receive similar ratings for most of the criteria used.
    This is mainly because GILS, GEO, CIP profiles and the OGC Web
    profile are all based on the Z39.50 protocol. NASA EOS has used
    Z39.50 based standards for collection and document search. But for
    inventory search, EOSDIS Version 0 is the only operational catalog
    interface standard used by NASA EOS.

-   The metadata and documentation standards analyzed received similar
    high to medium ratings for most of the evaluation criteria as many
    of the metadata standards, such as FGDC CSDGM, ISO 19115, and ECS
    data model, are all based on each other. We note that metadata
    standards are converging on the ISO 19115 when it becomes final in
    the near future.

We conducted a total of 45 interviews and surveys of data users from the
EOS User Working Group members at different DAACs and of data
producers/users from the 2002 NASA Science Data Processing Workshop.
Although the sample size is not large, the interview/survey results
illustrate feedback from data users and data producers on the data and
metadata standards currently in use in NASA missions. Statistics and
results are described in Chapter 5. A summary of the statistics/results
is presented here.

-   All of the users/producers answered questions related to data format
    standards. Only one-quarter of the users/producers answered
    questions related to metadata standards. This indicates that data
    producers and data users care more about (or are more familiar with
    or is more relevant to them) data format standards than metadata,
    and that many of them also have strong feelings about the data
    format standards.

-   Users/producers are most familiar with the HDF data standard as 35
    of the 45 total respondents were familiar with, and rated HDF, while
    only one respondent rated Fast Format and only three respondents
    rated BUFR. Many of those interviewed/surveyed are not familiar with
    multiple data formats, with only one-half of the respondents
    familiar with more than two data standards. In many cases they gave
    the standard they were most familiar with the highest rating. The
    results, therefore, may be biased.

-   The interview/survey results show that HDF and netCDF were rated
    highest for Portability and Suitability and lowest for Ease of Use.
    However, respondents did not give high ratings to HDF and netCDF on
    Interoperability, Acceptability, Availability, and Evolvability. On
    the contrary, respondents rated binary format the highest for
    Interoperability, Acceptability, Availability, and Evolvability.
    This could possibly be because the producers'/users' understanding
    of these criteria are different from what we described in Chapter 5.
    This could also be because users are more familiar with Binary
    format and favor Binary format rather than the HDF or netCDF
    formats.

-   The majority (60%) of respondents indicated that they had success
    with HDF and about one-third of the respondents recommend HDF as a
    future standard for NASA mainly because it is a powerful and
    versatile format. However, about one-quarter of the respondents also
    point out that HDF was an impediment to their work because HDF was
    too complicated, making it difficult to learn, and that there was a
    lack of available tools.

-   For the metadata standards surveyed, respondents are most familiar
    with the ECS Data Model, with as many as 11 responses out of the 16
    total metadata responses, while only one respondent rated ISO and
    only two respondents rated GCMD DIF. The ECS data model and the FGDC
    content metadata received comparable ratings for most of the
    criteria. This result is correspondent to the results derived from
    the standards analysis.

-   For future metadata standards, some respondents recommend adoption
    of the ISO 19115 to replace the current FGDC and adoption of the
    FGDC extensions for remote sensing based on the ECS data model.
    Others recommend XML standard descriptions for metadata and refining
    the ECS data model.

# Conclusions

Recommendations on data interface standards, data packaging standards,
metadata standards, documentation standards, and associated activities
for the near-term missions are summarized below. Acknowledging that
there are several levels of requirements/guidelines, the following
keywords are used to differentiate between them.

-   *must* - This is mandatory.

-   *should* - This guideline is mandatory *except* where valid reasons
    exist to allow for its modification. Care should be taken in
    modifying or ignoring these guidelines.

-   *may* - This is a guideline which, while the NTMS group suggests it
    is worthwhile, is not mandatory.

## 7.1 Data Packaging Standards {#data-packaging-standards .unnumbered}

The SEEDS Near-term Missions Study (NTMS) group recommends that data be
packaged in an interchange format for storage and be available in
multiple data distribution formats. The interchange format is for
sharing data among the ESE data systems components (*i.e.*, data centers
including PI-managed Mission Data Centers, "Backbone" Processing
Centers, Science Data Centers, Application Data Centers, Multimission
Data Centers, and similar centers or systems). Data distribution formats
are for end-users.

### NTMS recommends the following for packaging of near-term mission standard products. {#ntms-recommends-the-following-for-packaging-of-near-term-mission-standard-products. .unnumbered}

### 7.1.1 Data Distribution Formats Recommendations {#data-distribution-formats-recommendations .unnumbered}

1.  Data distribution facilities *must* enable packaging of standard
    data products in multiple distribution formats.

2.  Distribution formats *must* emphasize end-user needs and
    convenience.

##### Rationale {#rationale .unnumbered}

At present, and for the near future, most applications and end-user
practices are file-based. In such use, the format of the data files or
the API used to read the data files are key to data access. Some
communities have coalesced around particular file formats or access
tools. For the greatest success in reaching multiple application and
science disciplinary uses, ESE must support the preferences of these
communities. These distribution-packaging choices (most simply
understood as the formats in which data are sent to users) allow users
to have access to data in one of several well-used formats. NTMS finds
that several missions (including Landsat-7, QuikSCAT/SeaWinds, and many
NOAA missions) have successfully employed multiple data distribution
formats to satisfy the diverse requirements of their user communities
(see Chapter 3).

All "Backbone" Processing Centers, Science Data Centers, and
Multimission Data Centers *must* support a limited set of distribution
packaging standards based on their end-users' requirements. The choice
of distribution packaging *must* be made with the target community in
mind and governed by applicability to task and the convenience of
end-users.

NTMS does not have particular recommendations for distribution packaging
standards. Based on our study, GeoTIFF format, WMO BUFR and GRID
formats, Landsat Fast Format, and the API standards of netCDF and
HDF/HDF-EOS and others are appropriate distribution packaging options.

### 7.1.2 Data Interchange Formats Recommendations {#data-interchange-formats-recommendations .unnumbered}

1.  Interchange data sets *must* use a recognized packaging standard.
    The choice of standard *must* emphasize completeness and
    self-description.

2.  Most of the near-term missions have indicated an interest in using
    HDF/HDF-EOS or netCDF. We agree that HDF/HDF-EOS or netCDF are
    appropriate choices as interchange data formats among ESE data
    system components (*i.e.*, data centers).

3.  Each appropriate ESE Near-term mission community *must* develop a
    profile of HDF/HDF-EOS or netCDF appropriate not only to the narrow
    needs of a particular mission but also to the wider needs of the
    allied community (See Chapter 4 for a discussion of profiles).

4.  The development of each community\'s profile *must* be a process
    involving mission science teams, interested end-users, and
    experienced consultants. SEEDS NTMS has found that \"community
    based\" standards are more closely followed than standards imposed
    by outside forces.

5.  Each community\'s interchange format profile *must* be as specific
    as possible to eliminate differences between data products and allow
    for the generation of simple data packaging tools.

6.  If the HDF-EOS geolocation makes sense for a particular community,
    it *must* develop its interchange format profile based on this
    standard.

7.  Each community *should* review other Near-term mission\'s community
    interchange format profiles and incorporate sections of overlap in
    their profile.

8.  The interchange formats *may* be used as distribution formats.

##### Rationale {#rationale-1 .unnumbered}

An interchange packaging standard among PI-managed mission-data centers
and other ESE data systems components ensures that data are completely
and correctly transferred. Use of standards for this interchange
increases the flexibility of the ESE data systems. New components can
join with the ESE data systems to provide data services without
negotiating one-to-one interface agreements with each potential
provider. The effect of using standard packaging methods will result in
decreasing the complexity of the ESE data systems as a whole while
increasing potential for participation and novel use of NASA Earth
systems science data sets. A community-involved process for approving or
developing these data packaging standards ensures that the standards are
appropriate and reliable. The choice of interchange packaging standards
must consider completeness and correctness of representing data and
emphasize the self-descriptiveness and long-term stability of the
standards.

While HDF/HDF-EOS and netCDF, with a community-developed profile, are
not the only possible candidates for an interchange standard, at this
time, NTMS finds that they are the best choices based on our study (See
Chapter 5). In fact, HDF/HDF-EOS is the most commonly used data
packaging standard in the heritage missions. ESML (Earth Science Markup
Language) has the potential to provide a level of data description to
enable interchange packaging; however, NTMS finds this technology is not
yet sufficiently mature to recommend. There are certainly other choices
for standard interchange packaging as well.

NTMS finds that it is unlikely that certain data format standards will
fill the needs of a center-to-center data interchange standard. For
example, GeoTIFF is a good distribution standard for georeferenced
imagery to end-users, but it is incapable of conveying the full metadata
needed for the near-term missions or handling non-image data such as
atmospheric profiles. BUFR and GRIB are WMO standards designed for
dissemination of weather station data and for the output of numerical
weather prediction models. BUFR and GRIB may be used as distribution
formats for ocean or atmosphere data products used by weather prediction
modelers, but they are not suitable as interchange standards as they
lack self-describing power, tool support, and other criteria (see
Chapter 5).

Finally, some near-term missions have heritage in, or are considering
the use of, custom (*a.k.a.,* binary) data formats. We recommend that
while it may be appropriate to use custom formats for internal mission
science work, and certain communities may find them appropriate for
distribution packaging, such formats are unlikely to be acceptable as an
interchange standard.

## 7.2 Metadata Standards Recommendations {#metadata-standards-recommendations .unnumbered}

1.  Metadata standards are converging on the ISO 19115 standard, as FGDC
    will adopt the ISO 19115 after it becomes final in 2002. We
    recommend ISO19115 as the metadata standard for the near-term
    missions. However, since the ISO19115 is not finalized yet and the
    tool support for it has yet to be developed, we recommend the
    following implementation strategy.

2.  Data systems for near-term missions *must* be compliant with the
    EOSDIS Clearing HOuse (ECHO) metadata model, which will be used for
    the advertising and distribution of data from "Backbone" Processing
    Centers, Science Data Centers, Multi-mission Data Centers, and
    PI-managed Missions Data Centers. The ECHO metadata model is an XML
    implementation and a superset of the ECS data model. It provides a
    capability to map metadata into various content-equivalent
    representations. With the single investment of ECHO
    interoperability, near-term missions will be insulated from most
    metadata standards changes and will benefit from new ECHO interfaces
    as they become available.

3.  The ECHO implementation organization *should* be tasked with
    monitoring, developing, and maintaining metadata mapping capability
    between ECHO holdings and emerging FGDC remote sensing profiles of
    the ISO 19115 standard.

4.  We recommend continuing the use of the GCMD as a catalog (or
    collection) metadata standard for the near-term missions. We further
    recommend that the GCMD *should* coordinate with ECHO to implement
    seamless, automated interoperability of data products and services
    so that data centers do not need to prepare collection (or dataset)
    level metadata separately from inventory-level metadata.

## 7.3 Documentation Standards Recommendations {#documentation-standards-recommendations .unnumbered}

1.  The Guide standard *should* be maintained, and a community-based
    process for incorporating the Guide into the ECHO data model
    *should* be developed. The EDG Guide data set documentation standard
    is successful and generally adequate for minimal description of
    standard data products. However, the division of metadata between
    the EOSDIS Earth Science Data Model and the EOSDIS Guide Document
    appears arbitrary and is, we believe, a hindrance to the effective,
    efficient, and accurate discovery and use of EOS data.

2.  Algorithm Theoretical Basis Documents (ATBDs) are written by EOS
    scientists for every EOS instrument product. There are no detailed
    specifications for ATBDs, only a suggested outline that includes
    theoretical background, algorithm description, and validation plans.
    The ATBD is typically referenced in the Guide document. The ATBDs
    *should* be permanently accessible with a stable web address so that
    links to these documents in the Guide documents will remain valid

3.  NASA *should* perform a detailed analysis of the emerging XML-based
    documentation standards in the social and library/archive sciences.
    The purpose of performing a detailed analysis of the emerging
    XML-based documentation standards in the social and library/archive
    sciences is to: 1) borrow from their XML-based data models which
    combine free-text descriptions and constrained element lists in a
    hierarchical, cross-referenced fashion; 2) maximize interoperability
    with information systems outside of Earth Sciences; and 3) benefit
    from the extensive work that has been done defining the
    documentation required for long-term preservation of knowledge about
    digital objects. We are specifically referring to the Data
    Documentation Initiative (DDI) in the Social Sciences, the Metadata
    Encoding and Transmission Standard (METS) maintained by the Library
    of Congress, and the Reference Model for an Open Archival
    Information System (OAIS) developed by the Council of the
    Consultative Committee for Space Data Systems.

## 7.4 Catalog Interface Standards Recommendations {#catalog-interface-standards-recommendations .unnumbered}

1.  We recommend continuing the use of catalog interface standards for
    collection, inventory, and document search using the EOSDIS V0
    protocol or Z39.50 based standards such as GEO or CIP.

2.  Web Service standards, such as OGC Web Mapping Service, Web Coverage
    Service, and Web Registry Service standards, will have an impact on
    catalog interface standards in the future. SEEDS *should*
    direct/track developments in the science and business communities.

## 7.5 Standard Evolution Process and Other Activities Recommendations {#standard-evolution-process-and-other-activities-recommendations .unnumbered}

1.  For Earth systems science systematic measurements including EOS
    data, weather data, atmospheric and oceanographic modeling data, and
    land use/land processes data, there are several data packaging or
    API standards that have similar purposes. ESE *should* invest
    resources in guiding the evolution of these data formats through
    their respective governing processes with the goal of harmonizing
    them toward seamless interoperability. We recommend that particular
    attention be focused on guiding the evolution of netCDF,
    HDF/HDF-EOS, geoTIFF and the WMO BUFR and GRIB formats

2.  SEEDS *should* empanel a Standards and Interfaces Evolution Process
    Working Group for developing and executing a plan for evolution of
    interchange packaging standards over the life of data sets. The
    benefits of adopting additional interchange packaging standards need
    to be weighed against the increase in cost and complexity that will
    occur with the addition of each new packaging standard.

3.  Near-term missions *must* plan for evolution of end user
    requirements for packaging of mission science data (including data
    distribution packaging formats, data distribution system interface,
    and metadata) over the lifetime of the missions.

4.  The evolution of the interchange packaging standards *must* keep
    pace with technology and market trends and emerging expectations
    (see lessons learned from SDTS in Chapter 3). SEEDS *should* adopt
    new technology as it develops. The number of interchange packaging
    standards *should* be limited and as closely related to each other
    as is practical.

5.  SEEDS *should* coordinate respective activities to support the
    near-term missions such as interchange packaging standards
    maintenance and translation tools development/maintenance. This
    group *should* also provide interchange packaging standards user
    training and help desk support to educate producers/consumers/tool
    vendors.

6.  The development of conversion software for data distribution formats
    *should* be a separately funded task and the responsibility for this
    development *should not* necessarily fall upon the mission science
    teams.

#   {#section .unnumbered}

# Acronym List

  ----------- -----------------------------------------------------------
  ACRIM       Active Cavity Radiometer Irradiance Monitor

  ACRIMSAT    Active Cavity Radiometer Irradiance Monitor SATellite

  ADEOS       Advanced Earth Observing Satellite

  ADS         Archive and Distribution Segment

  AGS         Alaska Ground Station

  AIRS        Atmospheric Infrared Sounder

  AIX         IBM\'s UNIX Operating System

  ALI         Advanced Land Imager

  ALT         Dual-Frequency Radar Altimeter

  AMI         Active Microwave Instrument

  AMSR        Advanced Microwave Scanning Radiometer

  AMSU        Advanced Microwave Sounding Unit

  ANZLIC      Australia New Zealand Land Information Council

  APAS        Astrophysical, Planetary, and Atmospheric Sciences
              Department

  API         Application Platform Interface

  APID        Applications Package Identification

  ASCI        Accelerated Strategic Computing Initiative

  ASPS        AIRS Science Processing System

  ASTER       Advanced Spaceborne Thermal Emission And Reflection
              Radiometer

  ATMOS       Atmospheric Observations Satellite

  ATMS        Advanced Technology Microwave Sounder

  AVHRR       Advanced Very High Resolution Radiometer

  AVISO       Validation and Interpretation of Satellites Oceanographic
              data

  AVS         Advanced Visual Systems

  BOREAS      Boreal Ecosystem-Atmosphere Study

  BSQ         Band Sequential

  BUFR        Binary Universal Format For Representation \[Of Data\]

  C3S         Command Control & Communication Segment

  CAP         Cooperative Agreements Program

  CARS        Climate Analysis and Research

  CASE        Computer Aided Software Engineering

  CBS         Commission for Basic Systems

  CCI         Carbon Cycle Initiative

  CCIWG       carbon cycle interagency working group

  CCS         Climate Calibration Segment

  CDF         Common Data Format

  CDHF        Central Data Handling Facility

  CDL         Common Data Form Language (used by netCDF

  CDMS        Climate Data Management Segment

  CDR         Climate Data Record

  CEOS        Committee on Earth Observation Satellites

  CERES       Clouds and the Earth\'s Radiant Energy System

  CI          Catalog Interoperability

  CIP         Catalog Interoperability Protocol

  CLAES       Cryogenic Limb Array Etalon Spectrometry

  CMIS        Conical Microwave Imager/Sounder

  CMS         Climate Mission Storage System

  CNES        Centre National D\'etudes Spatiales (France)

  CNIDR       Clearinghouse for Networked Information Discovery and
              Retrieval

  COTS        Commercial Off-The-Shelf

  CPF         Calibration Parameter File

  CPOZ        Compressed Ozone

  CrIS        Cross-Track Infrared Sounder

  CRTT        Calibrated Radiance and Temperature Tape

  C-SAFS      Central Standard Autonomous File System

  CSDGM       Content Standard for Digital Geospatial Metadata

  CZCS        Coastal Zone Color Scanner

  DAAC        Distributed Active Archive Center

  DAO         Data Assimilation Office

  DC          Dublin Core

  DEM         Digital Elevation Model

  DFD         Deutsches Fernerkundungsdatenzentrum (German Remote Sensing
              Data Center)

  DFR         Dual Frequency Radar

  DIAL        Data and Information Access Link

  DIF         Directory Interchange Format

  DLL         Dynamic Link Library

  DLT         Digital Linear Tape

  DMF         Data Models and Formats

  DMSP        Defense Meteorological Satellite Program

  DoD         Department of Defense

  DODS        Distributed Oceanographic Data System

  DOE         Department of Energy

  DOMSAT      Domestic Satellite

  DOQQ        Digital Orthophoto Quarter Quads

  DORIS       Doppler Orbitography And Radiopositioning Integrated By
              Satellite

  DOS         Disk Operating System

  D-PAF       German Processing and Archiving Facility

  DPR         Dual-frequency Precipitation Radar

  DPS         Data Processing System

  DRFP        Draft Request for Proposal

  DRG         Digital Raster Graphics

  DSP         Directory Service Protocol

  DSWG        Data System Working Group

  DTD         Document Type Definition

  EA          Engineering Analysis

  ECHO        EOS Clearing House

  ECMWF       European Centre for Medium-Range Weather Forecasts

  ECS         EOSDIS Core System

  EDC         EROS Data Center

  EDG         Earth Observing Systems (EOS) Data Gateway

  EDOS        EOS Data and Operations System

  EDR         Environmental Data Record

  EMOS        ECS Mission Operations System

  ENSO        El Niño/Southern Oscillation

  ENVISAT     Environmental Satellite

  EO-1        Earth Orbiting Satellite #1

  EOC         EOS Operations Center

  EOS         Earth Observing System

  EOSAT       Earth Observation Satellite

  EOSDIS      Earth Observing System Data and Information System

  EP          Earth Probe

  EPA         Environmental Protection Agency

  EPSG        European Petroleum Survey Group

  EROS        Earth Resources Observation System

  ERS         Earth Resources Satellite

  ERSDAC      Earth Remote Sensing Data Analysis Center (Japan)

  ERTS        Earth Resource Technology Satellite (later renamed Landsat
              1 (Land Saltellite?)

  ESA         European Space Agency

  ESC         Engineering Support Center

  ESCAT       ESA scatterometer

  ESDIS       Earth Science Data and Information System

  ESDT        Earth Sciences Data Type

  ESE         Earth Science Enterprise

  ESIP        Earth Science Information Partner

  ESIPS       EOSDIS Science Investigator-Led Processing System

  ESRI        Environmental Systems Research Institute (GIS software
              company)

  ETM         Enhanced Thematic Mapper

  EUV         Extreme Ultraviolet

  FAST-L7A    FAST-Landsat 7 Format

  FDF         Flight Dynamics Facility

  FGDC        Federal Geographic Data Committee

  FIFE        First ISLSCP Field Experiment

  FIPS        Federal Information Processing Standard

  FMI         Finnish Meteorological Institute

  FTP         File Transfer Protocol

  FX          File Transfer Subsystem

  GAC         Global-Area Coverage

  GCCP        Global Carbon Cycle Program

  GCMD        Global Change Master Directory

  GCTE        Global Change and Terrestrial Ecosystems

  GDAAC       Goddard Distributed Active Archive Center

  GDAL        Geospatial Data Abstraction Library

  GDR         Geophysical Data Record

  GEO profile Geospatial Metadata Application

  GeoTIFF     Georeferenced Tagged Image File Format

  GES         Goddard's Earth Sciences

  GHRC        Global Hydrology Resource Center

  GILS        Global Information Locator Service

  GIS         Geographic Information System

  GIVIT       Granule Insert Validation and Inspection Tool

  GLAS        Geoscience Laser Altimeter System

  GLOBEC      Global Ocean Ecosystem Dynamics

  GMT         Greenwich Mean Time

  GOES        Geostationary Operational Environmental Satellites

  GOFC        Global Observation of Forest Cover

  GOME        Global Ozone Monitoring Experiment

  GPM         Global Precipitation Measurement

  GRACE       Gravity Recovery And Climate Experiment

  GRASS       Geographic Resources Analysis Support System (public domain
              software)

  GRiB        GRidded Binary

  GRS-1       Generic Record Syntax

  GSFC        Goddard Space Flight Center

  GUI         Graphical User Interface

  HAO         High Altitude Observatory

  HDF         Hierarchical Data Format

  HDF-EOS     HDF Earth Observing System (EOS) format

  HDSBUV      High-Density Solar Backscatter Ultraviolet Instrument
              (SBUV)

  HE4         HDF-EOS based on HDF version 4

  HE5         HDF-EOS based on HDF version 5

  HIRDLS      High-Resolution Dynamics Limb Sounder

  HRDLS       High-Resolution Dynamics Limb Sounder

  HRPT        High Resolution Picture Transmission

  HSB         Humidity Sensor of Brazil

  HTTPD       Hyper Text Transfer Protocol Daemon

  IAS         Image Assessment System

  ICD         Interface Control Document

  ICESat      Ice, Cloud, And Land Elevation Satellite

  IDL         Interactive Display Language

  IDN         International Directory Network

  IDPS        Interface Data Processing System

  IETF        Internet Engineering Task Force

  IFOV        instantaneous field of view

  IGBP        International Geosphere And Biosphere Research Program

  IGDR        Interim Geophysical Data Record

  IGS         International Ground Stations

  IMS         Information Management System

  IPD         Information Processing Division

  IPO         Integrated Program Office

  ISCCP       International Satellite Cloud Climatology Project

  ISLSCP      International Satellite Land Surface Climatology Project

  ISO         Greek prefix "iso" as used by the International
              Organization for Standardization

  ITSS        Information Technology and Scientific Services

  IWGDMGC     Interagency Working Group on Data Management for Global
              Change

  JAI         Java Advanced Imaging

  JEB         Java EOS Browser

  JERS        Japanese Earth Resources Satellite

  JMR         Jason Microwave Radiometer

  JPL         Jet Propulsion Laboratory

  KLM         NOAA K-, L-, M- system

  KNMI        Koninklijk Nederlands Meteorologisch Instituut
              (Netherlands)

  LAC         local-area coverage

  LaRC        Langley Research Center

  LASP        Laboratory for Atmospheric and Space Physics at the
              University of Colorado

  LBA         Large-Scale Biosphere-Atmosphere Experiment

  LDCM        Landsat Data Continuity Mission

  LGN         Landsat Ground Network

  LGS         Landsat Ground Station

  LIS         Lightning Imaging Sensor

  LP          Level Processor

  LPDS        Level 1 Product Distribution System

  LPGS        Level 1 Product Generation System

  LPS         Landsat Processing System

  LTER        Long-Term Ecological Research

  MBLA        Multi-Beam Laser Altimeter

  MCF         Metadata Configuration File

  METI        Ministry Of Economy Trade And Industry (Japan)

  MFLOPS      Millions of Floating Point Operations per Second

  MINC        Medical Image netCDF

  MISR        Multi-Angle Imaging Spectro-Radiometer

  MIT         Massachusetts Institute of Technology

  MLE         Maximum Likelihood Estimator

  MLS         Microwave Limb Sounder

  MOBY        Marine Optical Buoy data

  MOC         Missions Operations Center

  MODAPS      MODIS Adaptive Processing System

  MODIS       Moderate-Resolution Imaging Spectroradiometer

  MOPITT      Measurements Of Pollution In The Troposphere

  MSCD        Mirror Scan Correction Data

  MSFC        Marhsall Space Flight Center

  MSS         Multispectral Scanners

  MTMGW       Machine-To-Machine Search and Order Gateway

  MUSE        Multi-User Science Environment

  NASA        National Aeronautics and Space Administration

  NASDA       National Space Development Agency (Japan)

  NCAR        National Center for Atmospheric Research

  NCDC        National Climatic Data Center

  NCEP        National Centers for Environmental Prediction

  NCL         NCAR Command Language

  NCSA        National Center for Supercomputing Applications

  NDVI        Normalized Differential Vegetation Index

  NESDIS      NOAA/National Environmental Satellite, Data, and
              Information Service

  netCDF      Network Common Data Format

  NIVR        Netherlands\'s Agency for Aerospace Programs

  NMC         National Meteorological Center

  NOAA        National Oceanic and Atmospheric Administration

  NODC        NOAA National Oceanographic Data Center

  NPOESS      National Polar Orbiting Environmental Satellite System

  NPP         NPOESS Preparatory Project

  NRL         Naval Research Laboratory

  NRT         Near Real-Time

  NSCAT       NASA scatterometer

  NSF         National Science Foundation

  NSIDC       National Snow and Ice Data Center

  NSSDC       National Space Science Data Center

  NTMS        Near-Term Missions Standards

  NTMS        Near-Term Missions Study

  NWP         Numerical Weather Product

  NWS         National Weather Service

  ODL         Object Description Language

  OFL         Off-Line

  OGC         OpenGIS Consortium

  OMB         Office of Management and Budget

  OMI         Ozone Mapping Instrument

  OMPS        Ozone Mapping and Profiler Suite

  ORNL        Oak Ridge National Labs

  OSDPD       Office of Satellite Data Processing and Distribution

  OSDR        Operational Sensor Date Record

  OSF         Observation Schedule File

  OSSIM       Open Source Software Image Map

  OSTM        Ocean Surface Topography Measurement

  OTTER       Oregon Transect Ecosystem Research

  PCD         Payload Correction Data

  PCF         Process Control File

  PCI         GIS software by PCI Geomatics

  PDPS        Planning and Data Processing System

  PDR         Product Delivery Record

  PDS         Precipitation Data System

  PGE         Product Generation Executive

  PI          Principal Investigator

  PMEL        Pacific Marine Environmental Laboratory

  PNG         Portable Network Graphics

  PO.DAAC     Physical Oceanography DAAC

  POES        Polar-Orbiting Environmental Satellite

  POSC        Petrotechnical Open Software Corporation

  PP          Pre-Processors

  PPLUS       graphics package by Plot Plus Graphics

  PR          Precipitation Radar

  PSA         Product Specific Attribute

  PV-wave     Visualization package by Visual Numerics, Inc.

  QAMUT       Quality Assurance Metadata Update Tool

  QuickScat   Quick Scatterometer

  QuikTOMS    Quick TOMS

  RBV         Return-Beam Vidicons

  RDBMS       relational database management system

  RDF         Resource Description Framework

  RDR         Raw Data Record

  RFP         Request for Proposal

  RGB         Red, Green, Blue

  RSI         Research Systems, Inc.

  RSS         Remote Sensing Systems

  S4P         Scalable Script-Based Science Processor

  SA          Science Analysis

  SAA         Satellite Active Archive

  SAGE        Stratospheric Aerosols and Gas Experiment

  SAIC        Science Applications International Corporation

  SAN         Storage Area Network

  SAR         Synthetic Aperture Radar

  SASS        Seasat-A scatterometer system

  SBUV        Solar Backscatter Ultraviolet Instrument

  SCF         Science Computing Facility

  SCIAMACHY   SCanning Imaging Absorption SpectroMeter for Atmospheric
              ChartographY

  SCLI        Science data server Command-Line Interface

  SDP         Science Data Processing

  SDPS        SeaWifs Data Processing System

  SDS         Science Data Segment

  SDSRV       Science Data Server

  SDTS        Spatial Data Transfer Standard

  SeaPAC      SeaWinds Processing and Analysis Center

  SeaWiFs     Sea-viewing Wide Field-of-view Sensor

  SEE         Solar EUV Experiment

  SEEDS       Strategic Evolution of ESE science Data and information
              Systems

  SERF        Service Entry Resource Format

  SFDU        Standard Formatted Data Unit

  SGDR        Sensor Geophysical Data Record

  SGI         Silicon Graphics, Inc.

  SGML        Standard Generalized Markup Language

  SGS         Svalbard Norway Ground Station

  SIGF        Solar Irradiance Gap Filler

  SIM         Spectral Irradiance Monitor

  SIPS        Science Investigator Processing System

  SMI         Standard Mapped Image

  SMMR        Scanning Multichannel Microwave Radiometer

  SMMS        Spatial Metadata Management System

  SNOE        Student Nitrous Oxide Experiment

  SOLSTICE    Solar Stellar Irradiance Comparison Experiment

  SORCE       SOlar Radiation and Climate Experiment

  SPOT        Systeme Pour l\'Observation De La Terre (France)

  SPS         Science Processing System

  SQL         Structured Query Language

  SRM         Subscription Request Manager

  SRTM        Shuttle Radar Topography Mission

  SSALT       Single-Frequency Solid-State Radar Altimeter

  SSM/I       Special Sensor Microwave/Imager

  SUTRS       Simple Unstructured Text Record Syntax

  SWIR        Short Wave Infrared

  TBD         To Be Determined

  TDI         Transport Data Interface

  TES         Tropospheric Emission Spectrometer

  TIFF        Tagged Image File Format

  TIM         Total Irradiance Monitor

  TIMED       Thermosphere Ionosphere Mesosphere Energetics and Dynamics

  TIROS       Television Infrared Observation Satellite

  TM          Thematic Mapper

  TMI         TRMM Microwave Imager

  TMR         TOPEX Microwave Radiometer

  TOMS        Total Ozone Mapping Spectrometer

  TOPEX       Topography (Ocean) Experiment

  TOVS        Tiros Operational Vertical Sounder

  TRMM        Tropical Rainfall Measurement Mission

  TSDIS       TRMM Science Data and Information System

  TSIS        Total Solar Irradiance Sensor

  TSU         TSDIS Science User

  UARS        Upper Atmospheric Research Satellite

  UCSS        UARS CDHF Software System

  UDDI        Universal Description, Discovery, and Integration

  UFM         User Friendly Metadata

  UML         Universal Modeling Language

  USDA        United States Department of Agriculture

  USFS        USDA Forest Service

  USGCRP      US Global Change Research Program

  USGS        United States Geological Survey

  USMARC      U.S. Machine Readable Cataloging

  UTM         Universal Transverse Mercator

  VCL         Vegetation Canopy Lidar

  VDC         Visual Database Cookbook

  VIIRS       Visible/Infrared Imager/Radiometer Suite

  VIRS        Visible and Infrared Scanner

  VNIR        Visible And Near Infrared

  W3C         World Wide Web Consortium

  WAIS        Wide Area Information System

  WGISS       Working Group on Information Systems and Services

  WHO         World Health Organization

  WIM         Windows Image Manager

  WMO         World Meteorological Organization

  WNS         Wind Scatterometer

  WOCE        World Ocean Circulation Experiment

  WRS         Worldwide Reference System

  WXP         Weather Processor

  XDR         eXternal Data Representation

  XML         eXtensible Markup Language

  XPS         XUV Photometer System

  XSL         eXtensible Stylesheet Language

  XSLT        XSL Transformations

  ZMT         Zonal Mean Tape
  ----------- -----------------------------------------------------------

[^1]: Any tool that reads a TIFF file should also read a GeoTIFF file
    (though most will complain about the extra "unsupported" tags). The
    packages listed here are those that use the additional information
    contained in a GeoTIFF file to geolocate the data.