**Earth Science Data Information System Project**

> **423-10-69**

Requirements for Archiving,

Distribution and User Services

in EOS Data and Information System (EOSDIS)

> **April 2006**
>
> **Expires: April 2011**
>
> ![](media/image1.wmf)

This page intentionally left blank.

**Requirements for Archiving, Distribution and User Services in EOS Data
and Information System (EOSDIS)**

**April 2006**

Reviewed by:

+---------------------------------+-----+------------------------------+
|                                 |     |                              |
+---------------------------------+-----+------------------------------+
| H.K. Ramapriyan                 |     | Date                         |
|                                 |     |                              |
| Assistant Project Manager       |     |                              |
|                                 |     |                              |
| GSFC -- Code 423                |     |                              |
+---------------------------------+-----+------------------------------+

**Approved by:**

+---------------------------------+-----+------------------------------+
|                                 |     |                              |
+---------------------------------+-----+------------------------------+
| Mary Ann Esfandiari             |     | Date                         |
|                                 |     |                              |
| ESDIS Project Manager           |     |                              |
|                                 |     |                              |
| GSFC - Code 423                 |     |                              |
+---------------------------------+-----+------------------------------+

Goddard Space Flight Center

Greenbelt, Maryland

This page intentionally left blank.

# Change Record Page {#change-record-page .Heading-No-Number}

  ----------- ---------- ------------------------ ------------------------
  ISSUE       DATE       PAGES AFFECTED           DESCRIPTION

  Baseline    TBD        All                      

                                                  

                                                  

                                                  

                                                  

                                                  

                                                  

                                                  

                                                  

                                                  

                                                  

                                                  

                                                  

                                                  

                                                  

                                                  

                                                  

                                                  

                                                  

                                                  

                                                  

                                                  

                                                  

                                                  

                                                  

                                                  

                                                  

                                                  

                                                  

                                                  

                                                  

                                                  

                                                  

                                                  
  ----------- ---------- ------------------------ ------------------------

#  This page intentionally left blank. 

# List of Affected Pages {#list-of-affected-pages .Heading-No-Number}

  -------- ---------- -------- ---------- -------- ---------- -------- ----------
  Page No. Revision   Page No. Revision   Page No. Revision   Page No. Revision

  Title    Original                                                    

  i        Original                                                    

  ii       Original                                                    

  iii      Original                                                    

  iv       Original                                                    

  v        Original                                                    

  vi       Original                                                    

  vii      Original                                                    

  viii     Original                                                    

  ix       Original                                                    

  x        Original                                                    

  1-1      Original                                                    

  1-2      Original                                                    

  2-1      Original                                                    

  2-2      Original                                                    

  2-3      Original                                                    

  2-4      Original                                                    

  2-5      Original                                                    

  2-6      Original                                                    

  2-7      Original                                                    

  2-8      Original                                                    

  A-1      Original                                                    

  A-2      Original                                                    

  AB-1     Original                                                    

  AB-2     Original                                                    

                                                                       

                                                                       

                                                                       

                                                                       

                                                                       

                                                                       

                                                                       

                                                                       

                                                                       
  -------- ---------- -------- ---------- -------- ---------- -------- ----------

#  This page intentionally left blank. 

Abstract

The purpose of this document is to provide generic requirements for data
archiving, data distribution and user services for EOSDIS-supported
data. These services are currently being provided by the Distributed
Active Archive Centers (DAACs), even though some of the Science
Investigator-led Processing Systems (SIPSs) are providing these services
to a limited extent.

This document is under configuration control. The Earth Science Data
Information System (ESDIS) project is responsible for processing changes
to it. Changes to this document will be made by a Document Change Notice
(DCN) or by complete revision.

Questions concerning this document should be addressed to:

Configuration Management Office\
ESDIS, Code 423\
Goddard Space Flight Center\
Greenbelt, Maryland 20771

# This page intentionally left blank.

Contents

Change Record Page

List of Affected Pages

Section 1. Scope and Assumptions

Section 2. Requirements Specification for Archive, Distribution and User
Services

2.1 Overview 2-[1](#overview)

2.2 Data Ingest 2-[1](#data-ingest)

2.3 Data Archive and Distribution Function
2-[2](#data-archive-and-distribution-function)

2.3.1 Overview 2-[2](#overview-1)

2.3.2 Data Archive 2-[2](#data-archive)

2.3.3 Storage Management 2-[3](#storage-management)

2.3.4 Data Distribution 2-[3](#data-distribution)

2.3.5 Performance 2-[4](#performance)

2.3.6 Operational Requirements 2-[4](#operational-requirements)

2.4 Information Management 2-[5](#information-management)

2.5 User Services 2-[6](#user-services)

2.6 Information Security 2-[6](#information-security)

Appendix A. Performance Metrics in Use by ESDIS Project

**Abbreviations and Acronyms**

This page intentionally left blank.

# Section 1. Scope and Assumptions {#section-1.-scope-and-assumptions .Heading-No-Number}

The purpose of this document is to provide generic requirements for data
archiving, data distribution and user services for EOSDIS-supported
data. These services are currently being provided by the Distributed
Active Archive Centers (DAACs), even though some of the Science
Investigator-led Processing Systems (SIPSs) are providing these services
to a limited extent. Rather than refer to specific DAACs or SIPSs, this
document refers to an entity providing such services as an "X Data
Center (XDC)". The data system that provides the functionality to
provide the services is referred to as an "X Data System (XDS)". A
generic Data Production System (DPS) is discussed here only in its
relation to the archive function of an XDS. There is no implication here
regarding the collocation of an XDS and a DPS. It is assumed that the
reader is familiar with Earth Observing System (EOS) terminology.

The intent of this document is to specify "what" requirements as opposed
to any implied implementation or "how".

This document does not include discussion of network requirements and
communications systems. Security requirements are covered by
pre-existing security plans tracked by the ESDIS Computer Security
Officer.

This page intentionally left blank.

# Section 2. Requirements Specification for Archive, Distribution and User Services {#section-2.-requirements-specification-for-archive-distribution-and-user-services .Heading-No-Number}

# 2.1 Overview

The XDS has the responsibility to meet the objectives of the EOS mission
in data product, archiving, distribution, and search and ordering.

Data to be archived are ingested from one or more data production
systems (DPS) and various other EOS and external data providers, as
necessary, through interfaces described in specific Interface Control
Documents (ICDs). Requirements to archive and distribute the data for a
given mission extend four years beyond its termination date. Beyond that
four-year period, Long-Term Archiving agencies are responsible to
archive and distribute the data. XDCs will need to ensure that the data
are transitioned to the appropriate Long-Term Archive (LTA) when
National Aeronautics and Space Administration (NASA) notifies the XDC
that it is appropriate to do so. The XDS has a search and
order/request/access function that 1) provides users with information
about the available data products (e.g., guide documents), 2) gives
users the capability to identify and select their desired information
and data products before ordering/requesting/accessing, and 3) delivers
the requested products to users. The XDS shall provide users with
functionalities to manipulate certain data products prior to ordering
(e.g., spatial and/or spectral subsetting). The XDC shall provide a User
Services function to assist users with questions, for example, regarding
data formats, data usage, system access, etc. While the XDC will
probably be staffed only on a 40-hour work-week schedule, ESDIS expects
the XDS to be available every day on a 24 hour basis with the exceptions
of planned downtime.

In addition, the XDC provides the ESDIS Project metrics related to data
ingesting, data archiving, data distribution, and user satisfaction. To
facilitate integrated reporting to NASA Headquarters, the ESDIS Project
management determines the specific metrics information to be gathered
based on the particular needs of the mission and the detailed
information needed to monitor performance of the XDC. ESDIS Project is
currently refining the metrics collection process and developing an
interface control document to define the interfaces between XDSs and the
EOSDIS Metrics System (EMS). Appendix A shows the metrics that the
Project routinely uses.

# 2.2 Data Ingest

1\. The XDS shall ingest data from each provider in accordance with the
interface specification documented in the applicable Interface Control
Document (ICD).

2\. The XDS shall be capable of ingesting data from multiple providers.

3\. The XDS shall ensure ingest data integrity through the use of
checksums in order to satisfy interface requirements with external
systems.

4\. The XDS shall verify the quality of the data upon ingest (e.g. data
set size, data set name) for each unique data set.

5\. The XDS shall provide ingest metrics to the ESDIS Metrics System as
described in the Appendix.

# 2.3 Data Archive and Distribution Function

Each XDC has the responsibility for archiving and distribution of a
subset of EOS mission-related data products. This includes EOS standard
and special mission output products, metadata, and any
ancillary/auxiliary or correlative data sets necessary for the
production and validation of those output products, as well as data set
documentation. Data sources may include EOS Data and Operations System
(EDOS), Flight Dynamics System (FDS), other XDCs, and/or external data
providers, as appropriate.

## 2.3.1 Overview

The XDC archives designated data products and distributes them on
request to users. Some products will be created for distribution through
an on-demand processing system and are subject to the same delivery
requirements as products that are stored in the archive. The XDC
distributes data electronically over designated Mission Support Networks
and Science Support Networks. Electronic data transmission must comply
with all applicable NASA security standards.

## 2.3.2 Data Archive

1.  The XDS shall store all designated data products or create products
    on demand. The XDC shall assure that products generated on demand
    (virtual products) are identical to the corresponding standard
    products that would otherwise be archived, having undergone
    operational and scientific quality assessment. Designated products
    will be established by the NASA Headquarters (HQ) EOSDIS Program
    Executive for each XDC and listed in a separate document to be
    provided to the ESDIS project by the XDC.

2.  The XDS shall have the capability to archive multiple versions of
    selected archive data.

3.  The XDC shall maintain metadata on all products that are available
    from the XDS. The metadata content shall be compliant with the
    Federal Geographic Data Committee (FGDC) standard is specified in
    the document titled "Content Standard for Digital Geospatial
    Metadata: Extensions for Remote Sensing Metadata,
    FGDC-STD-012-2002."

4.  The XDC shall maintain archiving, distribution and user services
    functions for designated data products until 4 years beyond the end
    of flight operations for the corresponding platform or platforms.

5.  The XDC shall be able to recover Level 0 data (if necessary) from
    the ESDIS Project's off-site back-up archive in White Sands, NM.

6.  The XDC shall maintain an off-site backup copy of all other data
    which would be impossible or difficult to recover in case of loss
    (e.g., ancillary data, metadata, command history, Science Software,
    engineering data, calibration data, systems and applications
    software, selected data products, depending on need).

7.  The XDS shall have the capability to restore its archive to avoid
    permanent loss of archive data.

8.  The XDS shall be able to create and refresh archive media and system
    components as needed without impacting operations.

9.  The XDS shall allow for new technology integration and scaleable
    systems for archival data.

10. The XDS shall allow old versions of data to be removed from the
    archive.

11. The XDS shall provide archive metrics to the ESDIS Metrics System.
    The metrics are described in the Appendix to this document.

## 2.3.3 Storage Management 

1.  The XDS shall maintain an electronic inventory of all archived data.

2.  The XDS shall provide the capability of retrieving any data granule
    stored in the archive.

## 2.3.4 Data Distribution

1.  The XDS shall distribute to users, upon request, standard and
    special data products, metadata, ancillary/auxiliary data,
    calibration data, science software, browse data, and documentation.

2.  The XDS shall be capable of responding to users' requests for data.
    The XDS shall be capable of providing subscriptions to datasets.

3.  On an XDS that supports an order-based system, the XDS shall provide
    automated status information to users regarding their data orders.

4.  The XDS shall distribute data in standard formats commonly accepted
    within the user community for the types of data for which the XDC is
    responsible.

5.  The XDS shall distribute data to users via electronic networks.

6.  The XDS shall send data to the DPS and instrument teams' Science
    Computing Facilities (SCFs) to support product generation and
    quality assurance in a timely manner to support production
    schedules.

7.  The XDS shall provide subsetted data and/or subsetting, reprojection
    and format conversion tools appropriate to the XDC's data holdings
    to ensure efficient distribution of data to users.

8.  The XDS shall provide distribution metrics to the ESDIS Metrics
    System. The metrics are described in the Appendix to this document.

9.  The XDS shall ensure data integrity through the use of checksums on
    distribution in order to satisfy interface requirements with
    external systems.

10. The XDS shall make available for distribution standard metadata
    using appropriate tagging (e.g., extensible markup language \[XML\],
    FGDC-standard).

11. The XDS shall use common distribution notification (DN) messages
    following the example of the EOSDIS Core System DN. (Several SIPSs
    currently rely on the ECS distribution notice as part of their
    science data processing systems.)

## 2.3.5 Performance 

The performance requirements are based on current performance at the
DAACs. The ESDIS Project conducts an annual survey of users to obtain
the "American Consumer Satisfaction Index (ACSI)" for the services users
receive from the EOSDIS. For fiscal year (FY) 2005, the ACSI was 78. The
ACSI is a metric that NASA reports annually to the Office of Management
and Budget (OMB). The goal of the performance requirements below is to
maintain or increase the ACSI. The XDC's may propose alternative
performance measures that achieve this goal, depending on their specific
system implementations.

1.  The XDS shall be capable of receiving and archiving the data
    products resulting from forward processing of data simultaneously
    with those resulting from reprocessing.

2.  The XDS shall make the data available for electronic pickup by the
    user for at least 24 hours.

3.  The XDS shall be capable of transmitting data over communications
    network(s) in support of: data distribution requests at a rate
    equivalent to the daily designated product archive volume (per 24
    hour period) for the data products for which the XDC is responsible.

## 2.3.6 Operational Requirements

1\. The XDC shall notify ESDIS of off-nominal operational instances by
the next business day of the occurrence. Off-nominal occurrences include
unanticipated downtime of 8 hours, or severely compromised security
incidences.

2\. The XDC shall provide access to network information to ESDIS and
shall be IPv6 compliant according to NASA-established schedule.

3\. The XDS shall operate 24 hours a day, 7 days a week with an
operational availability of .96 during any given 30-day period. The XDC
itself may only be staffed on a 40-hour per week basis, but the XDS
shall meet this availability requirement around the clock with the
exception of scheduled downtime. (Operational availability is defined
excluding scheduled downtime).

4\. The XDS shall be able to clear, within 48 hours, any backlogs
accumulated during scheduled or unscheduled downtimes.

# 2.4 Information Management 

A primary role of information management is to give the users efficient
access to the XDC-held data products, providing them with all of the
information and tools to search, locate, select, and
order/request/access those products and services required to perform
their science investigations. These products may be stored in the
archives or may entail either higher level processing of an archived
product or the placement of an acquisition and processing request.
Information management may include XDS-specific, stand-alone client(s)
in addition to the Warehouse Inventory Search Tool (WIST). WIST is an
adaptation of the EOS Data Gateway (EDG) to work through the EOS
Clearinghouse (ECHO). The XDS shall support the export of inventory
metadata to ECHO using appropriate software tools to facilitate
cross-XDS access to EOS data holdings. The ESDIS Project will support
ECHO as operational middleware with at least one client (WIST).

1.  The XDS shall maintain (export, update, delete) its inventory
    metadata in ECHO at least on a daily basis. The XDC shall work along
    with the ESDIS Project to ensure the currency, correctness and
    completeness of XDC's metadata in ECHO to enable users' search and
    ordering of the XDC-held data products using WIST.

2.  The XDS shall provide and update Directory Interchange Formats
    (DIFs) appropriate to the XDC's holdings to the Global Change Master
    Directory (GCMD).

3.  The XDS shall manage the accessibility to granule- and/or
    collection-level data and services in ECHO using rules, conditions,
    permissions, and restrictions as available in ECHO.

4.  The XDS shall maintain an ECHO provider profile and provider policy
    information.

5.  The XDC shall support data product access via ECHO, which includes
    rejecting, accepting, canceling, closing and tracking orders or
    access requests.

6.  The XDS standard product metadata shall contain all information
    required of ECHO and GCMD product metadata. Optional
    Product-Specific metadata, documents and on-line universal resource
    locators (URLs) may also be provided by the XDS to ECHO and GCMD.

7.  The XDC shall work with ECHO support staff and coordinate any
    significant changes to the peak daily rate of metadata provided to
    ECHO.

8.  The XDC shall work with the ECHO support staff to resolve user
    community problems (e.g. orders/requests, availability of data
    holdings for search and order/request/access).

9.  The XDC shall ensure the currency, correctness and completeness of
    XDC's metadata in the XDC's inventory to enable users' search and
    ordering/requesting/accessing of the XDC-held data products using
    available search and order/request mechanisms.

# 2.5 User Services

1.  The XDC shall assist users in the resolution of problems in using
    XDC services.

2.  The XDC shall assist users with scientific information about the
    data products for which the XDC is responsible.

3.  The XDC shall assist users in resolving problems with formats and
    other data related issues.

4.  The XDC shall assist users in tracking download failures within two
    business hours after receiving request for assistance.

5.  The XDC shall assist ESDIS in conducting user satisfaction surveys.
    (These are annual surveys of users of each of the XDCs, and the
    results of the survey are aggregated by the ESDIS Project and
    reported to the OMB as a performance metric).

# 2.6 Information Security

1.  The XDC and XDS shall comply with all of NASA's information
    technology and information security requirements as in NASA
    Procedural Requirement (NPR) 2810.1 (Security of Information
    Technology, Revalidated 8/12/04) and NASA Policy Directive (NPD)
    2810.1C (NASA Information Security Policy, 4/7/04). The XDC shall
    provide copies of security documentation to appropriate NASA
    officials and the ESDIS office.

2.  The XDC and XDS shall maintain confidentiality of user product
    requests and accounts per the draft ESDIS Privacy policy guidelines.

3.  The XDS shall comply with all applicable guidelines for website and
    outreach communications as established by NASA.

This page intentionally left blank.

# Appendix A. Performance Metrics in Use by ESDIS Project  {#appendix-a.-performance-metrics-in-use-by-esdis-project .Heading-No-Number}

1.  Ingest rates and statistics on all products by XDC, Level,
    Discipline, Mission, Instrument, by time period (including volumes,
    numbers of granules).

2.  Archive rates and statistics on all products by XDC, Level,
    Discipline, Mission, Instrument, by time period (including volumes,
    numbers of granules).

3.  Data distribution metrics by XDC shall include volumes, number of
    data requests, number of granules, number of products, types of
    products, types of users) for given time periods. Various groupings
    of these metrics are used, for example by: user e-mail address
    (.com, nasa.gov, noaa.gov, umd.edu, etc. as well as individual
    e-mail addresses used for user satisfaction survey), instrument,
    mission or campaign, discipline, distribution type

4.  Number of user accesses to XDCs.

5.  Deleted products (by XDC, granule, product, time period).

This page intentionally left blank.

# Abbreviations and Acronyms {#abbreviations-and-acronyms .Heading-No-Number}

ACSI American Consumer Satisfaction Index

DAAC Distributed Active Archive Center

DCN Document Change Notice

DIFs Directory Interchange Formats

DPS Data Production System

ECHO EOS Clearinghouse

EDG EOS Data Gateway

EDOS EOS Data and Operations System

EMS EOSDIS Metrics System

EOS Earth Observing System

EOSDIS EOS Data and Information System

ESDIS Earth Science Data Information System

FDS Flight Dynamics System

FGDC Federal Geographic Data Committee (on standards)

FY Fiscal Year

GCMD Global Change Master Directory

GES GSFC Earth Sciences

GSFC Goddard Space Flight Center

HQ Headquarters

ICD Interface Control Document

LTA Long-Term Archiving

MODAPS MODIS Adaptive Processing System

MODIS Moderate-resolution Imaging Spectroradiometer

NASA National Aeronautics and Space Administration

NPD NASA Policy Directive

NPR NASA Procedural Requirement

OMB Office of Management and Budget

SCF Science Computing Facility

SIPS Science Investigator-led Processing System

URL Universal Resource Locator

WIST Warehouse Inventory Search Tool

XDS X Data System

XDC X Data Center

XML Extensible Markup Language