# Presentation: 854515

## ECS SDP Internal Training

**External**
**Data Provider**

**User**

- Revised July 1999 (Reflects Release 5A)

## Objectives

**Overall objective:  Describe ECS structure and function for Science Data Processing (SDP)**
**Identify subsystems and Computer Software Configuration Items (CSCIs)**
**Specify major components and functions/processes of CSCIs**
**Describe role of CSCIs/functions/processes in the context of ECS operational scenarios**
**ASTER-specific functions (e.g., DAR, expedited data support)**
**Producing and distributing data products (including media)**
**Updating QA metadata**
**On-demand processing workaround**
**User registration**
**Landsat data insertion and access**

## What This Lesson Is (and Is Not)

**Is**
**Brief illustration of ECS high-level structure**
**Introduction to subsystems that make up ECS at a site**
**Examination of each subsystem and its Computer Software Configuration Items (CSCIs), with components**
**Introduction of all system elements and brief description of functions**
**Background for subsequent scenario-based presentation of system functional flows**
**Detailed look at system functioning in the context of operational scenarios**
**Is Not**
**Full description of overall ECS structure and function**
**Description of specific individual ECS entities (e.g., SMC)**
**Software development lesson**
**Complete description of interfaces and event sequences**
**Operations training**

## Program Overview

## EOSDIS Principal Components

** **

## EOSDIS Data Flow

**Data from other Mission Data Systems**
**(Landsat-7, TRMM, UARS, . . .)**

**Communications and**
**Internetworking Infrastructure**

**ECS Mission**
**Operations**
**System**
**(EMOS)**

**EDOS**

**Flight**
**Dynamics**
**System**

**Totally supplied**
**by ECS**

**Contains ECS**
**components**

## Relationship of ECS to Global Change Research

**WORLDWIDE**
**GLOBAL CHANGE RESEARCH**

## ECS Context

## Subsystems and Functions

***Science Data Processing Segment (SDPS)***
**Data Server Subsystem (DSS)**
**Data storage and management:  archive science data (with related insert, search and retrieve functions), archive management, data resource staging**
**Ingest Subsystem (INS)**
**Interface with external data providers and transfer data into ECS (with related staging functions and operator interfaces)**
**Data Processing Subsystem (DPS)**
**Dispatches and monitors execution of science software**
**Planning Subsystem (PLS)**
**Long- and short-term planning of science data processing, and management of production resources**

## Subsystems and Functions (Cont.)

***SDPS (Cont.)***
**Client Subsystem (CLS)**
**Provides interfaces and access for external users**
**Data Management Subsystem (DMS)**
**Enables cross-site data search and retrieval; gateways for interface of ECS with EOSDIS Data Gateway Web Client (Version 0 IMS) protocol**
**Interoperability Subsystem (IOS)**
**Advertising Service; support for other subsystems in locating data or DSS services**

## Subsystems and Functions (Cont.)

***Communications and System Management Segment (CSMS)***
**System Management Support Subsystem (MSS)**
**System maintenance, management, and administration (includes trouble ticketing, baseline and configuration management, fault and performance monitoring, and user account management and order tracking)**
**Communications Subsystem (CSS)**
**General system infrastructure functions (includes DCE and network communications, libraries to standardize software mechanisms, application error handling, interfaces to          e-mail, file transfer and network file copy functions)**
**Internetworking Subsystem (ISS)**
**Networking hardware devices and embedded software**
***NOTE:  The ISS is part of the ECS infrastructure and is not addressed in detail in this course. ***

## Subsystems and CSCIs:  DSS

## Subsystems and CSCIs:  DSS (Cont.)

**Data Server Subsystem (DSS)**
**Provides capabilities to store, search,                          retrieve, and distribute earth science                              and related data**
**Client-server information transfer is by commands and requests**
**Uses Universal References to identify ECS entities**
**GranuleUR:  represents a granule in the data server (e.g., as follows)**

- UR:10:DsShESDTUR:UR:15:DsShSciServerUR:13:[GSF:DSSDSRV]:16:SC:MOD10_L2:1411

- [GSF:DSSDSRV]

**DsServerUR:  represents a specific running data server application (e.g.,   ****                           ****)**

**Uses MSS Event services to log system-level events**
**Interfaces with virtually all ECS subsystems and components**
**Uses several COTS tools:  RogueWave tools and libraries, Sybase relational database, Spatial Query Server, HDF, ODL**

## Subsystems and CSCIs:  DSS (Cont.)

**Science Data Server (SDSRV) CSCI**
**Provides the ECS with a catalog of Earth Science Data holdings, and the Earth Science Data Type (ESDT) services that operate on the data**
**Manages and provides user access to data collections through its catalog of metadata and mechanisms to acquire data from the archive**
**Four major components**
**Science Data Server**** - services requests for storage, search, retrieval, and manipulation of science data**
**HDF EOS Server**** - provides science data subsetting**
**Science Data Server GUI**** - provides operator interface**
**Sybase/SQS Server**** - manages catalog (metadata)**

## Subsystems and CSCIs:  DSS (Cont.)SDSRV Architecture and Interfaces

**Subsystems and CSCIs:  DSS (Cont.)****SDSRV Architecture and Interfaces**

## Subsystems and CSCIs:  DSS (Cont.)

**Storage Management (STMGT) CSCI**
**Stores, manages, and retrieves data files on behalf of other science data processing components**
**Five major components**
**Archive Server**** - provides GUI and access to stored data**
**Staging Monitor**** - manages the group of data files that have been retrieved from the archive and placed into a cache area on staging disk**
**Resource Manager**** - schedules access to shared peripheral devices**
**Pull Monitor**** - manages files in the user pull area, deleting them as they are retrieved by users or as their time-out periods expire**
**Data Base**** - contains data tables for STMGT devices, cache management, event and log management, requests, and related functions**

## Subsystems and CSCIs:  DSS (Cont.)STMGT Architecture and Interfaces

**Subsystems and CSCIs:  DSS (Cont.)****STMGT Architecture and Interfaces**

## Subsystems and CSCIs:  DSS (Cont.)

**Data Distribution (DDIST) CSCI**
**Formats and distributes data to users, either electronically or on physical media (e.g., 8mm tape cartridges)**
**Directs STMGT to place data in working storage, to copy data to tape, or to push data as required via FTP**
**Sends e-mail notifications**
**Three major components**
**Data Distribution Server**** - provides control and coordination for data distribution through request processing**
**Data Distribution GUI**** - allows operations staff to initiate, track, and manipulate distribution requests**
**Data Base**** - contains the request list; updates and provides the request configuration**

## Subsystems and CSCIs:  DSS (Cont.)DDIST Architecture and Interfaces

**Subsystems and CSCIs:  DSS (Cont.)****DDIST Architecture and Interfaces**

## Subsystems and CSCIs:  DSS (Cont.)

**Subsystems and CSCIs:  DSS (Cont.)**

## Subsystems and CSCIs:  INS

**Ingest Subsystem (INS)**
**Transfer of data into ECS (SDPS repositories) in accordance with approved ICDs**
**Supports varied data formats and structures**
***Ingest Client*****:  A set of ingest software configured for requirements of a specific situation**
**Ingest clients perform data preprocessing, metadata extraction, and metadata validation on incoming data**
**Data staged to one of two areas**
**Level 0 (L0) data from ongoing missions, and EDOS ancillary data, staged to INS working storage area**
**Non-L0 data (e.g., non-EDOS ancillary data, L1A-L4 data) staged directly to DSS working storage area **
**Uses several COTS tools:  RogueWave class libraries, Sybase relational database, Tivoli Client, DCE Client, DCE Driver, MSAccess, HP OpenView Client**

## Subsystems and CSCIs:  INS (Cont.)

**Ingest (INGST) CSCI**
**Acquires data by various methods and transfers the data into ECS**
**Automated transfer: in response to notification from the data provider, Ingest transfers the data from a specified network location**
**Polling:  transfer of data from predetermined network locations which Ingest periodically checks for new data**
**With Delivery Record**
**Without Delivery Record**
**Media:  reading data from tapes**
**Stores and manages request information**
**Provides for data preprocessing and insertion**

## Subsystems and CSCIs:  INS (Cont.)

**Ingest (INGST) CSCI (Cont.)**
**Seven major components**
**Automated Network Ingest Interface**** - provides basic capability to ingest data electronically from an external source**
**Polling Ingest Client Interface**** - creates polling request, detects new files in a specified external location, creates and submits ingest request**
**Media Ingest Interface**** - provides operators ability to perform ingest from physical media**
**Cross-Mode Ingest Interface ****- provides an E-mail gateway server to receive distribution notifications and store them as files in a location that can be polled to create delivery records**
**Ingest Request Manager**** - manages ingest request traffic and processing**
**Ingest Granule Server**** - provides services for required preprocessing of data and subsequent insertion into Data Server**
**Ingest Database**** - stores and provides access to Ingest Subsystem internal data (e.g., History Logs)**

## Subsystems and CSCIs:  INS (Cont.)Architecture and Interfaces

**Subsystems and CSCIs:  INS (Cont.)****Architecture and Interfaces**

## Subsystems and CSCIs:  INS (Cont.)

**Subsystems and CSCIs:  INS (Cont.)**

## Subsystems and CSCIs:  CLS

**Client Subsystem (CLS)**
**User access to ECS services and data, and other **   ** systems interoperable with ECS (e.g., Version 0)**
**Search and retrieval of data are performed by the EOS Data Gateway (Version 0 Web Client)**
**Includes applications programs accessible through user interfaces**
**User Registration Tool (URT)**
**EOSView**
**Java Data Acquisition Request (DAR) Tool**
**Uses several COTS tools:  Netscape Navigator, Netscape Enterprise Server, XVT (widget set and development tool for EOSView), Interactive Data Language (IDL) (used in EOSView visualization features), and Rogue Wave Tools.h++ (libraries provide strings and collections to the Java DAR Tool and User Registration Tool)**

## Subsystems and CSCIs:  CLS (Cont.)

**Workbench (WKBCH) CSCI**
**Includes applications and libraries for access to ECS data and services**
**Release 5 includes 3 tools**
**URT**** (HTML-based)**
**EOSView**** (X/Motif-based)**
**Java DAR Tool**** (Java/HTML-based)**
**Since the CLS contains no hardware CIs, the DMS hardware (Interface Servers) provides the processing and storage for the WKBCH software**
**In addition, the ****User Profile Gateway**** Server is used to provide user profile information to the EOS Data Gateway for ECS users**

## Subsystems and CSCIs:  CLS (Cont.) Search & Order Tool Interfaces

**Subsystems and CSCIs:  CLS (Cont.)**** Search & Order Tool Interfaces**

## Subsystems and CSCIs:  CLS (Cont.)WKBCH Architecture and Interfaces

## Subsystems and CSCIs:  CLS (Cont.)

**Subsystems and CSCIs:  CLS (Cont.)**