# Presentation: 861907

## Architectural Vision

- PDS NASA-HQ Review

- PDS Systems Engineering
- September 8, 2004

*http://pds.jpl.nasa.gov/*

## Purpose and Topics

**Current Architecture**
**PDS System Architecture**
**OODT****1**** Middleware**
**Looking Forward**
**Architectural Vision**
**Grid Technology Investigation **
**Benchmarking with non-planetary archives**

- Purpose: Describe the PDS System Architecture

- Definition : Architecture - The fundamental organization of a system embodied in its components, their relationships to each other, and to the environment, and the principles guiding its design and evolution.  (ANSI/IEEE Std. 1471-2000)

- 1 Object Oriented Data Technology (OODT)

## PDS System Architecture

## Requirements

- The System shall have an information centric architecture where information is searchable and accessible via standard interfaces.
- The System shall have an open architecture, where all changes and enhancements can be made in a collaborative manner, and all Application Programming Interfaces (APIs) are fully documented and shared.
- The System shall have non-proprietary core components. This includes all components that enable communication and transfer of data from one location to another.
- The System shall be modular and capable of being built incrementally.
- The System shall be service based, where each service can be built independently of each other and are capable of evolving independently.
- The System shall be a distributed system. It shall provide the ability to locate data, resources, and services from multiple physical locations.

## System Design Approach

- Hardware: Network, Storage, and Servers

- OS Platform: Linux, Java (JVM)

- Database: Sybase, MySQL

- Application Server Container: Tomcat

- Middleware: OODT, PDS modules

- PDS Custom Software

-  Build System in layers – limiting most development costs to top layer

- Buy and Integrate

- Buy/Download
- and Integrate

- Buy/Download
- and Integrate

- Download and
- Integrate

- Download, Customize,
- and Integrate

- Develop and
- Integrate

## Software Design: The 4-Tier Architecture

- Client Tier

- Web Tier

- Service Tier

- Storage Tier

- Web browsers
- and other net-
- enabled tools,
- running on the
- user’s machine

- Web applications
- running in a J2EE
- servlet container

- Middleware that
- transmits and
- transforms
- information from
- the storage tier

- Information stored
- in file systems and
- data bases

## Storage Tier Components

- PDS Repositories
- Online disk storage containing PDS-compliant “mission archives“
- These are the online data and ancillary files that mission projects generate and PDS ingests, distributes, and archives
- Catalogs
- Data bases containing metadata used to locate data products and other system resources
- Data product catalogs are typically loaded from index tables delivered with the data
- The Data Set Catalog at PDS central node is used to locate data sets and associated resources

## Storage Tier Components – MRO Example

- PDS-MRO repositories and catalogs
- HiRISE, data node at U of A, Tuscon
- MARCI-WA, Imaging Node, Flagstaff
- CTX, Imaging Node,  Flagstaff
- CRISM, Geosciences Node, St. Louis
- SHARAD, Geosciences Node, St. Louis
- Gravity, Geosciences Node, St. Louis
- MCS, Atmospheres Node, Las Cruces
- Accelerometer, Atmospheres Node
- SPICE, NAIF Node, Pasadena
- ONC, Imaging Node, Pasadena
- Data mirroring and data set catalog at Central Node

- PDS Repositories and Catalogs are distributed across the USA

## Service Tier Components

- Product Servers
- Fetch products from PDS repositories
- Optionally transforms the product returned (e.g., packaging it with others in a .zip file or converting it from PDS .img format to .gif format)
- Profile Servers
- Fetch information matching search criteria from catalogs
- Information returned as an “XML profile”
- Query Server
- Manages queries across distributed services.
**All are part of OODT middleware package**

## Web Tier Components

- Data Set View
- Searches for data sets and associated resources (data set search)
- Data Set Browsers
- Controls the search, display, and retrieval of data products and ancillary files belonging to a specified data set
- Ingest-Workflow Manager
- Executes data ingestion pipeline
- Validation Service
- Checks mission archives for PDS-compliance
- Tracking Service
- Reports the status of resources submitted to the PDS

## Web Tier Components (Continued)

- PDS Explorer
- Enables direct viewing and downloading from data repositories
- Subscription Manager
- Enables users to subscribe to be notified when new data (or other resources) become available
- Authentication Server
- Determines user access to the System

- Subscription Manager Sign-Up

- Display from Data Set View

## Client Tier Components

- Web Browsers
- Netscape 7.x, Mozilla, Internet Explorer 6.x and above currently supported
- Bulk Download Client
- Java client for downloading large collections of products
- Server Manager (part of OODT)
- Java client for administering distributed OODT components

## Client Tier – Examples

- Web and Java APIs enable wide variety of custom clients from outside developers
- Science domain-specific clients
- EPO clients
- Repository/Catalog management tools (e.g., combining Server Manager and PDS Explorer type capabilities)
- NASAView
- Visual desktop tool that views PDS files, both labels and data files (“PDS object viewer”)
- Prototyped client/server version

- Server Manager (OODT)

## The 4-Tier Architecture - Summary

- Client Tier

- Web Tier

- Service Tier

- Storage Tier

- Repositories

- Product Catalogs

- Data Set Catalog

- Product
- Servers

- Profile
- Servers

- DS Cat
- Profile
- Server

- Product
- Servers

- Product
- Servers

- Profile
- Servers

- Profile
- Servers

- Ingest-Workflow MGR

- Validation Service

- Tracking Service

- Data Set View

- Data Set Browsers

- Query Server

- PDS Explorer

- Subscription MGR

- Volume Packer

- Authentication Server

- Web Browser

- Bulk Downloader

- Server Manager

## OODT Middleware

## OODT Middleware - Service Layer Details 

- Software Framework initially funded by NASA, for data sharing –
- Across heterogeneous, distributed data repositories (location independent)
- Across multiple science and engineering disciplines (discipline independent)
- With many interconnected nodes (scalable)
- An architectural approach first, an implementation second
- Adapt to different distributed computing deployments
- Promotes a REST-style architectural pattern for search and retrieval
- PDS uses OODT components with customized plug-ins
- A PDS-OODT package is installed at every PDS data distribution node
- 2003 Runner Up for NASA Software of the Year

## OODT Middleware - Service Layer Details (cont)

- OODT is Open Source
- Developed using open source software  (i.e. Java/J2EE and XML)
- Implemented reusable, extensible Java-based software components
- Provided messaging as a “plug-in” component that can be replaced independent of the other core components.  Messaging components include:
- CORBA, Java RMI, JXTA, Web Services, etc
- Provided client APIs in Java, C++ and HTTP
- Simple installation on a variety of platforms (Windows, Unix, Mac OS X, etc)
- Used international data architecture standards
- ISO/IEC 11179 – Specification and Standardization of Data Elements
- Dublin Core Metadata Initiative
- W3C’s Resource Description Framework (RDF) from Semantic Web Community

**Notes:**
OODT was developed using all open source software.  This was critical since many of our users could not afford expensive licensing.  The ability to grid together these distributed assets meant that we needed to have minimal cost entry.
OODT implemented three key components: a profile service component, a product service component and a catalog and archive service component.  The profile service component captures metadata information about resources. It describes information about available resources and where and how you get that information, the product server provides a flexible mechanism to request a product including the format you want that product returned in.  The planetary community, for example, can request format conversion on images.  The catalog and archive service provides the rule-based capture and process infrastructure for building the data processing portions of a data management system.
The messaging is provided as its own component so that OODT is not bound to any one implementation.
OODT provides common interfaces using a variety of interface mechanisms.
It also provides very simple installation. A user remarked that he could install OODT’s core software in 10 minutes using a wizard.
We were also very diligent about using standards for our data architecture such as ISO 11179, Dublin Core and W3C’s RDF.

## OODT Collaborations

- Center for Grid Technologies at ISI
- Kesselman, Chervenak
- USC Center for Software Engineering
- Medvidovic
- National Cancer Institute – Early Detection Research Network
- Srivastava, Downing
- Seawinds Project
- Fouser
- Deep Space Mission Systems
- McVittie, Shames
- Mission Control Technologies (MCT) Group at Ames
- Walton
- Consultative Committee for Space Data Systems (CCSDS)
- GSFC, NASA, ESA, CSA, CNES, ...
- Dartmouth University
- Chambers
- White Sands Missile Range
- Kessel

## Looking Forward- Architectural Vision

## PDS Drivers

- Implement a data system framework that supports the PDS science-driven vision by
- Scaling to meet future data volume and processing needs
- Enabling interoperability across data systems
- Reducing system development and operations costs
- Maximizing automation during the preparation and distribution of science data
- Supporting the community in developing tools that connect to the PDS and use the data products

## PDS Leverage and Alignment 

**PDS works with OODT Architecture Team**
**OODT deployed at Odyssey Data Nodes**
**PDS to distribute Odyssey data on-line**

**Leverage archive metadata to enable system interoperability**
**Extend middleware to link all PDS data sets**
**Common tools to manage deployed system**
**Common interfaces between PDS  wide data systems**

**Common mechanisms to enable information architecture**
**Services to automate ingestion and validation pipeline**
**Use of Standard Grid Technologies and Specifications (OGSA)**

***Deploy Scalable Grid Infrastructure***

***Initiate System Architecture***

**PDS-D D01**
**2001 Mars Odyssey Distribution**

**Computational and Data Grid Infrastructure**

**Middleware for linking all PDS data repositories**

**Middleware for linking Odyssey**
**distributed data repositories **

***Support Integrated Science Environments***

**Scalable data and computational infrastructure**
**IT systems development coordinated across NASA organizations**
**Integrated information architectures between science disciplines**
**Integrated modeling**

- High Volume Missons (LRO, JIMO)
- PDS integrated with NVO
- Modeling
- Integrated Science Support Across Pipline
- Data Mining
- Geometry Engine

**PDS-D D02**
**Distribution of most of archive**
**Integration of SBN Target Database**

**MRO Distribution**
**Cassini Distribution**
**PDS in the Pipeline**
**Product subsetting and transformation**
**Correlative Search**
**Global Product Search**

**2002**

**2004**

**2006**

**2010**

**Computational and Data Grid Integrated Across Science Pipeline**

- PDS Leverage and Alignment

## Architectural Focus

- Data Architecture
- Common XML schema for managing information about data resources
- Common XML schema for messaging between distributed services
- Reuse of domain models within architecture
- Technology Architecture
- Information management services that sit on top of traditional middleware
- Archive Service (Product Mgmt), Profile (Registry) Services, Product (Repository) Services, Query Services, Domain Services
- Core services that publish well known interfaces using standard protocols (HTTP, TCP/IP, SOAP, CORBA, ...)
- The architecture fits with current middleware technologies (i.e. Web Services, RMI/J2EE, CORBA, Globus, ...)
- Process Architecture
- Enable data discovery, data capture, data distribution/exchange, and processing across highly distributed environments
- Metadata-driven architecture

## Vision

- “Architectural Style” for building “Data Intensive Systems”
- Connect highly distributed applications and systems
- Deal with different information models
- Address resource discovery
- Deal with information management and discovery across highly diverse computing environments
- Build “lights out” operational systems that support process and management of information across a pipeline
- Fit information management into Grid Computing

- Developing Proposal – Software Architectures for Multi-disciplinary Data Intensive Systems; Develop an architecture style for these types of systems.
- PI: Prof Nenad Medvidovic, USC; Co-I:Crichton, Thornquist

## Data Grid Investigation

## Remaining Technology Drivers for PDS

- Internet bottlenecks will not allow the PDS to meet online distribution expectations of many of its customers. E.g. MRO HiRISE distribution of derived products at end of mission
- Existing computational capabilities do not scaled to meet on-the-fly data processing requirements such as product calibration and derived product generation. E.g. 2001 Mars Odyssey THEMIS
- Investigating Grid Technology
- Computational Grid focuses on applying the resources of many computers in a network to a single problem at the same time - usually to a scientific or technical problem that requires a great number of computer processing cycles or access to large amounts of data.
- Data Grid focuses on infrastructure in the form of core data transfer services and generic data management libraries

- "Access to distributed data is typically as important as access to distributed computational resources.“ - The Globus Data Grid Effort

## PDS and the Data Grid

- PDS has initiated a data grid framework
- Based on a middleware framework
- Provides standard interfaces for resource search and retrieval
- Provides a uniform information structure
- PDS data model provides a common data architecture
- OODT profile concept provides a common resource description abstraction
- “Provides access to distributed data”
- Proposed Technology Infusion
- Leverage Globus/ISI technologies for data movement
- Unify Globus and OODT into “download-and-go” technology package
- Provides bi-directional benefit to both NASA/JPL/PDS and other ISI/Grid efforts

## PDS/OODT/GRID Grid Computing Proposal

- NASA Proposal between JPL and USC-ISI (Kesselman, et al)
- Merge OODT and Globus to support PDS data product processing and distribution
- Computational Grid for Data Product Generation
- Data Grid for Data Distribution and Interoperability
- Envision a Planetary Science Infrastructure that links all science production, distribution and archive functions across the country

## Implemented Framework

- PDS-D Framework (2002 - 2004)
- Information integration across PDS nodes to support distribution
- Includes resource discovery, unified repository access, simple transformations, bulk transfer of multiple products, and unified catalog access
- PDS-D Framework (2006)
- On-demand computing capabilities
- On-demand processing
- Infrastructure for moving huge data sets and meeting needs of a large number of users

## Benchmarking withnon-planetary archives

## PDS Benchmark of Other Data Systems(info exchanges past and in progress, not complete)

***Videocon Discussions, Lessons Learned***

## Findings

- NVO and PDS/OODT
- Identified similar patterns and approaches in implementing Grid Systems
- PDS has the strength of a common information model for the planetary science community
- Discussions with Robert Hanisch, Roy Williams, Reagan Moore
- PDS Archive and Astrophysics Archives
- Astrophysics has essentially a single data format for the entire community.
- Astrophysics archives are typically designed from the outset to be of wide service to a user community outside the mission PIs. Historically planetary PIs view the data as primarily for the use of the science teams - although this view is changing.

## Backup Material

## PDS-D Framework

- Archive
- Client

- OBJECT ORIENTED DATA TECHNOLOGY Middleware

**Notes:**
This is the idea of having pluggable OODT components.  You will notice various OODT services, each of which can be distributed, that are provided as part of the framework.  The archiving service enables archiving and process management, the profile service enables applications to locate “resources”, the product service enables applications to “access resources”. The query service provides query management by routing queries and managing associated query threads across all the deployed OODT servers.

This picture also shows the ability to extend the services into domain. For example, the idea of adding in a navigation service that might do coordinate transformations.

## Search and Retrieval Components

- Query Server

*Node 1*
*Profile Server*

- XMLQuery

- XMLQuery

- XMLQuery

- XMLQuery

- XMLQuery

**Resource**
**Catalogs**

- QueryClient

- Web server
- Plugins

- Web Server

- Repository Node
- Product Server

- XMLQuery

**Node 1 Products**

**Node N**
**Products**

- Web I/F

- Desktop I/F

- XMLQuery

- XMLQuery

- Name Server

- Repository Node
- Product Server

**Node 2 Products**

*Node 1*
*Profile Server*

*Node 1*
*Profile Server*

*Node 1*
*Profile Server*

- Repository Node
- Product Server

***...***

- Name Server

- Name Server

## Catalog and Archive Components

**Archive Server**
**(Process, Catalog, Version and**
**Archive)**

**Metadata**
**Registry**

**PDS DD &**
**Product Schemas**

**Validate**

**Rule-base**

- Java
- Agent 1

**Product****Catalog (DE-based)**

- Userquery

**PDS Portal**

- QueryClient

- Web server

- search.jsp

**XMLQuery (Profiles + Data)**

- Java
- Agent 2

- Java
- Agent N

**PDS Missions**

**Ingest PDS Data Products**
**(Metadata + Data)**

**PDS Product and Profile Servers**

**Process**

**Execute**

**DE Database**

**PDS **
**Repositories**

## Distributed Processing Framework

**Spacecraft**
**& Ancillary**
**Files**

**Metadata**
**Validation**

**Science**
**Level**
**Processors**
**(LP)**

**Science **
**Analysis and **
**Quality**
**Reporting**
**(SA)**

**File**** Transfer (FX)**

**User Interface (Process Monitoring & Control, Data Verification) **

**Agent-based Process Management**

**Product Delivery (PM)**

**Science**
**Products**
**Released**

**Pre-Processors**
**(PP)**

**Distributed Client Systems**

**XMLQuery for Information Products**

## Proposed Technical Approach (GRIP Project)

- Prototype end-to-end, multi-mission science framework that focuses on
- Construction of ***registries*** (catalog) for publishing information about data products
- Construction of ***repositories*** for managing data products
- Deploying a ***data-driven framework*** for processing, registering, validating and storing products
- Provides a distributed grid infrastructure for locating, accessing/transferring data products
- Provides a distributed grid infrastructure for ***shared computational processing***
- Goal - A ***computational*** and ***data*** grid infrastructure capable of supporting interfaces for highly distributed, heterogeneous implementations