# Presentation: 531451

## How Information is Accessed

- PDS NASA-HQ Review

- PDS Systems Engineering
- September 8, 2004

*http://pds.jpl.nasa.gov/*

## Purpose and Topics

**PDS-D (Distribution)**
**Overview and requirements**
**Hardware**
**Software Architecture Components – Middleware and XML**
**Software Architecture Components - Custom Software**
**PDS-D D02, D03, D04, and D05 **
**Looking Forward**

- Purpose: Describe how information in the distributed archive is accessed through the on-line distribution system, PDS-D.

## PDS Pipeline by Project Phase

***OPERATIONS***

***PLANNING/DESIGN***

- Phases A-D				Phase E-F

**Trade Studies*****1***

**ICD / SIS’s per Instrument**

**Data Mgmt/Archive Plan**

**Validation*****2***

**Ingest Data (Receive/Store/Catalog)**

**Distribution/Access**

**Track**

**Archive/Transfer*****1***

**Instrument Team Data Production**

**Delivery to PDS**

**Science Users**

***1, 2****** - Areas of Risk to Schedule/Resources***

_**Core System:**_**  Within planned budget/staffing, with the exception of need for (1) Storage Technology ****	****  Study to determine MRO-era hardcopy archive media and (2) changes in Validation area**
_**Extended System:**_**  Requirements/planning to be done on deferred LOE basis**

**Development*****2***

**Pre-Brief / MOU**

## Requirements - Summary

- The system shall provide the ability to locate and retrieve any resource on the system based on its cataloged attributes.
- The system shall be able to retrieve any resource description in the system.
- The system shall provide a mechanism for running routines on (“process”) any collection of resources or resource descriptions, performing the appropriate set of transformations or manipulations, and returning the results.
- The system shall be able to store 120TB of data products online and ensure that all data are accessible to all system functions and services.

- Definition – Resource - A resource is anything that can be described by a set of attributes. Resources in the PDS include data products, data sets, software, documents, web services, spacecraft, instruments, and personnel.

## Hardware

## Hardware Design

- Driven by need to ingest, store, and backup over 120 TB data for MRO Era
- Hardware architecture characteristics
- Storage Area Network (SAN)
- High speed fiber channel and Gbit Ethernet connects storage to local devices (including switches, backup systems, data bricks)
- Internet connects storage to users
- Scalable architecture: easy to add more storage and servers
- Importance of fast local interconnect
- Enables ingest, backup, and transfer of large data volume
- Current 1 Gb/sec rate means transferring  up to 1 TB in 2.2 hours
- Multi-port switches allow more than one simultaneous target (e.g., filling up two data bricks at once)

## Hardware Design – SAN Architecture

## Hardware Design (Continued)

- Also driven by need to distribute as much data as possible as efficiently as possible over the Internet
- Hardware architecture characteristics
- Clustering servers – Multiple servers behind single address, sharing load
- Load balancing – Maximizing throughput by shifting load
- Failover – Server B takes over when Server A goes down
- Mirror MRO data at Central Node – Better to have more than one distribution site (also helps ensure data preservation)
- Looked into mirroring and load balancing between Central Node and Geosciences Node (distributes CRISM)
- Great idea... but cost too much

## Software Architecture Components -  Middleware and XML

## Conceptual Framework

- Standard Interfaces (OODT Middleware)

**PDS PPI**

- Data Products
- and Metadata

**Documents and**
**Ancillary Files**
**PDS CN**

**PDS GEO**

**PDS Imaging**

**PDS GEO**

**PDS NAIF**

**PDS ATMOS**

- Science

- Educational

- General
- Public

- _Distributed _
- _Data Repositories_
- _and Catalogs_

- _Distributed Clients_

- _Users_

- Data Set View

*IDL*

## XML - Describing Resources using Profiles

- A profile is an XML document that describes a resource; such as a data system, data products, etc, by specifying
- Common data attributes using Dublin Core (I.e. Title, Author, Subject)
- Mechanisms for describing where the product is located and how to access it
- Domain data elements that are useful for describing the product (i.e. TARGET_NAME, MISSION_NAME, INSTRUMENT_NAME, etc)
- Enables “search and retrieval” of distributed resources including data products
- Searches to a *Profile Server* yields information regarding the characteristics of distributed resources (i.e. descriptive information about the product, access information, etc)
- XML was selected because
- Globally accepted language for information interchange
- Excellent tool support
- A single schema as been defined that is capable of describing any resource

## Data Product Profile Example

- -<profile>
-    -<profAttributes>
-      <profId>1.3.6.1.4.1.1306.2.104.10018791</profId>
-      <profVersion>null</profVersion>
-      <profType>profile</profType>
-      </profAttributes>
-    -<resAttributes>
-      <Identifier>ODY-M-THM-2-IREDR-V1.0+I00903005EDR</Identifier>
-      <Title>ODY-M-THM-2-IREDR-V1.0+I00903005EDR</Title>
-      <Description>null</Description>
-      <resContext>NASA.PDS</resContext>
-      <resAggregation>null</resAggregation>
-      <resClass>data.product</resClass>
-      <resLocation>http://...PDS.ASU.Product...data/odtie0_xxxx/i009xxedr/I00903005EDR.QUB...
-      </resAttributes>
-    -<profElement>
-      <elemName>MISSION_NAME</elemName>
-      <elemValue>2001 MARS ODYSSEY</elemValue>
-      </profElement>
-    -<profElement>
-      <elemName>INSTRUMENT_ID</elemName>
-      <elemValue> THEMIS</elemValue>
-      </profElement>
- ...

## Middleware and Component Framework

*Discovery*
*Query*

**Applications**

**Query Server**

**Webserver (QueryServlet)**

- Java API

- HTTP

*Retrieval*
*Query*

*Packaged *
*Products*

*Product*
*Descriptions*

*User Queries*
*And Results*

- Middleware

## Profile Server

- Middleware component that negotiates the interfaces between the data system implementations despite the heterogeneity
- Extends framework to support common access to distributed catalog databases
- A Profile Server searches and retrieves profiles from a profile database
- For search, a profile server allows any profile attribute as a query constraint including domain dependent or common data elements
- For retrieval, a profile server returns all matching profiles
- Profile servers can point to other profile servers for scalability

**Profile Server**

- Query: XML

- Result: XML

- Generic Server

- Implementation Class

- File
- Sys

- Database

## Product Server

- Middleware component that negotiates the interfaces between the data system implementations despite the heterogeneity
- Extends framework to support common access to distributed data repositories
- A PDS Product Server retrieves products from a data repository
- Understands PDS Data Products
- Query constraint is a unique product identifier
- Returns requested result
- Product label, Raw File, Transformed product
- ZIP package containing all associated files

## Query Server – Manages the Search Algorithm

- _:Query__Server_

- _:Profile__Server_

- _:Product__Server_

- target = Jupiter

- target = Jupiter

- urn:eda:rmi:PSn

- urn:eda:rmi:PSn

- target = Jupiter

- target = Jupiter,urn:eda:rmi:PSn

- Search - First, do a *profile query* to resolve resources that *could* provide the data
- Tells you *where to look (*In the form of a URI)
- Retrieve - Then, do a *product query* based on the profile query results to get the data
- Actually retrieves data (In a MIME-compliant format)

## Search Component Framework

*Discovery*
*Query*

**Product Catalog**

**Profile Server**

**Query Server**

*Product*
*Descriptions (profiles)*

***User Queries***

- Middleware

***Profile Results***

**searches catalog**
**returns product description**
**       ****location**
**       ****identification**
**       ****description**
**       ****relations**

**TARGET_NAME = MARS**
**  ****& LATITUDE > 80.3**
**  ****& INSTRUMENT_ID = HIRISE**

**<Identifier>  **
**<Format>        **
**<Location>    **
**<elem>DATA_SET_ID**
**<elem>IMAGE_ID **
**<elem>OBSERVATION_ID **
**<elem>PRODUCT_TYPE**

**A profile server is configured for each data set**
**A catalog is created for each data set**

## Scalability of Searching Distributed Resources

- Profile
- Server
- A

- Profile
- Server
- C

- Profile
- Server
- F

- Profile
- Server
- B

- Profile
- Server
- E

- Resource
- Descriptions

- Distributed
- Query

- Resource
- Descriptions

- Resource
- Descriptions

- Resource
- Descriptions

- Resource
- Descriptions

- Resource
- Descriptions

- Resource
- Descriptions

- Distributed
- Data
- Products

- Distributed
- Data
- Products

- Distributed
- Data
- Products

- Distributed
- Data
- Products

## Retrieval Component Framework

**Local Product Server**

**Data Repository**

**Query Server**

*Retrieval*
*Query*

*Packaged *
*Products*

- Middleware

**Retrieval**
**Query**

**Product/**
**Package**

**Returns product packaged with all associated files, raw file, product label, etc**

**IDENTIFIER = product_unique_id **
**& RETURN_TYPE = requested_format**

**ZIP file**
**RAW file**

**A “compliant” repository contains the data set**
**A product server is ready to serve products “out-of-the-box”**

## Server Interfaces

**API protocols for services**
**HTTP Queries**
- Generic query (keyword/value constraints) resulting in
- Data Product or Catalog Entities Profiles
- Raw XML results
- JSP formatted results
- Data Products
- Label
- Raw File
- ZIP Package containing file comprising data product
- PDS Query (by identifier) resulting in
- JSP formatted results
**JAVA interface**
**C, IDL interfaces**
**Web Services (SOAP Interface)**
**Additional interfaces to repositories**
**FTP**
**Web Server HTTP**

## Process Component Framework (General)

**Local Product Server**

**Data Repository**

**Query Server**

*Retrieval*
*Query*

*Packaged *
*Products*

- Middleware

**Object Access Library**
**OAL**

*Retrieval*
*Query*

**Label Parsing**
**Label Access**
**Data Conversion**
**Product Access**
**Routines for generic**
**    ****transformations**

** ****Existing generic processing **

- *

**Product**

**PDS Standard Product**

## Software Architecture Components -  Custom Software

## PDS Custom Software

**Data Set View**
**Searches for data sets and associated resources (data set search)**
**Data Set Browsers**
**Control the search, display, and retrieval of data products and ancillary files belonging to a specified data set**
**PDS Explorer**
**Enables direct viewing and downloading from data repositories **
**Subscription Manager**
**Enables users to subscribe to be notified when new data (or other resources) become available**

## Bulk Download Client
Java client for downloading large collections of products
Web and Java APIs enable wide variety of custom clients from outside developers
Science domain-specific clients
EPO clients
Repository/Catalog management tools (e.g., combining Server Manager and PDS Explorer type capabilities)
NASAView
Visual desktop tool that views most PDS files, both labels and data files (“PDS object viewer”)
Prototyped client/server version
Web Browsers
Netscape 7.x, Mozilla, Internet Explorer 6.x and above currently supported

**Java client for downloading large collections of products**
**Web and Java APIs enable wide variety of custom clients from outside developers**
**Science domain-specific clients**
**EPO clients**
**Repository/Catalog management tools (e.g., combining Server Manager and PDS Explorer type capabilities)**
**NASAView**
**Visual desktop tool that views most PDS files, both labels and data files (“PDS object viewer”)**
**Prototyped client/server version**
**Web Browsers**
**Netscape 7.x, Mozilla, Internet Explorer 6.x and above currently supported**

- PDS Custom Software

## PDS-DD02, D03, D04, and D05

## Implementation Plan: 2004 - 2006

**Start with current System as defined by the 2004 delivery of D02**
**Increment the System through yearly software deliveries at end of FY 2004, FY 2005, and FY 2006**
**Complete Core System software components**
**Operational in October 2006**
**Ready for May 2007 first delivery of MRO data **
**Hardware will continue to be added to meet capacity requirements**

- System matures with each Development Cycle

## PDS-D D02 Capabilities – FY 2004

**Current distribution functionality**
**Currently supports all active missions and 90% of pre-PDS-D02 data**
**Supports data set searches for any data set in the PDS Catalog **
**Supports product searches for data sets that have Data Set Browsers**
**Supports retrieval of any PDS product in any PDS repository (Central Node has most of existing archive loaded)**
**Bulk downloader for large downloads**
**Supports browsing repositories with PDS Explorer **
**Subscription and notification for Mission data and PDS docs/software**
**Prototyped “profile-of-profile servers” to locate any product in the System**

## PDS-D D03

- The PDS-D D03 release is a relatively small, quick follow-on to D02 in order to sync up the system development life-cycle with a fiscal-year end delivery.
- D03 represents a bridge to a mature development cycle.
- D03 deliverables were agreed-to by the PDS Management Council at the June ’04 meeting.

## PDS-D D03 Deliverables

- Provide online Issue Tracking software to track all issues and anomalies
- On track
- Enable release of MER data through online distribution system Involves all discipline nodes that curate or provide resources for MER data sets
- Complete
- Integrate “record level searches” of SBN target database into online interface
- Preliminary Design Review with node successful
- Implementation on track
- Creating profile server for the SBN target database
- Integrated target information from PDS catalog and SBN target database
- Enable transfer and receipt of large data volume through “data bricks”
- On Track
- Use capability to transfer Odyssey THEMIS data from ASU to CN repository

## PDS-D D03 Deliverables

- Resolve any remaining issues with product server installations
- Prototype IHW distribution at SBN for Stardust distribution
- Upgrade product server at PPI
- Distribute Java-based Object Description Langugate (ODL) parser with PDS Tools for acceptance testing
- ODL parser tested against data products in CN Data Repository
- ODL parser being tested in PDS-IV validation prototype
- Complete analysis and design of online validation service (PDS-IV)
- Requirements review scheduled
- Complete assessment of PDS-to-NSSDC interface
- On track
- Place online system and tools development under configuration management
- On track

## PDS-D D04 and D05 Capabilities

**FY 2005 - Continue to round out distribution capabilities**
**Complete global search capability**
**Deploy special processing (e.g. PDSImageReader)**
**Deploy an authentication server for secure access to pre-release data**
**FY 2006 – Complete core requirements for distribution**
**Ensure support for MRO data sets in Data Set Catalog and Basic Browsers**
**Add capability to subset JPEG2000 images**

## Process Component Framework (Special)

**Local Product Server**

**Data Repository**

**Query Server**

*Retrieval*
*Query*

*Packaged *
*Products*

- Middleware

**PDSImageReader**

*Retrieval*
*Query*

**Transformations among PDS, VICR, FITS, JPEG, JPEG2000, GIF, binary, ASCII, etc.**

**“****Plug-in" capability for domain specific processing**

**Product**

**Requested**
**Image Format**

## Looking Forward

## Additional Search Types

- Correlative Search - Determine if attributes to be used in a search are correlated across all contexts
- E.g. TARGET_NAME=MARS & LATITUDE>80.3
- Do all Mars data sets use the same reference frame? NO!
- Derived Parameter Search – Compute attribute values to be used in an index search
- E.g. DATA_SET_ID=MGS_MARS_IMAGES & LATITUDE>80.3
- Convert LATITUDE between Planetographic and Planetocentric reference frames for product search and registration between Viking and MRO images.
- Data Mining – Search based on extracting information from content
- E.g. Obtaining Mars altimetry data from the MGS MOLA instrument requires point data extraction from large raw data products.
- Retro-fitting metadata to data products
- E.g. Add Feature Identifiers not identified until after archive.

## Services “plug-in” Framework

**Product Server**

**Data**
**Repository**

**Query Server**

*Retrieval*
*Query*

*Queries*

- Middleware

**Data Mining**
**Server **

*Discovery*
*Query*

**Product Catalog**

**Profile Server**

**Geometry**
**Engine**
**Server**

**Application**

**Application "plug-in" capability for domain specific/external processing**

## Data Distribution

- Additional *server-side* processing
- On-the-fly calibration
- Product sub-setting (Images and Tables)
- X-Windows paradigm (Screen updates – NASAView DSI)
- Proposal for Planetary Science Grid Services Infrastructure

## Backup Material

## Data Transfer Rates

**1MB**

**1GB**

**1TB**

**1PB**

**Online**
**Transfer**

**Data Brick**

**48 Hrs**

**6 Hrs**

**6 Mins**

**36 Secs**

**Bulk Download**