# Presentation: 208682

## A Data Architecture for Interoperable Space Sciences Data Systems1st Annual ERDN WorkshopEarly Detection Research NetworkChicago, IL  September 27, 2000

- Principal Investigator Object Oriented Data Technology Task
- Steve Hughes, Lead System Engineer, Planetary Data System,
- Co Investigator, Object Oriented Data Technology Task
*Jet Propulsion Laboratory, California Institute of Technology*
*National Aeronautics and Space Administration*

***A Data Architecture for Interoperable Space Sciences Data Systems******1st Annual ERDN Workshop******Early Detection Research Network******Chicago, IL****** **** September 27, 2000*

## About JPL and Enterprise Computing

- JPL is a federally funded research and development center (FFRDC) run by Caltech for NASA
- JPL is NASA’s lead center for robotic exploration of the universe
- JPL has an enormous amount of data that it needs to manage from scientific data, to engineering, to institutional
- We represent several efforts in both the research and enterprise side of JPL that is addressing enterprise architectures for integrating data at both JPL and NASA.  Such efforts include
- Knowledge Management - JPL/NASA KM Initiative
- Enterprise Data Architecture Task - JPL IT Data Infrastructure
- Planetary Data System - NASA Planetary Archive
- Object Oriented Data Technology - I.T. Data System Research

## Motivation for Managing Data

- Overview
- Problem: Science data is highly distributed across geographically heterogeneous data systems.  It is difficult to access, and the systems do not interoperate well.  There is no common interchange mechanism, nor is their a common architecture. Correlation of data across these systems is problematic.
*Solution: Provide a software framework that provides cross disciplinary solutions for data management and archiving including interoperability among astrophysics, planetary, and space science data systems.*
- Focus on 3 capabilities: science data archiving, science search and retrieval, and science application support.
- Reduce the cost, risk, and development time for building ground data systems by leveraging off an existing framework
- Apply new technologies to improve science data management systems
- Distributed and object technologies
- Database and data system technologies
- Metadata and interoperability solutions
- Solutions for enterprise systems and enterprise application integration

## What is an Enterprise Data Architecture?

- An enterprise data architecture provides the infrastructure necessary to enable development of interoperable, enterprise-wide applications
- Focus on providing the key data management infrastructure
- Data Archiving
- Managing Local Data
- Search and Retrieval
- Data Location
- Managing Profiles
- Data Access
- Data sharing between data systems
- Data Interoperability

## Data Archiving

- Data Archive
- A data archive is the basic service for managing data products produced by instruments, scientists, and other systems
- Near Term Requirements
- Support convenient and efficient data retrieval
- Reduce data redundancy
- Support multiple users
- Provide data security
- Improve consistency

## Data Archiving

- Data Archive – Long Term Requirements
- After data is no longer managed by the research team and data system
- Ensure data remains viable
- Ensure data remains useable
- Ensure data remains understandable

## Data Archiving

- Data Archiving – Science archive example
- Products are collected into a data set together with ancillary data
- Data sets are managed within the archive
- Archives may be distributed and organized by domain expertise
- Metadata is used to address most archive requirements

## Archive Management

## Data Search and Retrieval

- Space scientists can not easily locate or use data across the hundreds if not thousands of autonomous, heterogeneous, and distributed data systems currently in the Space Science community.
- Heterogeneous Systems
- Data Management - RDBMS, ODBMS, HomeGrownDBMS, BinaryFiles
- Platforms - UNIX, LINUX, WIN3.x/9x/NT, Mac, VMS, ...
- Interfaces - Web, Windows, Command Line
- Data Formats - HDF, CDF, NetCDF, PDS, FITS, VICR, ASCII, ...
- Data Volume - KiloBytes to TeraBytes
- Heterogeneous Disciplines
- Moving targets and stationary targets
- Multiple coordinate systems
- Multiple data object types (images, cubes,  time series,  spectrum, tables,
- binary, document)
- Multiple interpretations of single object types
- Multiple software solutions to same problem.
- Incompatible and/or missing metadata

## Solutions to Data Search

- Build metadata “profiles*”* that describe data system resources
- Encapsulate individual data systems resources. (Hide uniqueness.)
- Communicate using metadata. (Provide metadata with data)
- Enable interoperability based on metadata compatibility.
- Refocus problem on metadata development.
- Provide a core framework of software components to interconnect distributed data systems

## Focus on Metadata

- Metadata is data about data
- Provides descriptive information about the data
- Classification, identification, etc
- Metadata Example
- Data Value: 55 (not descriptive)
- Metadata Values:
- Data Element Name:Vehicle_Speed
- Unit: Miles per Hour
- Description: The average velocity of a vehicle.
- Build metadata repositories that manage information about distributed data products (E.g. location, target, observation date, etc)
- Use standards where appropriate
- ISO/IEC 11179 – A framework for the Specification and Standardization of Data Elements
- Dublin Core – A metadata element set intended to facilitate discovery of electronic resources.

## OODT Metadata Development

- Metadata Registry – Develop a data management system for managing the semantics of data that is shared within and between domains.
- Terminology Base – Domain specific name space.
- Data Dictionary – Inventory of domain terms with definitions and other distinguishing attributes.
- Ontology – A set of concepts, their relationships and constraints, all within the scope of a domain.
- XML for metadata registry and communication
- Several I.T. efforts have shown the criticality of metadata in enabling data sharing and system interoperability.

## Difficulties in Data Access to Distributed Databases

- Access to distributed data systems and databases is difficult
- Vendor database products
- Data model implementations
- Representations of data
- Platforms
- O/S
- etc
- ...are all different

## Accessing Distributed Databases

- Provide a common interface to access multiple data systems
- Provide a standard language for communication
- Use XML Query language for data interchange
- Use rich metadata to describe queries and results
- Plug into an overall data architecture solution
- Consistent metadata
- Consistent data interchange
- Build “product servers” which negotiate the interface between the infrastructure and the data system implementation
- Provide a middleware framework to tie the data architecture together

## Focus on Middleware Framework

- Middleware defined as
- 	In the computer industry, middleware is a general term for any programming that serves to “glue together” or mediate between two separate and usually already existing programs. A common application of middleware is to allow programs written for access to a particular database to access other databases.
- 	Messaging is a common service provided by middleware programs so that different applications can communicate. The systematic tying together of disparate applications is known as enterprise application integration.
- 							*http://www.whatis.com*
- Middleware allows for the encapsulation of individual data systems
- Hide uniqueness by introducing the data architecture layer
- Ties distributed applications together an often works with a Electronic Data Interchange (EDI) type mechanism
- Enables reuse and promotes standards

## Role of Middleware

- Middleware can tie application, data, and user interfaces together and hide the unique interfaces

- Applications

- User Interface

- Data

- Middleware

## OODT Focus

- Focus on building *middleware* components for an enterprise data architecture
- Focus on building “profiles” for managing metadata information about cross-disciplinary resources
- Provide sufficient layers of abstraction in the architecture to isolate technologies choices from the architecture choices
- XML for the data content
- CORBA for the data transport
- Research technologies for implementing a distributed data architecture
- Distributed Object Computing (CORBA, DCOM, etc)
- Database Technology (RDBMS, ODBMS)
- Data Access Technologies (O/JDBC, STEP, XML, etc)
- Directory Implementations (LDAP)
- Data Interchange (XML)
- Communication Technologies (Web/HTTP, MOM, RPC, etc)

## OODT Pilot Activity

- Build a generic XML Document Type Definition (DTD) that will support PDS data dictionary and metadata infrastructure
- Demonstrate how a science query can return data across the PDS nodes
- Demonstrate how the same interface can return information between planetary and astrophysics data systems
- Partner with the Planetary Data System (PDS) to address interoperability across 10 PDS silos

## OODT Query Flow

- Query Server

- Profile Serverjpl

- Product Serverjpl.pti

- Product Serverjpl.pds

- Product Serverjpl.pds.mola

- Userquery

- XSL(profiles ordata productsformatted)

- XMLQuery(no results)

- XMLQuery(profiles ordata resultsas requested)

- XMLQuery(no results)

- XMLQuery(profiles of resources to handle query)

- XMLQuery (data results)

- XMLQuery (product search)

**Search Web Page**

**Profile DB**

**PTI Repository**

**PDS DVD Jukebox**

**PDS MOLA Oracle DB**

- QueryClient

- Web server

- search.jsp

## What is the PDS?

- PDS is the official planetary science data archive for the NASA.
- PDS is chartered to ensure that planetary data are archived and available to the scientific community.
- Publish and disseminate documented data sets for use in scientific analysis.
- Work with projects to help design, generate, and validate data products for placement in archive
- Develop and maintain archive data standards to ensure future usability.
- Provide expert scientific help to the user community.
- PDS is a distributed system designed to optimize scientific oversight in the archiving process.

## What has the PDS Accomplished?

- Produced a high-quality peer-reviewed archive of Solar System Exploration Data
- Stored for long-term viability
- Described by metadata
- Distributed either online or on CD media
- Developed a robust standards architecture
- Planetary Science Data Dictionary - Provides the “domain of discourse” for the planetary science community.
- Planetary Community Model - Provides formalized descriptions of the entities and their relationships within the planetary science community.
- Developed science driven management structure
- Responsive to changing mission project environment through distributed, science discipline oriented nodes.

## PDS Nodes and Institutions (Silos)

- NAIF/JPL

- Small Bodies/UMD

- Atmospheres/New Mexico State

- Geosciences/Washington University

- Planetary Plasma/UCLA

- Rings/Ames

- Radio Science/Stanford

- Central Node/JPL

- Imaging/USGS

- Imaging/JPL

## More Information

- “Science Search and Retrieval using XML” by OODT Team.  Presented at Second National Conference on Scientific and Technical Data, National Academy of Sciences,  Washington D.C.
- http://oodt.jpl.nasa.gov/doc/papers/codata/paper.pdf
- Planetary Data System
- http://pds.jpl.nasa.gov
- Dublin Core
- http://purl.oclc.org/dc
- Extensible Markup Language
-  http://www.w3c.org/XML
- ISO/IEC 11179: Specification and Standardization of Data Elements
- Object Management Group (CORBA and UML standards)
- http://www.omg.org
- Federal CIO Statement on Metadata
- http://www.cio.gov/docs/metadata.htm
- National Information Standards Organization Z39.50 Information Retrieval Protocol
- 	   http://www.niso.org/z3950.html

## Backup Slides

***Backup Slides***

## JPL Org Chart (partial)

- Caltech President

- JPL Office of the
- Director

- Institutional Computing/
- Chief Information Officer

- Science and Earth Science
- Programs

- Enterprise Applications
- Office

- Enterprise Infrastructure
- Office

- Engineering and Science
- Directorate

- Science Data Management
- and Archiving

- Planetary Data System

- Object Oriented Data
- Technology

## Org Chart - Responsibility Flow

- Science Data Management
- and Archiving

- Enterprise Applications
- Office

- Planetary Data System

- Object Oriented Data
- Technology

- Funding, Programmatic
- Oversight

- Deliverables

- Deliverables

- Deliverables

- Program Offices

- Implementation Organizations

- Task Management
- Design and Implementation Responsibility

## Data Archiving

- Archiving is a time-consuming and sometimes expensive task that culminates in giving one's data away. So why do it? ***
- Provide basic infrastructure for managing data long term
- Reinforces open scientific inquiry
- Encourages diversity of analysis and opinions
- Promotes new research and allows for the testing of new or alternative methods.
- Improves methods of data collection and measurement through the scrutiny of others
- Reduces costs by avoiding duplicate data collection efforts.
- Provides an important resource for training in research
****ICPSR Guide 1997*

## Institutional Enterprise Data Architecture Breakdown

- Paradigm shift from stove pipe implementations to horizontal solutions that cross organizational boundaries
- Include such services as
- Enterprise Application Standards
- Object Services
- Data Infrastructure Services
- Database Hosting
- Metadata Management
- Data Interchange
- Information Architecture Services
- Institutional directory and security access
- Data system APIs for access
- Data mining and data warehousing
- Data Management Services
- Data archiving

## JPL Enterprise Architecture (Logical View)

## What is a profile?

- A profile is a set of resource definitions implemented in XML for data products residing in one or more distributed systems
- Profile servers are CORBA servers that manage XML profile definitions
- Profile servers communicate via XML-over-CORBA
- Developed Java classes that map XML profiles to a Java object

- Profile Distributed Node Architecture

## Profile Server Architecture

## Profile DTD

- <!ELEMENT profiles
-   (profile+)>
- <!ELEMENT profile
-   (profAttributes,
-    resAttributes,
-    profElement*)>
-     <!ELEMENT profAttributes
-       (profId, profVersion*, profTitle*, profDesc*, profType*,
-        profStatusId*, profSecurityType*, profParentId*, profChildId*,
-        profRegAuthority*, profRevisionNote*, profDataDictId*)>
-     <!ELEMENT resAttributes
-       (Identifier, Title*, Format*, Description*, Creator*, Subject*,
-        Publisher*, Contributor*, Date*, Type*, Source*,
-        Language*, Relation*, Coverage*, Rights*,
-        resContext*, resAggregation*, resClass*, resLocation*)>
-     <!ELEMENT profElement
-       (elemId*, elemName, elemDesc*, elemType*, elemUnit*,
-        elemEnumFlag*, (elemValue | (elemMinValue, elemMaxValue))*,
-        elemSynonym*,
-        elemObligation*, elemMaxOccurrence*, elemComment*)>

## Profile Example

- <<profile>
-   <profAttributes>
-     <profId>OODT_PDS_DATA_SET_INV_82</profId>
-     <profDataDictId>OODT_PDS_DATA_SET_DD_V1.0</profDataDictId>
-   </profAttributes>
-   <resAttributes>
-     <Identifier>VO1/VO2-M-VIS-5-DIM-V1.0</Identifier>
-     <Title>VO1/VO2 MARS VISUAL IMAGING SUBSYSTEM DIGITAL IMAGING MODEL</Title>
-     <Format>text/html</Format>
-     <Language>en</Language>
-     <resContext>PDS</resContext>
-     <resAggregation>dataSet</resAggregation>
-     <resClass>data.dataSet</resClass>
-   <resLocation>http://pds.jpl.nasa.gov/cgi-bin/pdsserv.pl?OBJECT_ID=PDS100751</resLocation>
-   </resAttributes>
-     <profElement>
-       <elemId>ARCHIVE_STATUS</elemId>
-       <elemName>ARCHIVE_STATUS</elemName>
-       <elemType>ENUMERATION</elemType>
-       <elemEnumFlag>T</elemEnumFlag>
-       <elemValue>ARCHIVED</elemValue>
-     </profElement>
-     <profElement>
-       <elemId>TARGET_NAME</elemId>
-       <elemName>TARGET_NAME</elemName>
-       <elemType>ENUMERATION</elemType>
-       <elemEnumFlag>T</elemEnumFlag>
-       <elemValue>MARS</elemValue>
-     </profElement>
- </profile>

## OODT Product Server 

- The Product Server plugs into the OODT framework and manages the “handshake” between the data system and the OODT system.
- Extensible by dynamically loading objects at runtime which are specific to the data system model
- Queries and results are passed using an OODT XML Query structure

**Product Server**

- Query

- Result

- Generic Server

- Implementation Class

- File
- Sys

- Database

## XML Query Structure

- Defined as follows
- The query description
- The results
- Result 1
- Result Header
- Result Data
- Result 2
- Result Header
- Result Data
- ...
- Result N
- Result Header
- Result Data

## Why XML for OODT?

- XML doesn’t provide a “silver bullet,” but it does allow us to refocus the problem on metadata
- Metadata is a key to interoperability
- XML is language neutral
- Allows the designer to separate the data and the transport (re: CORBA vs XML-over-CORBA)
- Transport mechanism and data are not tied together
- Could be XML/HTTP
- Simpler deployments
- Simpler interfaces
-  Allows technologies to grow and change independently
- Real value of XML is the content

## CORBA vs XML

- XML over CORBA/IIOP
**  ****module jpl {****  module user {****    interface UserManager {****      ****string do(string xml);                        ****      **
**       ****};****  };****};**
**  **
**  ****<transaction>****  <findUser>****    <user>****      <surname>Doe</surname>****    </user>****  </findUser>****</transaction>**

- CORBA method
**  **
**  ****module jpl {****  module user {****    interface UserManager {****       User findUser(string  **
**                        ****name);****    };****    interface User {****      String getName();****    };****  };****};****  **

## Middleware Framework for OODT

- Archive
- Client

- OBJECT ORIENTED DATA TECHNOLOGY FRAMEWORK