# Presentation: 026009

## A Distributed Component Framework for Science Data Product Interoperability 17th International CODATA Conference  October  15-19, 2000

- Steve Hughes/JPL Steve.Hughes@jpl.nasa.gov
- Sean Kelly/UTA Sean.Kelly@jpl.nasa.gov
-  Sean Hardman/JPL Sean.Hardman@jpl.nasa.gov
*Jet Propulsion Laboratory, California Institute of Technology*
*National Aeronautics and Space Administration*

***A Distributed Component Framework for Science Data Product Interoperability****** ******17******th****** International CODATA Conference****** **** ****October  15-19, 2000***

## Problem Statement

- Problem: Science data is highly distributed across geographically heterogeneous data systems.  It is difficult to access, and the systems do not interoperate well.  There is no common interchange mechanism, nor is there a common architecture. Correlation of data across these systems is problematic.
*Solution: Design an enterprise data architecture that supports cross disciplinary solutions for data management and archiving including interoperability among science data systems.*

## What is an Enterprise Data Architecture?

- An enterprise data architecture provides the infrastructure necessary to enable development of interoperable, enterprise-wide applications
- Focus on providing the key data management infrastructure
- Data Archiving
- Managing Local Data
- Search and Retrieval
- Data Location
- Managing Profiles
- Data Access
- Data sharing between data systems
- Data Interoperability

## Why is an EDA Critical?

- Interoperability is an important *key* to unlock knowledge discovery
- Allows scientists the ability to locate critical information
- Enables *knowledge management* across an agency
- A key to scientific discovery
- State of data systems across agency
- Difficult to access (no standard interfaces)
- Geographically distributed
- Have no standard language or protocol for interchange (no EDI) agency wide
- No common metadata language agency wide
- Have no system for registration of data products
- Have little or no interoperability
- Have few common terms for describing data

## Object Oriented Data Technology Task

- Research task funded by the Office of Space Science (OSS) at NASA
- Provides a framework for managing data access and interoperability
- Archive Service:  For managing data sets
- Profile Service: For managing metadata profiles about data systems, data sets, and data products
- Product Service: To tie individual data systems into a larger enterprise data system
- Build data system solutions that are cross disciplinary
- Presented a paper at CODATA in March 2000 called “Science Search and Retrieval using XML”

## OODT Goals

- Encapsulate individual data systems to hide uniqueness
- Provide data system location independence
- Require that communication between distributed systems use metadata
- Use a standard data dictionary for describing systems and resources
- Provide a scaleable and extensible solution
- Provide a mechanism for data product exchange
- Allow systems using different data dictionaries to be integrated

## OODT Focus

- Focus on building *middleware* components for an enterprise data architecture
- Focus on building “profiles” for managing metadata information about cross-disciplinary resources
- Provide sufficient layers of abstraction in the architecture to isolate technologies choices from the architecture choices
- XML for the data content
- CORBA for the data transport
- Research technologies for implementing a distributed data architecture
- Distributed Object Computing (CORBA, DCOM, etc)
- Database Technology (RDBMS, ODBMS)
- Data Access Technologies (O/JDBC, STEP, XML, etc)
- Directory Implementations (LDAP)
- Data Interchange (XML)
- Communication Technologies (Web/HTTP, MOM, RPC, etc)

## Focus on Middleware

- 	In the computer industry, middleware is a general term for any programming that serves to “glue together” or mediate between two separate and usually already existing programs. A common application of middleware is to allow programs written for access to a particular database to access other databases.
- 	Messaging is a common service provided by middleware programs so that different applications can communicate. The systematic tying together of disparate applications is known as enterprise application integration.
- 							*http://www.whatis.com*

## Role of Middleware

- Middleware can tie application, data, and user interfaces together and hide the unique interfaces

- Applications

- User Interface

- Data

- Middleware

## Middleware (Cont)

- Middleware allows for the encapsulation of individual data systems
- Hide uniqueness by introducing the data architecture layer
- Ties distributed applications together an often works with a Electronic Data Interchange (EDI) type mechanism
- Enables reuse and promotes standards

## Focus on Metadata

- Metadata is data about data
- Provides descriptive information about the data
- Classification, identification, etc
- Metadata Example
- Data Value: 55 (not descriptive)
- Metadata Values:
- Data Element Name:Vehicle_Speed
- Unit: Miles per Hour
- Description: The average velocity of a vehicle.
- Use standards where appropriate
- ISO/IEC 11179 – A framework for the Specification and Standardization of Data Elements
- Dublin Core – A metadata element set intended to facilitate discovery of electronic resources.

## Data Search and Retrieval

- Space scientists can not easily locate or use data across the hundreds if not thousands of autonomous, heterogeneous, and distributed data systems currently in the Space Science community.
- Heterogeneous Systems
- Data Management - RDBMS, ODBMS, HomeGrownDBMS, BinaryFiles
- Platforms - UNIX, LINUX, WIN3.x/9x/NT, Mac, VMS, ...
- Interfaces - Web, Windows, Command Line
- Data Formats - HDF, CDF, NetCDF, PDS, FITS, VICR, ASCII, ...
- Data Volume - KiloBytes to TeraBytes
- Heterogeneous Disciplines
- Moving targets and stationary targets
- Multiple coordinate systems
- Multiple data object types (images, cubes,  time series,  spectrum, tables,
- binary, document)
- Multiple interpretations of single object types
- Multiple software solutions to same problem.
- Incompatible and/or missing metadata

## Solutions to Data Search

- Build metadata “profiles*”* that describe data system resources
- Encapsulate individual data systems resources. (Hide uniqueness.)
- Communicate using metadata. (Provide metadata with data)
- Enable interoperability based on metadata compatibility.
- Refocus problem on metadata development.
- Provide a core framework of software components to interconnect distributed data systems

## Profile DTD

- <!ELEMENT profiles
-   (profile+)>
- <!ELEMENT profile
-   (profAttributes,
-    resAttributes,
-    profElement*)>
-     <!ELEMENT profAttributes
-       (profId, profVersion*, profTitle*, profDesc*, profType*,
-        profStatusId*, profSecurityType*, profParentId*, profChildId*,
-        profRegAuthority*, profRevisionNote*, profDataDictId*)>
-     <!ELEMENT resAttributes
-       (Identifier, Title*, Format*, Description*, Creator*, Subject*,
-        Publisher*, Contributor*, Date*, Type*, Source*,
-        Language*, Relation*, Coverage*, Rights*,
-        resContext*, resAggregation*, resClass*, resLocation*)>
-     <!ELEMENT profElement
-       (elemId*, elemName, elemDesc*, elemType*, elemUnit*,
-        elemEnumFlag*, (elemValue | (elemMinValue, elemMaxValue))*,
-        elemSynonym*,
-        elemObligation*, elemMaxOccurrence*, elemComment*)>

## XML Profile Example (1 of 2)

- <profile>
-   <profAttributes>
-     <profId>OODT_PDS_DATA_SET_INV_82</profId>
- <profDataDictId>OODT_PDS_DATA_SET_DD_V1.0</profDataDictId>
-   </profAttributes>
-   <resAttributes>
-     <Identifier>VO1/VO2-M-VIS-5-DIM-V1.0</Identifier>
-     <Title>VO1/VO2 MARS VISUAL IMAGING SUBSYSTEM DIGITAL ...</Title>
-     <Format>text/html</Format>
-     <Language>en</Language>
-     <resContext>PDS</resContext>
-     <resAggregation>dataSet</resAggregation>
-     <resClass>data.dataSet</resClass>
-   <resLocation>http://pds.jpl.nasa.gov/cgi-bin/pdsserv.pl?...</resLocation>
-   </resAttributes>

## XML Profile Example (2 of 2)

- <profElement>
-       <elemId>ARCHIVE_STATUS</elemId>
-       <elemName>ARCHIVE_STATUS</elemName>
-       <elemType>ENUMERATION</elemType>
-       <elemEnumFlag>T</elemEnumFlag>
-       <elemValue>ARCHIVED</elemValue>
-     </profElement>
-     <profElement>
-       <elemId>TARGET_NAME</elemId>
-       <elemName>TARGET_NAME</elemName>
-       <elemType>ENUMERATION</elemType>
-       <elemEnumFlag>T</elemEnumFlag>
-       <elemValue>MARS</elemValue>
-     </profElement>
- </profile>

## Data Access

- Access to distributed data systems and databases is difficult
- Vendor database products
- Data model implementations
- Representations of data
- Platforms
- O/S
- etc
- ...are all different

## Solutions to Data Access

- Provide a framework to support common access to distributed data systems
- Plug into an overall data architecture solution
- Consistent metadata
- Consistent data interchange
- Build “product servers” which negotiate the interface between the infrastructure and the data system implementation
- Provide a middleware framework to tie the data architecture together
- Provide data abstraction
- Data and information hiding
- Location hiding and independence
- Provide a standard language for communication
- Use XML Query language for data interchange
- Use rich metadata to describe queries and results

## XML Query Example (1 of 2)

- <query>
-  <queryAttributes>
-   <queryId>OODT_XML_QUERY_V0.1</queryId>
-   <queryTitle>OODT_XML_QUERY - PDS DIS Query Example</queryTitle>
-   <queryDesc>PDS DIS Query for TARGET_NAME = MARS</queryDesc>
-   <queryType>QUERY</queryType>
-   <queryStatusId>ACTIVE</queryStatusId>
-   <querySecurityType>UNKNOWN</querySecurityType>
-   <queryRevisionNote>2000-05-12 JSH V1.2 Updated for new
-                                                                                   prof.dtd</queryRevisionNote>
-   <queryDataDictId>OODT_PDS_DATA_SET_DD_V1.0</queryDataDictId>
-  </queryAttributes>
-  <queryResultModeId>ATTRIBUTE</queryResultModeId>
-  <queryPropogationType>BROADCAST</queryPropogationType>
-  <queryPropogationLevels>N/A</queryPropogationLevels>
-  <queryMaxResults>100</queryMaxResults><queryResults>0</queryResults>
-  <queryKWQString>TARGET_NAME = MARS</queryKWQString>

## XML Query Example (2 of 2)

- <querySelectSet></querySelectSet>
-  <queryFromSet></queryFromSet>
-  <queryWhereSet>
-   <queryElement>
-    <tokenRole>elemName</tokenRole>
-    <tokenValue>TARGET_NAME</tokenValue>
-   </queryElement>
-   <queryElement>
-    <tokenRole>LITERAL</tokenRole>
-    <tokenValue>MARS</tokenValue>
-   </queryElement>
-   <queryElement>
-    <tokenRole>RELOP</tokenRole>
-    <tokenValue>EQ</tokenValue>
-   </queryElement>
-  </queryWhereSet>
-  <queryResultSet></queryResultSet>
- </query>

## Data Archiving

- Promote data archiving best practices at the data system level.
- Support short-term requirements
- Support convenient and efficient data retrieval.
- Reduce data redundancy.
- Support multiple users.
- Provide data security.
- Improve consistency.
-  Long Term Requirements
- Ensure data remains viable.
- Ensure data remains useable.
- Ensure data remains understandable.

## OODT Query Flow

- Query Server

- Profile Serverjpl

- Product Serverjpl.pti

- Product Serverjpl.pds

- Product Serverjpl.pds.mola

- Userquery

- XSL(profiles ordata productsformatted)

- XMLQuery(no results)

- XMLQuery(profiles ordata resultsas requested)

- XMLQuery(no results)

- XMLQuery(profiles of resources to handle query)

- XMLQuery (data results)

- XMLQuery (product search)

**Search Web Page**

**Profile DB**

**PTI Repository**

**PDS DVD Jukebox**

**PDS MOLA Oracle DB**

- QueryClient

- Web server

- search.jsp

## OODT Product Server 

- The Product Server plugs into the OODT framework and manages the “handshake” between the data system and the OODT system.
- Extensible by dynamically loading objects at runtime which are specific to the data system model
- Queries and results are passed using an OODT XML Query structure
- Encapsulates one or more data sources for standardized access

**Product Server**

- Query

- Result

- Generic Server

- Implementation Class

- File
- Sys

- Database

## Results Slide

## OODT Insertion in the PDS

- Focused research activity on information technology in support of space science data systems
- Providing a long term architecture to improve the ability for scientists to retrieve data within the PDS
- Refocus the problem away from technology solutions
- Provide and leverage a metadata infrastructure
- Providing new solutions for data management in order to access and correlate heterogeneous data products archived in distributed heterogeneous data systems
- Reusing a metadata infrastructure that exists
- Supporting the PDS distributed node architecture

## What is the PDS?

- PDS is the official planetary science data archive for NASA.
- PDS is chartered to ensure that planetary data are archived and available to the scientific community.
- Publish and disseminate documented data sets for use in scientific analysis.
- Work with projects to help design, generate, and validate data products for placement in archive
- Develop and maintain archive data standards to ensure future usability.
- Provide expert scientific help to the user community.
- PDS is a distributed system designed to optimize scientific oversight in the archiving process.

## What has the PDS Accomplished?

- Produced a high-quality peer-reviewed archive of Solar System Exploration Data
- Stored for long-term viability
- Described by metadata
- Distributed either online or on CD media
- Developed a robust standards architecture
- Planetary Science Data Dictionary - Provides the “domain of discourse” for the planetary science community.
- Planetary Community Model - Provides formalized descriptions of the entities and their relationships within the planetary science community.
- Developed science driven management structure
- Responsive to changing mission project environment through distributed, science discipline oriented nodes.

## PDS Nodes and Institutions (Silos)

- NAIF/JPL

- Small Bodies/UMD

- Atmospheres/New Mexico State

- Geosciences/Washington University

- Planetary Plasma/UCLA

- Rings/Ames

- Radio Science/Stanford

- Central Node/JPL

- Imaging/USGS

- Imaging/JPL

## OODT Outside Opportunities

- Early Detection Research Network from the National Cancer Institute (NCI)
- Interested in reusing the OODT technology to link data from distributed data centers in support Biomarkers Research
- Children’s Hospital, Los Angeles and Johns Hopkins Medical Institute
- Interested in reusing the OODT technology to link pediatric physiological data between the hospitals

## More Information

- “Science Search and Retrieval using XML” by OODT Team.  Presented at Second National Conference on Scientific and Technical Data, National Academy of Sciences,  Washington D.C.
- http://oodt.jpl.nasa.gov/doc/papers/codata/paper.pdf
- Planetary Data System
- http://pds.jpl.nasa.gov
- Dublin Core
- http://purl.oclc.org/dc
- Extensible Markup Language
-  http://www.w3c.org/XML
- ISO/IEC 11179: Specification and Standardization of Data Elements
- Object Management Group (CORBA and UML standards)
- http://www.omg.org
- Federal CIO Statement on Metadata
- http://www.cio.gov/docs/metadata.htm
- National Information Standards Organization Z39.50 Information Retrieval Protocol
- 	   http://www.niso.org/z3950.html

## Backup Slides

***Backup Slides***

## OODT Metadata Development

- Metadata Registry – Develop a data management system for managing the semantics of data that is shared within and between domains.
- Terminology Base – Domain specific name space.
- Data Dictionary – Inventory of domain terms with definitions and other distinguishing attributes.
- Ontology – A set of concepts, their relationships and constraints, all within the scope of a domain.
- XML for metadata registry and communication
- Several I.T. efforts have shown the criticality of metadata in enabling data sharing and system interoperability.

## Data Archiving

- Archiving is a time-consuming and sometimes expensive task that culminates in giving one's data away. So why do it? ***
- Provide basic infrastructure for managing data long term
- Reinforces open scientific inquiry
- Encourages diversity of analysis and opinions
- Promotes new research and allows for the testing of new or alternative methods.
- Improves methods of data collection and measurement through the scrutiny of others
- Reduces costs by avoiding duplicate data collection efforts.
- Provides an important resource for training in research
****ICPSR Guide 1997*

## JPL Enterprise Architecture (Logical View)

## Why XML for OODT?

- XML doesn’t provide a “silver bullet,” but it does allow us to refocus the problem on metadata
- Metadata is a key to interoperability
- XML is language neutral
- Allows the designer to separate the data and the transport (re: CORBA vs XML-over-CORBA)
- Transport mechanism and data are not tied together
- Could be XML/HTTP
- Simpler deployments
- Simpler interfaces
-  Allows technologies to grow and change independently
- Real value of XML is the content

## CORBA vs XML

- XML over CORBA/IIOP
**  ****module jpl {****  module user {****    interface UserManager {****      ****string do(string xml);                        ****      **
**       ****};****  };****};**
**  **
**  ****<transaction>****  <findUser>****    <user>****      <surname>Doe</surname>****    </user>****  </findUser>****</transaction>**

- CORBA method
**  **
**  ****module jpl {****  module user {****    interface UserManager {****       User findUser(string  **
**                        ****name);****    };****    interface User {****      String getName();****    };****  };****};****  **

## Middleware Framework for OODT

- Archive
- Client

- OBJECT ORIENTED DATA TECHNOLOGY FRAMEWORK