# Presentation: 109295

## Object Oriented Data Technology

- Dan Crichton
- Steve Hughes
- JPL/Caltech
- CCSDS XML Workshop
- Oxford, England
- April 3-5, 2002

- Copyright Jet Propulsion Laboratory

## Overview

- Funded in 1998 by the NASA Office of Space Science to research technology in middleware and distributed object computing architectures to enable interoperability of data management systems

- Copyright Jet Propulsion Laboratory

## OODT Focus Areas

- Information Architectures
- Metadata Management
- Data Languages (i.e. XML)
- Distributed Architectures
- Messaging
- Distributed Object Computing
- P2P Architectures
- Development of an architecture for distributed information management and retrieval
- Application to the Planetary Data System to support Mars Odyssey Data Distribution

- Copyright Jet Propulsion Laboratory

## Middleware

- Web, GUI, console,
- 2D/3D visualization,
- immersion

- product data,
- real-time data

- Applications

- s/w, h/w,
- models

- Data

- User Interface

- N. Lamarra

- Copyright Jet Propulsion Laboratory

## Middleware Review

- History
- Pre-web: IDE (dev), client/server (apps)
- Post-web: n-tier c/s, thin/thick, frameworks, ...
- Communications: RPC, MOM, Web , ...
- Applications: Desktop and Remote, ...
- Architecture: Open Reference Model, HLA, TOGAF, ...
- Standards
- O/S: POSIX, Unix, Windows
- Communications: TCP/IP, OSI, DCE/ONC, HLA/RTI, HTTP
- DOC: CORBA/COSS, DCOM/COM+, Java RMI
- Data access: O/JDBC, EJB persistence, STEP, HTML, XML
- Security: SSL, SSH, SHTTP, IPSec, PKI, CDSA, BioAPI, etc.
- Directory: X.500, LDAP
- Systems Management: CIM, SNMP, CMIS/CMIP
- GUI: X11, CDE, Java-Swing

- Copyright Jet Propulsion Laboratory

## OODT Principles

- Location Independence
- Information Hiding
- Model driven architecture
- Scalable, Extensible
- Client APIs for locating and accessing distributed data products

- Copyright Jet Propulsion Laboratory

## Model Driven Architecture

- Decouple solutions from platform specific technology
- Allows data architecture and technology architecture to evolve independently
- The technology architecture supports the data architecture (i.e. it supports the model)
- Apply architecture to different problem spaces

- Copyright Jet Propulsion Laboratory

## OODT Architecture

- Address two key architectural focuses
- Technology Architecture
- Basic communication between geographically distributed data systems
- Framework for plugging in new data systems
- Data Architecture
- Standard method for describing data (profile)
- Standard method for exchanging data (XMLQuery)

- Copyright Jet Propulsion Laboratory

## OODT Standards

- The architecture for accessing distributed resources is based on the following standards:
- W3C Extensible Markup Language (XML) 1.0 (Second Edition) (Recommendation 6 October 2000)
- W3C HTML 4.01 Specification (Recommendation 24 December 1999)
- W3C XHTMLT 1.0: The Extensible Hypertext Markup Language (Recommendation 26 January 2000)
- ISO/IEC 11179 -  Specification and Standardization of Data Elements,  Parts 1-6, ISO/IEC specification, http://www.iso.ch/iso.
- Dublin Core Metadata Initiative.  The Dublin Core Element Set Version 1.1, Dublin: DMCI, July 1999.

- Copyright Jet Propulsion Laboratory

## OODT Technology Implementation

- Implemented in Java
- Data layer implemented with XML
- Uses CORBA for remote object invocation
- We use JacORB
- SSL for data encryption
- Uses a standardized XML DTD (XMLQuery) as the messaging format (See http://oodt.jpl.nasa.gov/dtd/xmlquery.dtd)
- Supports a variety of client access methods
- Java API
- HTTP
- C++ API

- Copyright Jet Propulsion Laboratory

## Why XML?

- XML doesn’t provide a “silver bullet,” but it does allow us to refocus the problem on metadata
- Metadata is a key to interoperability
- XML is language neutral
- Allows the designer to separate the data and the transport (re: CORBA vs XML-over-CORBA)
- Transport mechanism and data are not tied together
- Could be XML/HTTP
- Simpler deployments
- Simpler interfaces
- Allows technologies to grow and change independently
- Real value of XML is providing a content language for documenting the information architecture

- Copyright Jet Propulsion Laboratory

## XML and CORBA

- XML over CORBA/IIOP
**  ****module jpl {****  module user {****    interface UserManager {****      ****string do(string xml);                        ****      **
**       ****};****  };****};**
**  **
**  ****<transaction>****  <findUser>****    <user>****      <surname>Doe</surname>****    </user>****  </findUser>****</transaction>**

- CORBA method
**  **
**  ****module jpl {****  module user {****    interface UserManager {****       User findUser(string  **
**                        ****name);****    };****    interface User {****      String getName();****    };****  };****};****  **

- Copyright Jet Propulsion Laboratory

## OODT Data Architecture

- Use *Profiles* to describe data resources
- System Profiles
- Dataset Profiles
- Product Profiles
- A profile is just a metadata description of the resource defined by an XML DTD (see oodt/dtd/prof.xml)
- Uses Dublin Core
- Uses ISO 11179

- Copyright Jet Propulsion Laboratory

## What is a profile?

- Profiles describe and provide the location for anything of interest
- Things of interest (data and system resources)
- Data Systems, Data Sets, Web Sites, etc
- Written as XML documents
- Grouped
- Data Product profiles grouped by data set
- All others grouped into single database
- Provide sufficient information to determine if the resource can resolve a user query

- Copyright Jet Propulsion Laboratory

## Profile Overview Implementation

- A profile includes a pointer to the resources
- A profile has three main sections.
- The **profAttributes **(Profile Attributes) section describes the profile.
- The **resAttributes** (Resource Attributes) section describes the resource by using a common set of keywords (Dublin Core)
- The **profElement** (Profile Element) section describes the resource by using domain specific attributes (PDS keywords)

- Copyright Jet Propulsion Laboratory

## Profile DTD

- <!ELEMENT profiles
-   (profile+)>
- <!ELEMENT profile
-   (profAttributes,
-    resAttributes,
-    profElement*)>
-     <!ELEMENT profAttributes
-       (profId, profVersion*, profTitle*, profDesc*, profType*,
-        profStatusId*, profSecurityType*, profParentId*, profChildId*,
-        profRegAuthority*, profRevisionNote*, profDataDictId*)>
-     <!ELEMENT resAttributes
-       (Identifier, Title*, Format*, Description*, Creator*, Subject*,
-        Publisher*, Contributor*, Date*, Type*, Source*,
-        Language*, Relation*, Coverage*, Rights*,
-        resContext*, resAggregation*, resClass*, resLocation*)>
-     <!ELEMENT profElement
-       (elemId*, elemName, elemDesc*, elemType*, elemUnit*,
-        elemEnumFlag*, (elemValue | (elemMinValue, elemMaxValue))*,
-        elemSynonym*,
-        elemObligation*, elemMaxOccurrence*, elemComment*)>

- Copyright Jet Propulsion Laboratory

## ?xml version="1.0"?>

- ..
- <!DOCTYPE profiles SYSTEM “http://ginger.fhcrc.org/dtd/prof.dtd">
- <profiles> <profile>
- <profAttributes>
- 	<profId>1.1.1.1.102</profId>
- 	<profType>profile</profType>
- 	<profStatusId>ACTIVE </profStatusId>
- </profAttributes>
- <resAttributes>
- 	<Identifier>EDRN_MOFFITT_PRODUCT_SERVER</Identifier>
- 	<Title>EDRN H. Lee Moffitt Cancer Center Product Server</Title>
- 	<Format>text/html</Format>
- 	<Description>A product server providing access to a database of clinical sample information for the Early Detection Research Network (EDRN) at the H. Lee Moffitt Cancer Center.</Description>
- 	<Language>en</Language>
- 	<resContext>NIH.NCI.EDRN.MOFFITT</resContext>
- 	<resAggregation>granule</resAggregation>
- 	<resClass>system.productServer</resClass>
- 	<resLocation>iiop://oodt.jpl.nasa.gov:10000/NIH.NCI.EDRN.MOFFITT.PRODUCT_SERVER</resLocation>
- </resAttributes>
- <profElement>
- 	<elemId>STUDY_PARTICIPANT_ID</elemId>
- 	<elemName>STUDY_PARTICIPANT_ID</elemName>
- 	<elemDesc>EDRN Participant ID</elemDesc>
- 	<elemType>CHARACTER</elemType>
- 	<elemEnumFlag>F</elemEnumFlag>
- </profElement>
- <...>

- Copyright Jet Propulsion Laboratory

## Adopt an XML Query Structure for Messaging

- Support queries across distributed resources
- Provide a standard for inter-communication between heterogeneous applications and computing environments
- Describe both queries and results

- Copyright Jet Propulsion Laboratory

## XMLQuery

- Implemented in XML at http://oodt.jpl.nasa.gov/dtd/xmlquery.dtd
- Defined as follows
- The query description
- The results
- Result 1
- Result Header
- Result Data
- Result 2
- Result Header
- Result Data
- ...
- Result N
- Result Header
- Result Data
- Final results are aggregated from all profile/product servers

- Copyright Jet Propulsion Laboratory

## XML Query Example (1 of 2)

- <query>
-  <queryAttributes>
-   <queryId>1.2.3.4...</queryId>
-   <queryTitle>OODT_XML_QUERY Query Example</queryTitle>
-   <queryDesc>Query for TARGET_NAME = MARS</queryDesc>
-   <queryType>QUERY</queryType>
-   <queryStatusId>ACTIVE</queryStatusId>
-   <querySecurityType>UNKNOWN</querySecurityType>
-   <queryRevisionNote>2000-05-12 JSH V1.2 Updated for new
-                                                                                   prof.dtd</queryRevisionNote>
-   <queryDataDictId>1.2.3.5...</queryDataDictId>
-  </queryAttributes>
-  <queryResultModeId>ATTRIBUTE</queryResultModeId>
-  <queryPropogationType>BROADCAST</queryPropogationType>
-  <queryPropogationLevels>N/A</queryPropogationLevels>
-  <queryMaxResults>100</queryMaxResults>
-  <queryResults>0</queryResults>
-  <queryKWQString>TARGET_NAME = MARS</queryKWQString>

- Copyright Jet Propulsion Laboratory

## XML Query Example (2 of 2)

- <querySelectSet></querySelectSet>
-  <queryFromSet></queryFromSet>
-  <queryWhereSet>
-   <queryElement>
-    <tokenRole>elemName</tokenRole>
-    <tokenValue>TARGET_NAME</tokenValue>
-   </queryElement>
-   <queryElement>
-    <tokenRole>LITERAL</tokenRole>
-    <tokenValue>MARS</tokenValue>
-   </queryElement>
-   <queryElement>
-    <tokenRole>RELOP</tokenRole>
-    <tokenValue>EQ</tokenValue>
-   </queryElement>
-  </queryWhereSet>
-  <queryResultSet></queryResultSet>
- </query>

- Copyright Jet Propulsion Laboratory

## OODT Components to Support the Architecture

- Name Server
- Query Server
- Archive Server
- Profile Server
- Product Server
- Server Manager

- Copyright Jet Propulsion Laboratory

## OODT Software Components

- Name Server
- CORBA name server that maps an object to a location (i.e. ern:NIH.NCI.EDRN.MOFFITT.PRODUCT_SERVER can be communicated at edrn.moffitt.edu:7877)
- All implementations of CORBA, DCE, DCOM, SOAP, etc use a name server
- Only needs to be run once at a central site
- Query Server
- Implemented by OODT
- Multi-threaded server for managing queries to distributed servers
- Aggregates results from multiple servers
- Only needs to be run once at a central site

- Copyright Jet Propulsion Laboratory

## Archive Server

- Provides a server framework for
- Adding products remotely to a generic archive
- Retrieving products remotely from a generic archive
- A generic data model that is driven by the metadata model
- Captures a catalog of archived products using well defined data elements
- Uses XML to label data products
- Extract *Profiles* of archived products

- Copyright Jet Propulsion Laboratory

## Profile Server

- Used to identify the existence and location of resources in a distributed data environment
- Two types of profile servers
- Dynamic Profile Server
- Front ends an existing database catalog to extract the resources and format them as profiles
- Implemented as a Java Interface class
- Static Profile Server
- Manages its own local resources
- XMLQuery is the messaging structure to query and return profiles
- As many profile servers can be run as needed to scale the system.  Typically one per catalog.

- Copyright Jet Propulsion Laboratory

## Profile Server Overview

- Profile Servers serve profiles
- Allow search and retrieval of profiles
- Retrieves from profile databases
- Static profiles stored as XML documents
- Dynamic profiles generated from information stored in databases
- Can be distributed for local management and scalability at a site

- Copyright Jet Propulsion Laboratory

## Profile Server Requirements

- A profile server shall search and retrieve profiles from a profile database
- For search, a profile server shall allow any profile attribute as a query constraint. Profile attributes include those from the profile element, resource attribute, and profile attribute sections of the profile document.
- For retrieval, a profile server shall return matching profiles. The user can request the complete profile or any subset of the profile.

- Copyright Jet Propulsion Laboratory

## Profile ServerXML Interface

- Input: XMLQUERY – A profile server accepts an XMLQUERY XML document as input. User query constraints have been encoded into the WHERE clause of the XMLQUERY. The requested profile attribute results have been encoded into the SELECT clause.
- Output: XMLQUERY – A profile server inserts the set of  matching profiles in the RESULTS clause of the XMLQUERY.

- Copyright Jet Propulsion Laboratory

## Profile ServerServlet Interface

- User calls optional servlet using “DIS” query
- Servlet creates XMLQUERY and calls predetermined profile server
- Results returned in XMLQUERY
- Query Server passes parallel query to each product server that has a match
- htpp://ginger.fhcrc.org/servlet/jpl.oodt.servlets.Profile_Servlet?
-  keywordQuery={CANCER_SPECIMEN_TYPE = BLOOD
-           &object= NIH.NCI.EDRN.DMCC.PROFILE_SERVER
-           &mimeType=XML
-           &mimeType=*/*}

- Copyright Jet Propulsion Laboratory

## Profile Server Distributed   Architecture

- Copyright Jet Propulsion Laboratory

## Product Server

- Used to return a product using the XMLQuery messaging structure
- Implemented as a Java Server
- Contains one or more query handlers
- Query handler implemented as a Java interface class
- Query handlers loaded at server start time
- Defined in a local configuration file
- Can use any method to access and return data

- Copyright Jet Propulsion Laboratory

## Product Server Implementation

- The Product Server plugs into the OODT framework and manages the “handshake” between the data system and the OODT system.
- Extensible by dynamically loading objects at runtime which are specific to the data system model
- Queries and results are passed using an OODT XML Query structure

**Product Server**

- Query

- Result

- Generic Server

- Implementation Class

- File
- Sys

- Database

- Copyright Jet Propulsion Laboratory

## Product Servers for Planetary Data System

- Profile Server

**Distributed**
**Profile**
**Servers**

**Product Servers for Planetary Data System**

- HTTP, IIOP,
- Java, C++
**  ****APIs**

- jdbc://

- jdbc://

- Data Source Interface
- For
- Dynamically Loaded Query Handlers

- Java Server Framework

- PDS.ASU.THEMIS.PRODUCT_SERVER

- Java Server Framework

- PDS.IMAGING.PRODUCT_SERVER

- Data Source Interface
- For
- Dynamically Loaded Query Handlers

- HTTP, IIOP,
- Java, C++
**  ****APIs**

- Copyright Jet Propulsion Laboratory

## Layered OODT Product Server Architecture

**Unix/Windows operating system**

**Java**

**OODT**

**CORBA/SSL**

**Local Product servers**

- Provided by JPL

- Provided by hosting site

- Stability

- Copyright Jet Propulsion Laboratory

**Notes:**
Millions of units and years of development have made for a stable computer platform.  Even the Windows operating system is somewhat stable.  But as you move up in the architecture, things get less stable and require more debugging and testing.

The problem arises in that each EDRN member site provides the hardware and operating system, whereas JPL implements the product server.  Debugging product servers has to somehow happen with a developer in California and a system in Florida, Texas, Washington, New Hampshire, Pennsylvania, etc.

## Server Manager

- Distributes software
- Manages distributed software
- Remote Management and Debugging
- Automatic process management
- APIs to allow tools to inquire about server status (i.e PDS Themis Node at ASU is up or down)

- Copyright Jet Propulsion Laboratory

## Challenges with Distributed System Integration

- Off site development
- Debugging by telephone
- New databases and database drivers
- Inexperience led to programming errors
- Institutional Challenges
- Finicky configurations
- Mysterious network issues

- Copyright Jet Propulsion Laboratory

**Notes:**
Although OODT’s software was highly tested and considered stable, the individual product servers were new development.  Worse, because JPL could not reproduce the environment in which they ran, they had to be run at each member site.  That meant debugging was often by email and telephone, a slow, and painful process.

JPL’s database experience is mostly in Oracle.  The developers on this project never before had exposure to Filemaker and Access, or the corresponding JDBC/ODBC drivers.

The requirement to encrypt all communication led to the adoption of an encryption package advertised to work with the freeware CORBA package selected for OODT.  However, perhaps as a result of “you get what you pay for,” configuring and keeping this CORBA package running was a challenge.

## Possible Solutions

- Remote PC control
- Unacceptable security risk at some sites
- Mandate Unix
- Lack of IT experience at member sites
- Retooling time too great
- Put remote management into the product server itself

- Copyright Jet Propulsion Laboratory

**Notes:**
Although a product like pcAnywhere or a freeware system like VNC would’ve saved everyone time and money and reduced headaches, one of the sites nearly went ballistic at the suggestion.

## Server Manager

- Manages servers
- Server manager runs at each site
- Client/server implementation
- XML-RPC with HTTP authentication
- One-click installation
- Well, almost: install Java first
- Capture Log Files
- Perform restarts

- Copyright Jet Propulsion Laboratory

## Remote Control Application

- Authenticates with a server manager
- Enables remote administration of server processes
- View log files and process output
- Set Java runtime parameters
- Update files
- Define, start, stop processes
- Execute upgrade scripts

- Copyright Jet Propulsion Laboratory

## Remote Control User Interface

- Copyright Jet Propulsion Laboratory

## Software Distribution Environment

- Installation
- First, you install Java
- Windows users download and run one self-installer
- pds-setup.exe
- Displays typical Windows-style wizard installer
- Unix/Mac users download a tarball and follow instructions in a README file

- Copyright Jet Propulsion Laboratory

## OODT Configurations

- oodtrc.xml
- Local configuration file that contains
- Server names
- List of profile servers
- Logging methods
- Timeout methods
- Database configuration parameters
- ...

- Copyright Jet Propulsion Laboratory

## OODT Applied to PDS

- Provides a common messaging infrastructure for PDS
- Provides data abstraction and location independence for PDS Nodes
- Common tools can look across the PDS sites
- Heterogeneous implementations are standardized
- Uses the PDS Data Dictionary
- Uses existing PDS databases and catalogs

- Copyright Jet Propulsion Laboratory

## Generic Architecture

- Query Server

- Profile Server

- Site 3
- Product Server

- XMLQuery

- XMLQuery

- XMLQuery

- XMLQuery

- XMLQuery

**Profile**
**Server**

- QueryClient

- Web server
- Plugins

- Web Server

- XMLQuery

**Site 1 DB**

**Site 2 DB**

**Site 3 DB**

- Web I/F

- Desktop I/F

- XMLQuery

- XMLQuery

- XMLQuery

- XMLQuery

- Name Server

**Client**
**Environment**

**Distributed Repositories**

- Site 2
- Product Server

- Site 1
- Product Server

- Copyright Jet Propulsion Laboratory

## Profile/Location

-  Server

- Middleware

- Single Point of Entry
- Data Set View

- Data Set Selection
- Page

- Copyright Jet Propulsion Laboratory

## OODT Projects

- PDS D01 will use OODT to support Mars Odyssey Distribution (October 2002)
- PDS D02 will rollout OODT to all of PDS
- OODT building a data architecture for the National Cancer Institute’s (NCI) Early Detection Research Network integrating up to 40 sites by 2005
- OODT Supporting development of a JPL Institutional Data Architecture with Data Management Services

- Copyright Jet Propulsion Laboratory

## OODT References

- OODT Overview
- _[http://oodt.jpl.nasa.gov](http://oodt.jpl.nasa.gov/)_
- OODT Papers
- _[http://oodt.jpl.nasa.gov/doc/papers](http://oodt.jpl.nasa.gov/doc/papers)_

- Copyright Jet Propulsion Laboratory