# Presentation: 495738

## Object Oriented Data Technology for Space Science Data Archiving and Retrieval: Potential Applications in Biomedical Research

- March 21, 2001
- Dan  Crichton
- Steve Hughes
- Jet Propulsion Labs
- California Institute of Technology
- National Aeronautics and Space Administration

## Who are we?

- JPL is NASA’s lead center for robotic exploration of the universe
- JPL has an enormous amount of data that it needs to manage from scientific data, to engineering, to institutional
- We represent several efforts in both the research and enterprise side of JPL that is addressing enterprise architectures for integrating data at both JPL and NASA.  Such efforts include
- Planetary Data System - NASA Planetary Archive
- Enterprise Data Architecture and Applications
- JPL IT Data Infrastructure
- Knowledge Management - JPL/NASA KM Initiative
- Object Oriented Data Technology - I.T. Data System Research

***Who are we?***

## Challenges to Data ManagementArchiving, Search, Retrieval and Integration

- Space scientists cannot easily locate or use data across the hundreds if not thousands of autonomous, heterogeneous, and distributed data systems currently in the Space Science community.
- Heterogeneous Systems
- Data Management - RDBMS, ODBMS, HomeGrownDBMS, BinaryFiles
- Platforms - UNIX, LINUX, WIN3.x/9x/NT, Mac, VMS, ...
- Interfaces - Web, Windows, Command Line
- Data Formats - HDF, CDF, NetCDF, PDS, FITS, VICR, ASCII, ...
- Data Volume - KiloBytes to TeraBytes
- Heterogeneous Disciplines
- Moving targets and stationary targets
- Multiple coordinate systems
- Multiple data object types (images, cubes,  time series,  spectrum, tables,
- binary, document)
- Multiple interpretations of single object types
- Multiple software solutions to same problem
- Incompatible and/or missing metadata

## Evolution of Data Systems(Trying to make order out of entropy)

- Data System Maturity

- _Basic Data _
- _Infrastructure_
-  - Data Acquisition
-  - Databases
-  - Data Analysis
-    Tools
-  - Homogeneous
-    Computing

- _Data Archiving_
-    - Catalog Systems
-    - Data sets
-    - Data Products
-    - Metadata

- _Data Location_
-    - Metadata
-    - Distributed Data sets
-    - Distributed Services

- _Data Product Exchange/_
- _Interoperability_
-    - Heterogeneous Servers
-    - Data Interchange
-    - Data Sharing
-    - Distributed Architectures

*Centralized Data*

*Distributed Data*

## Independent Access to PDS Archives

***Mars Odyssey***

**Mars Global**
**Surveyor**

**Viking**

**Ancillary**
**Files**

**Viking **
**Interface**

***Odyssey***
*** ******Interfaces***

## PDS Nodes and Institutions (Silos)

- NAIF/JPL

- Small Bodies/UMD

- Atmospheres/New Mexico State

- Geosciences/Washington University

- Planetary Plasma/UCLA

- Rings/Ames

- Radio Science/Stanford

- Central Node/JPL

- Imaging/USGS

- Imaging/JPL

## Managing Software Interfaces(# of Systems vs # of Interfaces)

- # of interfaces = (n2 - n) / 2 where the number of systems is n

- # of systems

- # of interfaces

## DIS Approach (circa 1998)

**PDS Distributed Inventory System (DIS)**
- Identified all resources available across the PDS
- Used Object Description Language (keyword/value) labels to describe data and non-data resources
- Included URL/FTP links to all online resources
- Used cgi-script engine to search label database
- Displayed results in an HTML page

## Product Label

**OBJECT = DATA_SET ;**
**  ****DATA_SET_NAME = VO1/VO2 MARS VISUAL IMAGING SUBSYSTEM DIGITAL IMAGING MODEL;**
**  ****DATA_SET_ID = VO1/VO2-M-VIS-5-DIM-V1.0;**
**  ****DATA_SET_DESC = http://pds.jpl.nasa.gov/cgi-bin/remote.pl?ds:VO1/VO2%2BMARS%...;**
**  ****DATA_SET_TERSE_DESC = http://pds.jpl.nasa.gov/cgi-bin/remote.pl?ds:VO1/VO2%2BM...;**
**  ****DATA_SET_RELEASE_DATE = 1991;**
**  ****LINK = http://www-pdsimage.wr.usgs.gov/PDS/public/mapmaker/mapmkr.htm;**
**  ****ARCHIVE_STATUS = ARCHIVED;**
**  ****ARCHIVE_STATUS_NOTE = Passed peer review with all liens resolved. Available through PDS and NSSDC;**
**  ****ARCHIVE_SCHEDULE_NOTE = 1999-08-18:ARCHIVED;**
**  ****DATA_OBJECT_TYPE = IMAGE;**
**  ****START_TIME = N/A;**
**  ****STOP_TIME = N/A;**
**  ****NODE_NAME = IMAGING;**
**  ****CURATING_NODE_ID = IMAGING;**
**  ****DATA_ENGINEER_ID = N/A;**
**  ****TARGET_NAME = MARS;**
**  ****TARGET_TYPE = PLANET;**
**  ****MISSION_NAME = VIKING;**
**  ****INSTRUMENT_HOST_NAME = VIKING ORBITER 1, VIKING ORBITER 2;**
**  ****INSTRUMENT_HOST_TYPE = SPACECRAFT;**
**  ****INSTRUMENT_NAME = VISUAL IMAGING SUBSYSTEM - CAMERA A, VISUAL IMAGING SUBSYSTEM - CAMERA B;**
**  ****INSTRUMENT_TYPE = VIDICON CAMERA;**
**  ****NSSDC_DATA_SET_ID = 75-075A-01F, 75-083A-01C;**
**  ****VOLUME_ID = VO_2001, VO_2002, ...;**
**  ****VOLUME_LINK = ftp://pdsimage.wr.usgs.gov/cdroms/vo_2001/, ftp://pdsimage.wr.usgs.gov/cdroms/vo_2002/, ...**
**  ****KEYVALUES = CAMERA, DIGITAL, DIM, IMAGING, MARS, MODEL, ORBITER, PLANET, SPACECRAFT, SUBSYSTEM, TABLE, VIDICON, VIKING, VIS, VISUAL, VO1, VO2, VO_2001, VO_2002, ...;**
**  ****LABEL_REVISION_NOTE = 2000-09-19 C:CN S:CN LN:AY:DN;**
**END_OBJECT;**

## DIS Links to Mars Archives

***Mars Odyssey***

**Mars Global**
**Surveyor**

**Viking**

**Ancillary**
**Files**

**Distributed Archives**

## Lessons Learned

- Identified metadata as key to the solution
- Identifies and describes data and resources
- Provides search attributes
- Helps determine whether query can be resolved by resource
- However
- ODL is not a universal interchange language
- Resource heterogeneity is still visible
- Navigation and access is limited to http links
- System interoperability is not supported

## OODT, The Next Generation XML/JAVA/CORBA

- Started in 1998 as a research and development task funded at JPL by the Office of Space Science to address
- Application of Information Technology to Space Science
- Research methods for interoperability, knowledge management and knowledge discovery
- Develop software frameworks for data management to reuse software, manage risk, reduce cost and leverage IT experience
- OODT Initial focus
- Data archiving – Manage heterogeneous data products and resources in a distributed, metadata-driven environment
- Data location – Locate data products across multiple archives, catalogs and data systems
- Data retrieval – Retrieve diverse data products from distributed data sources and integrate

## OODT System Design Goals

- Encapsulate individual data systems to hide uniqueness
- Provide data system location independence
- Require that communication between distributed systems use metadata
- Define a standard data dictionary structure and approach for describing systems and resources
- Provide a scalable and extensible solution
- Provide a mechanism for data product exchange
- Allow systems using different data dictionaries and metadata implementations to be integrated
- Define an architecture that can leverage off of open standard approaches

## OODT Technology Focus

- Focus on building *middleware* components
- Focus on creating metadata “profiles” about data system resources
- Provide sufficient layers of abstraction in the architecture to isolate technology choices from architecture choices
- XML (Extensible Markup Language) for the data content
- CORBA (Common Object Request Broker Architecture) for the data transport
- Research technologies for implementing a distributed data architecture
- Distributed Object Computing (CORBA, DCOM, etc)
- Database Technology (RDBMS, ODBMS)
- Data Access Technologies (O/JDBC, XML, etc)
- Directory Implementations (LDAP)
- Data Interchange (XML)
- Communication Technologies (Web/HTTP, MOM, RPC, etc)

## Motivation for Middleware

- Middleware allows for the encapsulation of individual data systems
- Hide uniqueness by introducing the data architecture layer
- Allows for data abstraction
- Provide common client interfaces to heterogeneous systems
- Manage risk associated with technical decisions.   Systems evolve independent of the clients.
- Enable reuse and promote standards
- Allow for incompatible systems to be tied together by introducing a middleware layer

## Middleware Data Encapsulation

- Middleware can tie application, data, and user interfaces together and hide the unique interfaces

- Applications

- User Interface

- Data

- Middleware

## OODT Component Framework

- Java based software middleware component architecture that provides a software framework for archiving, search and retrieval, and data product exchange
- Archive Component
- Provides centralized data archiving and cataloging of data products
- Distributed
- A Search and Retrieval Component
- Manage metadata associated with resources
- Locate resources across geographically distributed data systems
- Distributed
- Data Product Exchange Component
- Support interchange (data sharing) of data products
- Support heterogeneous implementations and systems
- Distributed
- Query Service Component
- Ties search and product exchange services together
- Distributed

## Component Framework for OODT

- Archive
- Client

- OBJECT ORIENTED DATA TECHNOLOGY FRAMEWORK

## Data Archiving Goals

- Provide basic functions
- Transport and management of data sets and products
- Identification of products using metadata
- Event driven processing associated with data sets
- Ability to add, get and delete products from the archive
- Management of large, complex datasets and products
- Provide extensible data management approach
- Database is dynamically generated and extended based on metadata content
- Separate the metadata and the archive.  Metadata can be exported to other components of the OODT architecture.
- Provide a n-tier, client/server architecture for archiving at remote locations
- Build a service that is accessible via clients using common programming languages (Java, C++, etc)

## Archive Management

## OODT Metadata Development

- Develop methods for managing the semantics of data that are shared within and between domains
- Terminology Base – Domain specific name space
- Data Dictionary – Inventory of domain terms with definitions and other distinguishing attributes.
- Ontology – A set of concepts, their relationships and constraints, all within the scope of a domain.
- XML for metadata registry and communication
- Several I.T. efforts have shown the criticality of metadata in enabling data sharing and system interoperability
- Use standards where appropriate
- ISO/IEC 11179 – A framework for the Specification and Standardization of Data Elements
- Dublin Core – A metadata element set intended to facilitate discovery of electronic resources.

## Why XML?

- XML doesn’t provide a “silver bullet,” but it does allow us to refocus the problem on metadata
- Metadata is a key to interoperability (http://www.cio.gov/docs/metadata.htm)
- XML is language neutral
- Allows the designer to separate the data and the transport (re: CORBA vs XML-over-CORBA)
- Transport mechanism and data are not tied together
- Could be XML/HTTP
- Simpler deployments
- Simpler interfaces
- Allows technologies to grow and change independently
- Real value of XML is the process of describing the data

## What is a profile?

- A profile is a set of metadata resource definitions implemented in XML for data products residing in one or more distributed systems
- Profile servers are CORBA servers that manage XML profile definitions
- Profile servers communicate via XML-over-CORBA
- Developed Java classes that map XML profiles to a Java object

**Profile Distributed Node Architecture**

- XML/CORBA

- XML/CORBA

- XML/CORBA

- XML/CORBA

- XML/CORBA

- XML/CORBA

## Solutions to Data Search

- Build metadata “profiles*”* that describe data system resources
- Encapsulate individual data systems resources (Hide uniqueness)
- Communicate using metadata (Provide metadata with data)
- Enable interoperability based on metadata compatibility
- Refocus problem on metadata development
- Provide a core framework of software components to interconnect distributed data systems
- Define profiles using standard industry approaches
- Use XML to describe profiles
- ISO/IEC 11179 – A framework for the Specification and Standardization of Data Elements
- Dublin Core – A metadata element set intended to facilitate discovery of electronic resources.

## Resource Profile Classifications

***Resource Classes***

_**Metadata**_

_**Data**_

_**Application**_

_**System**_

***Resource Context***
***(Discipline***** )**

_**Medicine **_

_**Space Science **_

## Profile DTD

- <!ELEMENT profiles
-   (profile+)>
- <!ELEMENT profile
-   (profAttributes,
-    resAttributes,
-    profElement*)>
-     <!ELEMENT profAttributes
-       (profId, profVersion*, profTitle*, profDesc*, profType*,
-        profStatusId*, profSecurityType*, profParentId*, profChildId*,
-        profRegAuthority*, profRevisionNote*, profDataDictId*)>
-     <!ELEMENT resAttributes
-       (Identifier, Title*, Format*, Description*, Creator*, Subject*,
-        Publisher*, Contributor*, Date*, Type*, Source*,
-        Language*, Relation*, Coverage*, Rights*,
-        resContext*, resAggregation*, resClass*, resLocation*)>
-     <!ELEMENT profElement
-       (elemId*, elemName, elemDesc*, elemType*, elemUnit*,
-        elemEnumFlag*, (elemValue | (elemMinValue, elemMaxValue))*,
-        elemSynonym*,
-        elemObligation*, elemMaxOccurrence*, elemComment*)>

## XML Profile Example (1 of 2)

- <profile>
-   <profAttributes>
-     <profId>OODT_PDS_DATA_SET_INV_82</profId>
- <profDataDictId>OODT_PDS_DATA_SET_DD_V1.0</profDataDictId>
-   </profAttributes>
-   <resAttributes>
-     <Identifier>VO1/VO2-M-VIS-5-DIM-V1.0</Identifier>
-     <Title>VO1/VO2 MARS VISUAL IMAGING SUBSYSTEM DIGITAL ...</Title>
-     <Format>text/html</Format>
-     <Language>en</Language>
-     <resContext>PDS</resContext>
-     <resAggregation>dataSet</resAggregation>
-     <resClass>data.dataSet</resClass>
-   <resLocation>http://pds.jpl.nasa.gov/cgi-bin/pdsserv.pl?...</resLocation>
-   </resAttributes>

## XML Profile Example (2 of 2)

- <profElement>
-       <elemId>ARCHIVE_STATUS</elemId>
-       <elemName>ARCHIVE_STATUS</elemName>
-       <elemType>ENUMERATION</elemType>
-       <elemEnumFlag>T</elemEnumFlag>
-       <elemValue>ARCHIVED</elemValue>
-     </profElement>
-     <profElement>
-       <elemId>TARGET_NAME</elemId>
-       <elemName>TARGET_NAME</elemName>
-       <elemType>ENUMERATION</elemType>
-       <elemEnumFlag>T</elemEnumFlag>
-       <elemValue>MARS</elemValue>
-     </profElement>
- </profile>

## OODT Profile Service Component

- Profiles are managed by profile servers
- Profile servers are written in Java
- OODT currently has three different registry methods for managing profiles which are configurable at run time
- Flat File
- RDBMS via JDBC (Oracle)
- LDAP (OpenLDAP)

## Data Product Exchange

- Exchanging products requires access to each data system (RDBMS/OODBMS, Flat file, etc) which is difficult
- Different vendor products
- Non-standard interfaces
- Different implementations (data model, home grown, COTS,etc)
- Representations of data are different
- Heterogeneous Platforms
- Heterogeneous O/S
- etc

## Solutions to Data Product Exchange

- Extend framework to support common access to distributed data systems by creating a “Product Service Component”
- Product Servers  - Middleware that negotiates the interfaces between the data system implementations
- Design the component to leverage off of
- Consistent metadata and data dictionary
- Consistent data interchange methods and protocols
- Provide data abstraction
- Data and information hiding
- Location hiding and independence
- Provide a standard language for communication
- Use the OODT XML Query language for data interchange
- Support “rich” query description including data elements and constraints
- Support “rich” query results that include results in many different formats

## OODT Product Server Component 

- The Product Server plugs into the OODT framework and manages the “handshake” between the data system and the OODT system.
- Extensible by dynamically loading objects at runtime which are specific to the data system model
- Queries and results are passed using an OODT XML Query structure
- Encapsulates one or more data sources for standardized access

**Product Server**

- Query: XML

- Result: XML

- Generic Server

- Implementation Class

- File
- Sys

- Database

## OODT Query Service Component

- Manages all queries for the identification and retrieval of data products
- All components are identified by a unique name and managed in a CORBA name server
- Queries to multiple profile or product servers occur concurrently
- Queries are described using the OODT XML Query structure
- Ties together the profile and product server components for the OODT framework

## XML Query Example (1 of 2)

- <query>
-  <queryAttributes>
-   <queryId>OODT_XML_QUERY_V0.1</queryId>
-   <queryTitle>OODT_XML_QUERY - PDS DIS Query Example</queryTitle>
-   <queryDesc>PDS DIS Query for TARGET_NAME = MARS</queryDesc>
-   <queryType>QUERY</queryType>
-   <queryStatusId>ACTIVE</queryStatusId>
-   <querySecurityType>UNKNOWN</querySecurityType>
-   <queryRevisionNote>2000-05-12 JSH V1.2 Updated for new
-                                                                                   prof.dtd</queryRevisionNote>
-   <queryDataDictId>OODT_PDS_DATA_SET_DD_V1.0</queryDataDictId>
-  </queryAttributes>
-  <queryResultModeId>ATTRIBUTE</queryResultModeId>
-  <queryPropogationType>BROADCAST</queryPropogationType>
-  <queryPropogationLevels>N/A</queryPropogationLevels>
-  <queryMaxResults>100</queryMaxResults><queryResults>0</queryResults>
-  <queryKWQString>TARGET_NAME = MARS</queryKWQString>

## XML Query Example (2 of 2)

- <querySelectSet></querySelectSet>
-  <queryFromSet></queryFromSet>
-  <queryWhereSet>
-   <queryElement>
-    <tokenRole>elemName</tokenRole>
-    <tokenValue>TARGET_NAME</tokenValue>
-   </queryElement>
-   <queryElement>
-    <tokenRole>LITERAL</tokenRole>
-    <tokenValue>MARS</tokenValue>
-   </queryElement>
-   <queryElement>
-    <tokenRole>RELOP</tokenRole>
-    <tokenValue>EQ</tokenValue>
-   </queryElement>
-  </queryWhereSet>
-  <queryResultSet></queryResultSet>
- </query>

## Insertion of OODT into PDS

**OODT Middleware**

**Distributed Archives**

## OODT Query Flow Example

- Query Server

- Profile Serverjpl

- Product Serverjpl.pti

- Product Serverjpl.pds

- Product Serverjpl.pds.mola

- Userquery

- XSL(profiles ordata productsformatted)

- XMLQuery/IIOP(no results)

- XMLQuery/IIOP(profiles ordata resultsas requested)

- XMLQuery/IIOP(no results)

- XMLQuery/IIOP(profiles of resources to handle query)

- XMLQuery/IIOP (data results)

- XMLQuery/IIOP (product search)

**Search Web Page**

**Profile DB**

**PTI Repository**

**PDS DVD Jukebox**

**PDS MOLA Oracle DB**

- QueryClient

- Web server

- search.jsp

## MGS Coverage Plot

## Query Results

***Query Results***

## Efforts in Bioinformatics

- September 2000, the Office of Science Policy at NIH and JPL entered into a joint inter-agency agreement to
- Explore knowledge environments to support the management and exchange of biomarkers and biomakers databases
- Insert technologies developed to support interoperability between biomedical databases
- Develop methodologies to are mutually beneficial to both biomedical and space science data systems
- A series of meetings lead to an initial effort that focused on interoperability within the Early Detection Research Network, a network within the National Cancer Institute (NCI) consisting of 18 laboratories and hospitals
- Two initial epidemiological centers were selected for locating cancer specimens
- Moffitt Cancer Research Center
- University of San Antonio, Texas
- A “mock up” data architecture for EDRN was presented on January 21st at the 3rd EDRN Steering Committee meeting in San Antonio, Texas

## EDRN Mock Query Interface

## EDRN Knowledge ArchitectureMockup Implementation at JPL

***EDRN Knowledge Architecture******Mockup Implementation at JPL***

**EDRN Mock Databases **
**Hosted at JPL **

- In:Query
- Out::Identified Resources

- In:Query
- Out::Data Products

- In:Query
- Out::Data Products

**OODT**
**Middleware: **
**Hosted at JPL**

## Component Based Mars Distributed Archive Architecture

**Component Based Mars Distributed Archive Architecture**
** ****2001 Mars Odyssey Orbiter Example**

**MARIE EDR,**
**RDR Archive**
**PDS PPI**

**GRS RDR**
**Archive**
**PDS GEO**

**THEMIS EDR,**
**RDR Archive**
**ASU**

**Documents and**
**Ancillary Files**
**PDS CN**

**GRS EDR**
**Archive**
**UofA**

- In:Query
- Out::Identified Resources

- In:Query
- Out::Data Products

- Data Products

- Virtual Volume

**OODT**
**Middleware**

- Virtual Volume

- Physical Volume

- Data Product
- and Resource
- Descriptions

- DVD
- Volume

## More Information and References

- OODT Papers (http://oodt.jpl.nasa.gov/doc/papers)
- “Science Search and Retrieval using XML” by OODT Team.  Presented at Second National Conference on Scientific and Technical Data, National Academy of Sciences,  Washington D.C. March 2000.
- “A Distributed Component Framework for Science Data Product Interoperability” by OODT Team.  Presented at the 17th Annual International CODATA conference.  Baveno, Italy.  October 2000.
- OODT Presentations (_[http://oodt.jpl.nasa.gov/doc/presentations](http://oodt.jpl.nasa.gov/EDRN/search.jsp)_)
- JPL Enterprise Data Architecture White Paper (e-mail: Dan.Crichton@jpl.nasa.gov)
- Planetary Data System
- http://pds.jpl.nasa.gov
- Dublin Core
- http://purl.oclc.org/dc
- Extensible Markup Language
-  http://www.w3c.org/XML
- ISO/IEC 11179: Specification and Standardization of Data Elements
- Federal CIO Statement on Metadata
- http://www.cio.gov/docs/metadata.htm

## Backup Slides

## OODT Prototype Environment

- - XML parser: Apache/IBM Xerces 1.0.3
- _[http://](http://xml.apache.org/)__[xml](http://xml.apache.org/)__[.apache.org](http://xml.apache.org/)_
- - XSLT: Apache Xalan 1.0.0
- _[http://](http://xml.apache.org/)__[xml](http://xml.apache.org/)__[.apache.org](http://xml.apache.org/)_
- - CORBA: Orbacus 4.0.3
- _[http://www.](http://www.ooc.com/)__[ooc](http://www.ooc.com/)__[.com](http://www.ooc.com/)_
- - Database: Oracle 8.1.5
- _[http://www.oracle.com](http://www.oracle.com/)_
- - LDAP server: OpenLDAP 1.2.11
- _[http://www.](http://www.openldap.org/)__[openldap](http://www.openldap.org/)__[.org](http://www.openldap.org/)_
- - Development language: Java 1.2
- _[http://java.sun.com](http://java.sun.com/)_
- - Web server: iPlanet Fasttrack 4.1
- _[http://www.](http://www.iplanet.com/)__[iplanet](http://www.iplanet.com/)__[.com](http://www.iplanet.com/)_
- - Server operating system: RedHat Linux 6.2
- _[http://www.](http://www.redhat.com/)__[redhat](http://www.redhat.com/)__[.com](http://www.redhat.com/)_
- - Version control system: CVS 1.10.5
- _[http://www.](http://www.redhat.com/)__[cvshome](http://www.redhat.com/)__[.org](http://www.redhat.com/)_

## Profile Server Node

## The Use of Metadata in the PDS

***The Use of Metadata in the PDS***

**Mission**

***Data Set Collection***

***Data Set***

***Instrument***

**Spacecraft**

***Target***

**Document**

***Image***

***Time Series***

***Spectrum***

**Planetary **
**Science**
**Model**

_**Locate and Use Data**_
- - Use context to find data
- - Use context to understand
-   data

_**System Interoperability**_
- - Use context to share data

_**Correlative Science**_
- - Use context to find new
-   relationships between
-   data

***Model***
***Attributes***

**Label**

**Data**