# Presentation: 328287

## A Multi-Discipline Metadata Registry for Science Interoperability

- J. Steven Hughes/JPL - steve.hughes@jpl.nasa.gov
- Daniel J. Crichton/JPL - daniel.crichton@jpl.nasa.gov
- Jason J. Hyon/JPL - jason.hyon@jpl.nasa.gov
- Sean C. Kelly/UTA - sean.kelly@jpl.nasa.gov

**Open Forum on Metadata Registries**
**January 17-21, 2000**

## A Multi-Discipline Metadata Registry for Science Interoperability

**Problem Statement**
**System Overview**
**Profile Development**
**Conclusion and Outstanding Issues**

## Problem Statement

**Space scientists can not easily locate or use data across the hundreds if not thousands of autonomous, heterogeneous, and distributed data systems currently in the Space Science community.**

**Heterogeneous Systems**
**Data Management - RDBMS, ODBMS, HomeGrownDBMS, BinaryFiles**
**Platforms - UNIX, LINUX, WIN3.x/9x/NT, Mac, VMS, ...**
**Interfaces - Web, Windows, Command Line**
**Data Formats - HDF, CDF, NetCDF, PDS, FITS, VICR, ASCII, ...**
**Data Volume - KiloBytes to TeraBytes**
**Heterogeneous Disciplines**
**Moving targets and stationary targets**
**Multiple coordinate systems**
**Multiple data object types (images, cubes,  time series,  spectrum, tables, **
**binary, document)**
**Multiple interpretations of single object types**
**Multiple software solutions to same problem.**
**Incompatible and/or missing metadata**

## Problem Statement Current Capabilities - Example Systems

- Astrobrowse - Web-based CGI-Script
- Distributed searches for ***products*** across hundreds of Astrophysics sites
- Search limited to target identifiers and RA/DEC specified areas
- Limited to stationary (identified) targets
- Not readily extensible to other coordinate systems or moving targets
- PDSBrowse - Web-based CGI-Scripts
- Searches for ***data sets and resources*** across eight distributed nodes
- Search allowed on any defined attribute
- Limited to identified targets
***Product*** searches relegated identified resources (catalogs)

## Problem Statement Current Capabilities - Example Queries

**Current**
**Planetary - PDSBrowse - Eight nodes**
- Constrain on “image” and read description of  the 15 resulting resources.
- Link to selected resources and perform search for image.
**Astrophysics - ASTROBrowse - Hundreds of nodes**
- Submit query for target = Jupiter
- “Jupiter” might not be associated with all images containing Jupiter.
- Object name might have aliases
- Convert images to usable format.
**Requested**
**OmniBrowse**
- Find science_products where data_object_type = image and target = “JUPITER”
- Find science_products  where data_object_type = image and target = all_aliases (“JUPITER”) and convert_to(JPEG)

## Proposed Solution 

**Encapsulate individual data systems. (Hide uniqueness.)**
**Communicate using metadata. (Provide metadata with data.)**
_**Enable interoperability based on metadata compatibility.**_
**Refocus problem on metadata development.**

## Proposed Solution (cont) 

**Domain independent data structures**
- XML - Standard interchange language
- Metadata management
- Message passing
**Domain independent system infrastructure**
- CORBA for interoperability between computer systems and languages
- Message passing to simply interface design
- Standardized reusable server components
**Resource descriptions (Metadata)**
- Resource profiles
- Domain data dictionaries
- Domain data models
**Object_Oriented Data Technology Task (OODT)**
- Domain independent data management infrastructure

## System OverviewObject Oriented Data Technology Framework

- PDS Systems

## System OverviewProfile Service

- Describes a scientific data system via XML
- Available datasets and metadata
- Types of resources and where they’re located
- Optionally describes other profile servers

- Data system 1

- Data system 2

## System OverviewQuery Service

- Knows how to “crawl” through servers to produce a result
- Crawls through profiles to discover other profiles and product servers
- Crawls through product servers to display available products
- Accessible through CORBA API or through web browser

## System Overview

**System Overview**
**Object Oriented Data Technology Framework**

## Profile Development Objective

**Design and develop an architecture that will manage the meta-data necessary for identifying and locating science data resources across distributed heterogeneous data systems.**

## Profile Development Approach

- Choose a common interchange format.
- Develop a domain generic language.
- Implement domain specific instances.
*Model the domain.*
- Capture the meta-data.
*Develop system to manage the results.*

## Profile Development Choose a common interchange format

**XML**
- eXtensible Markup Language
- More expressive than HTML
- More simple than SGML
- A meta-language used to define domain languages.
- XSIL - eXtensible Scientific Interchange Language.
- XIL - Instrument control language.
- Wide acceptance as an interchange format.
- Electronic data interchange (EDI) standard.
- Space sciences data systems

## Profile Development Develop a domain generic language

**Define a generic structure (XML DTD) that can describe**
**  ****heterogeneous domain-specific resources.**
- Profile - A resource description with sufficient information to
-  determine if the resource satisfies a query**.**
- Profile elements
- name, syntax, unit, instance, meaning, alias, ...
- encodes all domain attributes and their values specific to this resource
- Resource attributes - id, title, discipline, location_id, ...
- Profile attributes - id, title, desc, type, data_dictionary_id, ...

## Profile Development Develop a domain generic language prof.dtd

- <!ELEMENT PROFILES
-   (PROFILE+)>
- <!ELEMENT PROFILE
-   (PROFILE_ATTRIBUTES,
-     RESOURCE)>
-   <!ATTLIST PROFILE
-    PROFILE_ID CDATA #REQUIRED >
-   <!ELEMENT PROFILE_ATTRIBUTES
-     (ID, TITLE*, DESC*, TYPE*,
-      STATUS_ID*, SECURITY_TYPE*,
-      PARENT_ID*, CHILD_ID*,
-      REVISION_NOTE*,
-      DATA_DICTIONARY_ID*)>

- <!ELEMENT RESOURCE
-     (RESOURCE_ATTRIBUTES,
-      PROFILE_ELEMENT*)>
- <!ELEMENT RESOURCE_ATTRIBUTES
-       (RESOURCE_ID,
-        RESOURCE_TITLE,
-        RESOURCE_DISCIPLINE,
-        RESOURCE_AGGREGATION,
-        RESOURCE_CLASS,
-        RESOURCE_LOCATION_ID,
-        RESULT_MIME_TYPE)>
-     <!ELEMENT PROFILE_ELEMENT
-       (ELEMENT_NAME, ELEMENT_MEANING*, ELEMENT_ALIAS*,
-        VALUE_SYNTAX*, VALUE_UNIT*,
-        (VALUE_INSTANCE | (MINIMUM_VALUE, MAXIMUM_VALUE))*)>

## Profile Development Develop a domain generic language

***Specialize***** the profile class**
- Profile - One profile to one resource (e.g. catalog)
- Inventory - One profile to many resources (e.g. catalog entries)
- Minimized profile element
- Dictionary - One profile to one discipline
- Comprehensive profile elements
- aliases
- meanings

## Profile Development Develop a domain generic language

**Profile element hierarchy** (given a domain data dictionary)
- Dictionary
- keywords - all keywords in domain
- keyword values - union of all values in domain
- e.g. TARGET_NAME = {ADRASTEA, ..., VENUS}
- Profile
- keywords - all keywords associated with a domain resource
- keyword values - union of all values in domain resource
- e.g. TARGET_NAME = {MARS, DEIMOS, PHOBOS}
- Inventory
- keywords - keywords associated with inventory item.
- Keyword values - values for inventory item.
- E.g. TARGET_NAME = {MARS}

## Profile Development Implement domain specific instances

**Apply domain generic language to specific domain.**
- E.g. Space/Earth Science data and other resources.*** ***
***Model the domain***** **
- Planetary Science Data Dictionary
- Planetary Data System Data Model
- Entity-Relationship model
**Capture the meta-data**
- Extracted from PDS metadata repository

## Profile Development Implement domain specific instances

***Model the domain (from the start)***** **
- Planetary Science Data Dictionary
- Over 1000 Data Elements spanning Planetary Science
- Nomenclature Standard
- Meaning, type, ranges, enumerated values
- Planetary Science Data Model
- Developed as Planetary Science enterprise E/R model
- Planetary Science Entities - Spacecraft, Instruments
- Science Data Entities - Data Products
- Data Organization Entities - Volumes
- Management Entities - Nodes, Personnel
- Implemented as Data Set Catalog in an RDBMS

## Planetary Science Standards Architecture

**Planetary Science Standards Architecture**

- Metadata and data standards
- All data products are labeled
- Data Model relates entities

## Profile Development Implement domain specific instances  Profile Example - PDS Distributed Inventory System

- <PROFILE PROFILE_ID = "PROFILE_PDS_DIS_V1.3.n" >
-   <PROFILE_ATTRIBUTES>
-     <ID> PROFILE_PDS_DIS_V1.3.n </ID>
-     <TITLE> Planetary Data System - Distributed Inventory System - Profile V1.0 </TITLE>
-     <DESC> This profile describes the Planetary Data System (PDS) Distributed Inventory System (DIS)  ...
-     <TYPE> PROFILE </TYPE>
-     <DATA_DICTIONARY_ID> OODT_PDS_DATA_SET_DD_V1.0 </DATA_DICTIONARY_ID>
-   </PROFILE_ATTRIBUTES>
-  <RESOURCE>
-     <RESOURCE_ATTRIBUTES>
-       <RESOURCE_ID> PDS_DIS_V1.3.n </RESOURCE_ID>
-       <RESOURCE_TITLE> Planetary Data System - Distributed Inventory System </RESOURCE_TITLE>
-       <RESOURCE_DISCIPLINE> PDS </RESOURCE_DISCIPLINE>
-       <RESOURCE_AGGREGATION> GRANULE+ </RESOURCE_AGGREGATION>
-       <RESOURCE_CLASS> INVENTORY </RESOURCE_CLASS>
-       <RESOURCE_LOCATION_ID> http://pds.jpl.nasa.gov/pdsbrows.htm </RESOURCE_LOCATION_ID>
-       <RESULT_MIME_TYPE> text/html </RESULT_MIME_TYPE>
-     </RESOURCE_ATTRIBUTES>
-     ...

## Profile Development Implement domain specific instances  Profile Example (cont) - PDS Distributed Inventory System

-     ...
- <PROFILE_ELEMENT>
-       <ELEMENT_NAME> DATA_OBJECT_TYPE </ELEMENT_NAME>
-       <ELEMENT_MEANING> The data_object_type element provides the type ...
-       <VALUE_SYNTAX> ENUMERATION </VALUE_SYNTAX>
-       <VALUE_UNIT> N/A </VALUE_UNIT>
-       <VALUE_INSTANCE> IMAGE </VALUE_INSTANCE>
-     </PROFILE_ELEMENT>
-     <PROFILE_ELEMENT>
-       <ELEMENT_NAME> DATA_SET_NAME </ELEMENT_NAME>
-       <ELEMENT_MEANING> The data_set_name element identifies a PDS data set.  -- example ...
-       <VALUE_SYNTAX> ENUMERATION </VALUE_SYNTAX>
-       <VALUE_UNIT> N/A </VALUE_UNIT>
-       <VALUE_INSTANCE> VO1/VO2 MARS VISUAL IMAGING SUBSYSTEM DIGITAL ...
-       <VALUE_INSTANCE> VO2 MARS RADIO SCIENCE SUBSYSTEM RESAMPLED LOS ...
-     </PROFILE_ELEMENT>
-     <PROFILE_ELEMENT>
-       <ELEMENT_NAME> TARGET_NAME </ELEMENT_NAME>
-       <ELEMENT_MEANING> The target_name element provides the names of the targets ...
-       <ELEMENT_ALIAS> ADS.OBJECT_ID </ELEMENT_ALIAS>
-       <VALUE_SYNTAX> ENUMERATION </VALUE_SYNTAX>
-       <VALUE_UNIT> N/A </VALUE_UNIT>
-       <VALUE_INSTANCE> IDA </VALUE_INSTANCE>
-       <VALUE_INSTANCE> JUPITER </VALUE_INSTANCE>
-     </PROFILE_ELEMENT>
-    </RESOURCE>

## Profile Development Implement domain specific instances  Inventory Example - PDS Data Set

-  <RESOURCE>
-     <RESOURCE_ATTRIBUTES>
-       <RESOURCE_ID> VO1/VO2-M-VIS-5-DIM-V1.0 </RESOURCE_ID>
-       <RESOURCE_TITLE> VO1/VO2 MARS VISUAL IMAGING SUBSYSTEM DIGITAL IMAGING MODEL ...
-       <RESOURCE_DISCIPLINE> PDS </RESOURCE_DISCIPLINE>
-       <RESOURCE_AGGREGATION> GRANULE+ </RESOURCE_AGGREGATION>
-       <RESOURCE_CLASS> DATA </RESOURCE_CLASS>
-       <RESOURCE_LOCATION_ID> http://pds.jpl.nasa.gov/cgi-bin/pdsserv.pl?OBJECT_ID=PDS100676 ...
-       <RESULT_MIME_TYPE> text/html </RESULT_MIME_TYPE>
-     </RESOURCE_ATTRIBUTES>
-     <PROFILE_ELEMENT>
-       <ELEMENT_NAME> DATA_SET_NAME </ELEMENT_NAME>
-         <VALUE_INSTANCE> VO1/VO2 MARS VISUAL IMAGING SUBSYSTEM DIGITAL IMAGING MODEL ...
-     </PROFILE_ELEMENT>
-     <PROFILE_ELEMENT>
-       <ELEMENT_NAME> DATA_OBJECT_TYPE </ELEMENT_NAME>
-         <VALUE_INSTANCE> TABLE </VALUE_INSTANCE>
-     </PROFILE_ELEMENT>
-     <PROFILE_ELEMENT>
-       <ELEMENT_NAME> TARGET_NAME </ELEMENT_NAME>
-         <VALUE_INSTANCE> MARS </VALUE_INSTANCE>
-     </PROFILE_ELEMENT>
-     <PROFILE_ELEMENT>
-       <ELEMENT_NAME> VOLUME_ID </ELEMENT_NAME>
-         <VALUE_INSTANCE> VO_2001 </VALUE_INSTANCE>
-         ...
-         <VALUE_INSTANCE> VO_2014 </VALUE_INSTANCE>
-     </PROFILE_ELEMENT>
-   </RESOURCE>

## Conclusion Profile Development - Review

- Choose a common interchange format. (XML)
- Develop a domain generic language. (X2PL)
- (XML eXtensible Profile Language)
- Implement domain specific instances. (Science Resource Profiles)
*Develop system to manage the results. (Profile Servers)*

## Conclusion Outstanding Issues

- Metadata development remains the problem
- Data Dictionary interoperability is required
- NASA Data Entity Dictionary Specification Language (DEDSL) semantics being used to promote compatibility. (Subset of 11179-3)
- Need XML specification
- Data Model interoperability is required

## Backup Slides

## System OverviewProduct Service

- Queries for scientific datasets from existing data system and formats results in OODT standard format
- Configurable with “device drivers” that know how to access special data systems

**Product Server**

- Query

- Result

## System OverviewWeb Interface

- Standard Apache web server with Java servlet engine and JSP processor
- User retrieves products with any old web browser—no client-side Java necessary

- Web browser

- GET http://oodt/search.jsp

- Execute

- Query

- HTML Response

## System OverviewServer-side Java Components

- Servlets
- Replaces CGI scripts and programs
- Fast ...*very* fast
- JSP
- Mix Java source text and HTML in same file
- Java Beans
- Dynamic components to model business
- Run-time managed by application server

## System OverviewServer-side Java Components

- CORBA
- Standard interobject communication facility
- Provides remote method invocation features
- XML
- Self-describing data interchange format
- Has many applications:
- Serialization of objects, bridge to CORBA interfaces, data/metadata packaging, direct-to-user presentation with XSL, and more