# Presentation: 383288

## How Information

**Is Stored**

- PDS NASA-HQ Review

- PDS Systems Engineering
- September 8, 2004

*http://pds.jpl.nasa.gov/*

## Purpose and Topics

**Central Node Catalog**
- Data Model and Catalog Implementation
- Catalog Utilization
**Data Holdings at the Nodes**
**Looking Forward**

- Purpose: Describe how data and metadata is stored and deployed within the PDS archive

## PDS Pipeline by Project Phase

***OPERATIONS***

***PLANNING/DESIGN***

- Phases A-D				Phase E-F

**Trade Studies*****1***

**ICD / SIS’s per Instrument**

**Data Mgmt/Archive Plan**

**Validation*****2***

**Ingest Data (Receive/Store/Catalog)**

**Distribution/Access**

**Track**

**Archive/Transfer*****1***

**Instrument Team Data Production**

**Delivery to PDS**

**Science Users**

***1, 2****** - Areas of Risk to Schedule/Resources***

_**Core System:**_**  Within planned budget/staffing, with the exception of need for (1) Storage Technology ****	****  Study to determine MRO-era hardcopy archive media and (2) changes in Validation area**
_**Extended System:**_**  Requirements/planning to be done on deferred LOE basis**

**Development*****2***

**Pre-Brief / MOU**

## Central Node Catalog

- Requirements Summary
- The Central Node Catalog ensures that a resource description is cataloged for every PDS resource.
- For every resource stored online, a pointer is provided.
- Any resource can be located using its cataloged attributes.

- Definition – A resource is anything that can be described by a set of attributes. Resources in the PDS include data products, data sets, software, documents, web services, spacecraft, instruments, and personnel. Note that “resource” is synonymous with “entity”.

## PDS Data Model

- The PDS developed the Planetary Science Data Model in the late 1980’s
- Planetary science entities were included with all important relationships.
- Example: Spacecraft carry instruments
- The data model is used as the context within which all data in the archive is classified
- Example: The Viking Image data set is associated with:
- Mission: Viking
- Spacecraft: Viking Orbiter 1 and Viking Orbiter 2
- Instruments: Visual Imaging Subsystem Camera A x 2
-                          Visual Imaging Subsystem Camera B x 2
- Targets: Mars, Deimos, Phobos, Dark Sky, etc.
- Curating Node: Imaging
- Volumes: VO_1xxx, ...
- The model was formally developed and validated using standard data modeling methodologies.
- The model was extended by defining a generic resource that provides associations with resources not defined in the original data model

## Catalog Implementation

- The PDS data model was implemented in a relational DBMS and made operational in 1990.
- The catalog database is currently running under Sybase 12.x. on an Intel, Dual Processor 3.06Ghz, 2GB Memory, 36.4GB Disk running LINUX Redhat EL 3.0
- An evolved version of the original data model remains the key standard for collecting and organizing archive data and metadata
- ~50 defined objects (modeled entities)
- ~1,200 data elements (attributes)
- 1,000s of parameters
- Maintained in the Planetary Science Data Dictionary
- Data model evolves to reflect changing domain

## Catalog Loading

- Catalog templates are used to collect metadata for ingestion into the database.
- Administrative interfaces for the online update of catalog information are to be developed in 2005.

## Catalog Database Utilization

- Support for Data Set View Interface
- Allows data sets to be found based on relationships with other entities in the PDS data model. (e.g. mission, target, etc)
- Search constraints in pull downs are retrieved from catalog
- User selections in one pull down forces others to be recalculated
- The catalog database will also support views for all other entities such as target, instrument, etc
**	**

## Catalog Database Utilization

- Support for Data Set View Results Interface
- All data sets that match the search constraints are listed
- All external web resources related to each data set are listed
- New resources can be added as needed with minimal impact to system
- Links to summary data set information is provided.
**	**

## Catalog Database Utilization

- Support for Data Set View Summary Report
- Data set citation, terse description, and abstract are displayed
- Links to all related entities are provided (e.g. targets)
- Links to detailed data set information is provided.

## Catalog Database Utilization

- Support for Data Set View Detailed Report
- Provides a complete listing of all data set information in the catalog
- Links are provide to related information. E.g. Mars would link to target information.

## External Access to Catalog Database

- APIs for Catalog Database
- HTTP queries
- Returns XML documents (e.g. profiles)
- Returning JSP reformatted results
- JAVA interface to servers (JAVA Doc)
- Support for ad-hoc Interfaces
- APIs allow interfaces to be created by anyone, anywhere

## Miscellaneous

- All  resource information and resource links can be “spidered” for  “Google” like searches.

## External Users

- ADS (Astrophysics Data System)
- Data set and data product abstracts have been ingested into the ADS with links back to the catalog
- GeoVirgil – Science Educators Interface Task
- Uses HTTP interfaces for data set and resource information
- Discipline Node Development
- DN applications access catalog for data set and other resource information

## Central Node Catalog

- Current Population
- Data Sets : 983
- Targets : 1,203
- Missions : 42 (1)
- Spacecraft  : 90
- Instruments : 337
- Documents : 2,736
- Volumes : 3,927 (2)
- Discipline Nodes : 9
- Personnel : 949
- Resources : 2,177 (3)

- 1) Missions – Includes flight projects, campaigns, and experiments
- 2) Volumes – migrating from physical to virtual volume organization
- 3) Resources – Generic resources only

- Scalability
- MRO ERA and Post MRO ERA capacities can easily be met easily by existing database technologies
- Greatest increase is expected in number of targets (->300,000 Asteroids/Comets)
- Evolution
- Changes in model are coordinated with Disciplines Nodes
- Example: Expanded target model.

## Data Holdings

- Requirements Summary
- Every data product has a description.
- Includes catalog attributes
- Every data product can be located using its catalog attributes.
- Every data product is stored on a mountable computer storage media.
- Every data product can be retrieved from its on-line storage media.
- Every data product must be archived for future use
- “It is imperative that the metadata remain attached to the data or the data become meaningless and unusable.”
-     [French, J.C., et. al., 1990, "A Summary of the NSF Scientific Database Workshop"].

## Design – Product Descriptions (Metadata)
Data providers work with PDS data engineers to design data products and their descriptions (labels).
Data product descriptions are designed within the context of the data model
Identification Information – e.g. Image_id
Relational Information – e.g. Instrument and Target
Descriptive Information
Structural – e.g. Image Lines, Line Samples, and Sample Type
Context – e.g. Instrument Modes, Camera Models, Filter Names
Data product descriptions evolve as community needs evolve
Product descriptions are expanded

- Data providers work with PDS data engineers to design data products and their descriptions (labels).
- Data product descriptions are designed within the context of the data model
- Identification Information – e.g. Image_id
- Relational Information – e.g. Instrument and Target
- Descriptive Information
- Structural – e.g. Image Lines, Line Samples, and Sample Type
- Context – e.g. Instrument Modes, Camera Models, Filter Names
- Data product descriptions evolve as community needs evolve
- Product descriptions are expanded

**Data Holdings**

- DATA_SET_ID	= "VO1/VO2-M-VIS-5-DIM-V1.0"
- SPACECRAFT_NAME	= {VIKING_ORBITER_1, ...
- TARGET_NAME	= MARS
- IMAGE_ID		= MG88S045
- ^IMAGE		= 2
- SOURCE_IMAGE_ID	= {"383B23", "421B23", ...
- INSTRUMENT_NAME	= {VISUAL_IMAGING_SUBSYS...
- NOTE                	= "MARS DIGITAL IMAGE ...
- OBJECT		= IMAGE
-   LINES		= 160
-   LINE_SAMPLES	= 252
-   SAMPLE_TYPE	= UNSIGNED_INTEGER
-   SAMPLE_BITS	= 8
-   SAMPLE_BIT_MASK 	= 2#11111111#
-   CHECKSUM	=  2636242
- END_OBJECT

## Design – Archive Volumes
Volume standards are used to organize volume contents
Directories are defined for data, documents, catalog data, index files, etc
Volume Index file contains an inventory of data products on the volume together with product catalog attributes
Volume standards were originally developed for CD-ROM volumes but are now considered logical structures of variable size for organizing data on mass storage.
For online access, archive volumes are copied to standard data repositories.
Any node can implement one or more standard data repositories

- Volume standards are used to organize volume contents
- Directories are defined for data, documents, catalog data, index files, etc
- Volume Index file contains an inventory of data products on the volume together with product catalog attributes
- Volume standards were originally developed for CD-ROM volumes but are now considered logical structures of variable size for organizing data on mass storage.
- For online access, archive volumes are copied to standard data repositories.
- Any node can implement one or more standard data repositories

**Data Holdings**

## Data Holdings

- Design – Data Sets and Data Product Catalogs
- A Data Set is a collection of related Data Products
- E.g. Viking Image Data Set consists of ~49,000 images from Viking Orbiters 1&2
- ~1,000 data sets currently exist in the archive
- Data Products are classified by Data Product Types
- Viking images have ~40 attributes
- Galileo images have ~200 attributes
- ~4500 product types currently exist in the archive
- Each Data Product Type typically requires one Data Product Catalog
- Data Product Catalogs for frequently used data sets are created at the DNs
- Data Sets span one or more optical-media volumes
- One virtual volume if created on mass-storage media

## Data Archive

- Design – Physical Archive
- The PDS is required to archive the data
- Three copies of the archive exist
- Archive is partitioned by science discipline and deployed at discipline nodes
- Archive is replicated at the Central Node
- Archive is sent to NSSDC for long-term storage

## Data Archive

- Planetary Science Archive

**Discipline Nodes**

**CN**
**Replica**

**NSSDC**

- Three Copies

- Partitioned and distributed by Science Discipline

- Online Replicated Copy of Archive

- Deep Archive for Long-Term Storage

- Only data with archive status of “ARCHIVED” is sent to NSSDC
- Other archives have data with mixed statuses

## Data Archive

- Current PDS policy is for the mission to create 3 copies on hard media for transfer to the PDS; one copy is forwarded to the NSSDC
- Currently DVD-5 (4.7 GB) is the largest storage medium that has been certified by the PDS as a physical archive media
- This is not practical for future missions or the PDS
- Management of the physical media is the overwhelming factor. I.e. Copying, Storing, Shipping
- Media purchase and shipping costs are significant
- Study in progress to resolve this issue
- Focusing on electronic transfer of archive volumes to NSSDC
- NSSDC takes responsibility for storage and media migration to ensure long-term viability of the bits.
- PDS retains responsibility for archive volume creation and on-line distribution from mass storage media /w backup

## Distributed Data Repositories and Catalogs

- Online archives and catalog services
- Atmospheres
- Geosciences
- Imaging
- NAIF
- PPI
- RINGS
- SBN
- ASU (THEMIS data node)
- Replicated archive
- CN

## Data Holdings5

- Current
- CN
- Max Products/Cat : 2,000K1
- Products Total : 10,000K2
- Data Volume : 10TB
- DN3
- Max Products/Cat : 2,000K1
- Products Tot : 5,000K2
- Data Volume : 5TB

-  MRO Era
- CN
- Max Products/Cat : 4,000K1
- Products Total : 40,000K2
- Data Volume : 120TB
- DN
- Max Products/Cat : 4,000K1
- Products Tot : 20,000K2
- Data Volume : 60TB

- 1) Products per catalog
- 2) Product totals at Node
- 3) Assume largest DN
- 4) Assume Mass Storage Media
- 5) Approximations

- Scalability
- Data product catalog capacities can be met by archive partitioning and existing database technologies
- In the MRO ERA, volume capacities can be met by existing storage technology4.

- Scope – MRO Era
- Missions : ~50
- Instruments : ~400
- Catalogs : ~1000

## Looking Forward

## Mass Storage

- Advancements in the development and availability of large scale mass-storage devices are expected to meet PDS needs for the foreseeable future
- Working with JPL enterprise mass storage group for storage and backup strategies.
- Watching and learning from other Systems – (e.g. PODAAC)

## Planetary Science Ontology

- The Planetary Science data model is being ingested into an ontology tool (i.e. PDS is creating a knowledge base)
- Provides better documentation for the domain data model
- Re-captures semantic information lost in relational  implementation
- Provides better graphic visualization
- The data model/ontology has been presented at numerous conferences and workshops as an important and successful science domain use case
- Protégé Ontology Tool 7th International Workshop
- Digital Library, CODATA, Mass Storage, Metadata Workshops
- CCSDS Data Management Standardization Efforts
- Various task are interested in individual aspects of the ontology
- Instrument -> NASA taxonomy effort, Crew Exploration Vehicle Information modeling
- Mission -> AMES Research Mission Control Technologies (MCT) Task
- Allows the development of automated classifiers for classifying new data product types, instruments, etc in support of standards development
- Provide a key component for the development of Semantic Web applications

## Planetary Science Ontology

**Planetary Science Ontology hosted in Protégé Ontology Tool**

## Planetary Science Ontology

**Benefits**