# Presentation: 302503

## How Information is Made Usable in the PDS

- PDS NASA-HQ Review

- PDS Systems Engineering Team
- September 8, 2004

*http://pds.jpl.nasa.gov/*

## Place in Pipeline

## Purpose and Topics

- What makes data usable?
- Supporting information: metadata labels and ancillary files
- Standard structure: volume organization
- Standard data formats
- Resources that support data usability
- Planetary Science Data Dictionary
- PDS Standards
- PDS Data Engineering
- Archive lifecycle
- Project, Discipline Node, and Central Node Roles
- Looking forward

- Purpose: Describe how information is made usable in the PDS – focusing on data

## What makes data usable? – Labels

- Every data file in the PDS has a PDS label that...
- Describes properties of the file
- Points to the location of data
- Describes properties of the data
- Contains keywords that can be used for  search and correlation
- One label can describe multiple files and can be placed in the data file itself, or in a separate file
- PDS labels contain “metadata” – “data about the data”
- Written in Object Description Language (ODL) in a keyword = value syntax
- Using elements defined in the Planetary Science Data Dictionary

- Label for spectral data from Clementine

## Ancillary Files

- Ancillary files – information that documents or supports data
- Common types of ancillary files
- Documentation
- Calibration
- Software
- Geometry
- Browse files
- Etc...
- PDS-Specific files for searching
- Catalog files
- Index tables

- Browse image from Mars Odyssey

## Volume Structure

- Volume structure
- Specifies the directory structure for storing data and ancillary files in an archive volume
- Required/optional directories and files, file naming conventions, etc.
- Benefits
- Easier to validate
- Easier to navigate
- Work in progress for defining data transfer volumes and archive packages, as well as traditional archive volume

## Standard Data Formats

- “We find that the general user community desires to receive mission data products in simple, portable formats that are not tied to proprietary software”- Findings of  the PDS Working Group (PDSWG)
- Current PDS Standards suggest using TABLE, SPREADSHEET, IMAGE, SERIES, or SPECTRUM as the primary data objects
- Current PDS Standards do not specify or limit standard data formats
- PDS metadata is powerful enough to describe both simple formats and new, complex formats (e.g., TABLES with CONTAINER objects with BIT_COLUMNS inside)
- Projects have chosen proprietary formats, primarily for software support (e.g., over 10 years of software development to support the ISIS QUBE structure)

## Standard Data Formats - Proposed

- NASA HQ and many (but not all) in planetary community want to limit PDS support to a set of acceptable standard formats
- The PDS Working Group (PDSWG), recommended the following file structures for all mission data products distributed by the PDS:
- Flat (i.e., 2-dimensional, fixed-width) ASCII tables
- Flat (2D, fixed-width) binary tables using MSB integers and IEEE floating point numbers
- N-dimensional homogeneous arrays of binary values
- 2-dimensional, variable-length record binary files using MSB integers and IEEE floating point numbers
- Central Node System engineering agrees in principle with limiting support to specified set of standard formats
- Needed in order to define software support requirements
- Actual list to be determined by Project and Science user needs

## Level and Type of Processing

- Science data usability also determined by level and type of processing
- High level products
- Desired by science community
- Calibrated products vs. raw products with calibration files
- Products calibrated in physical units
- PDS Project Scientist and Discipline Nodes provide valuable feedback loop from science community to data providers and NASA Sponsors on types of higher level products desired

## Resources Supporting Usability – Data Dictionary

- Spectrum Object Definition

- Provides structure and a controlled vocabulary for PDS metadata
- Defines objects and the required and optional sub-objects and keywords they contain
- Defines keywords and the permitted set of values they can have
- Data dictionary is actually a set of tables in the Central Node Catalog
- Flat file representation of the data dictionary (pdsdd.full) used to validate PDS labels
- CN responsible for maintaining the data dictionary
- Discipline Nodes, missions, and science community all provide definitions

## PDS Standards

- PDS Standards – specification for all elements of data set archive
- How to describe data using Object Description Language (ODL)
- How to construct PDS labels
- Required and optional ancillary information
- How to assemble PDS compliant volumes
- This specification enables creation of tools – See Tools presentation
- Central Node maintains the PDS Standards
- Discipline Nodes, missions, and the science community request changes, additions, clarification, user support
- Ongoing work in “tightening up” the standards and making it a more rigorous specification

## PDS Data Engineering

- What is PDS Data Engineering?
- Supporting the design, generation, transfer (to PDS), validation, distribution, and long term preservation of planetary data set archives
- Archive Life Cycle
- The processes listed above can be thought of as phases in the “life” of a data set archive, hence the “Archive Life Cycle”
- PDS Data Engineering can be thought of as supporting the Archive Life Cycle for planetary science data
- Note: there is some variation in the way the phases of the Archive Life Cycle are listed and detailed across various PDS documents. This is because PDS is still in the process of defining and filling out the details of the Archive Life Cycle
- Life Cycle events
- Like any life cycle, there are critical events in the Archive Life Cycle, e.g., completing ICDs, data deliveries, peer review, etc.
- Archive Life Cycle events have been mapped to NASA’s Project Life Cycle

## Project & Archive Life Cycles

## Project Role

- Project is the data provider
- Project has a set of deliverables
- Design deliverables, e.g., the Archive Plan and ICDs
- Data deliverables, e.g., archive data products and ancillary files
- Central Node will provide online support for tracking project deliverables
- Helps in reporting quarterly status (See reporting template, opposite)
- Current URL: http://stardev:8080/pdswiki/bin/wiki/

## PDS Data Engineering Roles

- PDS Data Engineering is a partnership between
- Lead Discipline Node – Works with project and has the “big picture” for science data archiving
- Other Discipline Nodes – Work with instrument teams to support the creation of scientifically useful data set archives
- Central Node – Works with the Lead Discipline Node to help with coordination and tracking. Works with Discipline Nodes to ensure that the data set archives are PDS Standards Compliant and usable by the system
- Good example of the partnership is during peer review
- Peer review a major way of assuring high quality data products
- Typically lead by the Discipline Node
- Supported by the Central Node Data Engineer, who does thorough check on data archive quality and compliance to PDS Standards

## Additional Central Node Roles

- Check quality of all project design deliverables
- Archive Plans
- Catalog information for data sets
- Interface Control Documents (ICDs)
- Proposed new metadata (e.g., new keywords)
- Sample volumes
- Ensure that Online System has all information needed for a successful data release
- Catalog: data set information and resource links in place
- Data Dictionary: all product metadata are defined
- All data-to-be-released is assigned a repository and primary resource to handle product access (e.g., Image Atlas, PDS Explorer, etc.)

## Allocation of Tasks

- PDS DE tasks are allocated between:
- Lead Discipline Node (LDN)
- Discipline Nodes working with the instrument teams (DN)
- Central Node (CN)
- Task allocation will vary according to project and LDN needs
- To estimate workforce needs, CN sent list of DE tasks the Nodes, asking what percentage of effort they would allocate between the LDN, CN, and DN for each task (See Opposite)
- Just first order estimate
- By tracking DE activities, better estimates will be made

## CN Design vs. Operations

- Archive Life Cycle has natural divide between design (planning) and operations (production) phases
- CN has two teams along those lines
- Data Engineering / Mission Interface
- Operations
- Data Engineering team
- Executes all CN tasks relating to design of data set archives
- Tracks project deliverables
- Provides DN and project support
- Operations team
- Runs  the PDS Online System
- Ingests, distributes, maintains data
- Provides user support

## Looking Forward

- Completing definition and documentation of Archive Life Cycle
- Improve understanding and engineering of the processes
- Improve communication between project, discipline node, central node
- Enable accurate tracking and status reporting
- Status: Proposer’s Archive Guide (PAG) completed and Archive Preparation Guide (APG) in progress, describes Archive Life Cycle from proposal writers and instrument team perspectives, respectively
- Improving tools and interfaces for data engineering
- Enable projects and discipline nodes to do more – e.g., entering catalog information directly and validating products before delivery
- Eliminate tedious tasks – e.g., formatting catalog files by hand
- CN data engineers will show projects how to take advantage of these new tools and administrative interfaces
- CN data engineers will be able spend more time supporting the design process and have more technical involvement – e.g., data product modeling and data product production design

## New System Capabilities 

- CN Data Engineers will help Projects and Discipline Nodes take advantage of new system capabilities...
- Correlative searches
- Need to properly define correlative and geometric metadata
- IVT System under development (See IVT Presentation)
- Introduces validation data products against the ICD
- Implies more upfront design work, modeling of product types
- Will require additional CN Data Engineering support

## Resources

- Planetary Science Data Dictionary (online lookup): http://pds.jpl.nasa.gov/tools/data_dictionary_lookup.cfm
- PDS Standards Reference: http://pds.jpl.nasa.gov/documents/sr/index.html
- Proposer’s Archive Guide (PAG): http://pds.jpl.nasa.gov/documents/pag/index.html
- Archive Preparation Guide (APG) – DRAFT