# Presentation: 516441

## How Information is Maintained in the PDS

- PDS NASA-HQ Review

- PDS Systems Engineering Team
- September 8, 2004

*http://pds.jpl.nasa.gov/*

## Place in Pipeline

## Purpose and Topics

- Preserving the Bits
- Preventing data loss
- Ensuring data integrity
- Current archiving practices
- Archiving on CD/DVD
- Drivers for change
- The PDS-NSSDC interface
- Preserving usability: preventing obsolescence
- CN effort to ensure usable datasets into the future

- Purpose: Describe how information is preserved and kept usable in the PDS

## Preserving the Bits

- One often hears: ”The PDS is responsible for the long-term preservation of planetary science data” – What does this mean?
- Preventing data loss
- Unless data are explicitly removed from the archive...
- All data entering the archive must always remain in the archive
- All data in the archive must be findable
- All data in the archive must be loadable back into a computer system
- Ensuring data integrity
- Prove that the data read back from the archive are the same as the data originally stored in the archive
- No corruption during ingest, transfer, backup, media refresh, etc.

## Checksums

- An established method of ensuring data integrity
- A checksum is a character string derived from an input file, such that when even one bit in the file changes, the checksum changes
- Checksums can be calculated and recorded for every file, recursively for volumes or delivery
- Checksums can be re-calculated and checked against the original, after any operation, e.g., transfer, backup, media refresh, etc...
- Also to periodically check for “bit rot” on media
- Example: md5deep –r /mnt/cdrom

## Current Archiving

- The mission creates three archive copies
- Written on PDS-approved long-term media, e.g., CD-ROM, DVD-5
- One copy held at the PDS Discipline Node responsible for the data set
- One copy held at the PDS Central Node
- One copy held at the NSSDC
- In the past: PDS distributed the actual archive CD/DVDs
- Now: PDS distributes data through the PDS Online System
- This means: data can be delivered for distribution prior to archiving, so data loss must be prevented during distribution phase
- System mirroring and backup
- At least two geographically separated backup locations
- Missions to keep data (or ability to re-generate data) until archiving

## Drivers for Change

- Data volume growth has outpaced capacity of CD/DVD archive media
- MGS Primary ~ 600 GB
- Odyssey Primary ~ 8 TB
- MER ~ 1.5 TB
- MRO ~ 85 TB
- Established practices for archiving large volumes of science data, e.g., Earth Science data (EOSDIS)
- Desire to better utilize NSSDC
- Designated as official NASA deep archive for space data
- Already receives a copy of every PDS archive volume

- DVD-5 Archive Volumes

## PDS and NSSDC

- New PDS-NSSDC archiving process is being tested
- Part of PDS D03 development effort
- Testing and evaluation complete by start of FY05
- How it works
- PDS uses NSSDC-supplied software (MPGA) to create an Archive Information Package (AIP) from a PDS volume
- PDS transfers AIP to NSSDC – electronically or on agreed-upon media
- NSSDC handles all details of deep archive, I.e., bit preservation
- NSSDC using connection to offsite storage (Iron Mountain)
- PDS will then test data retrieval and integrity of retrieved data (checksums)
- “Backup is easy, Restoration is hard”
- For more details, see the NSSDC presentation to PDS Management Council, in Supporting Material

## Source Site

- _Source Site_

- _NSSDC Site_

**PDS Volume**

- MPGA
- Packaging
- Software

- Configuration
- Files for PDS
- Volume Creation

- AIP

- AIP

- Media or FTP

- DIOnAS
- Archiving
- Software
- (validation
- and DB update)

- Offsite
- Backup
- Storage

- DLT Jukebox
- Storage

**PDS Volume Reconstituted**

**MPGA = Multi-file Package Generator and Analyzer**

- PDS-NSSDC Interface

## Preserving Usability

- Upfront work
- Creating data set archives designed to last
- Choosing formats that last
- Standards compliance
- Metadata migration
- Updating index tables
- Updating labels, issues with attached labels
- Propose checksums for data portion of all data with attached labels
- Current research into format migration
- Issue: Ensuring data integrity during format migration
- One strategy: on-the-fly translation, keeping original data

## Resources

- NSSDC Home Page: http://nssdc.gsfc.nasa.gov/
- NSSDC Presentation at PDS Management Council, June 2004