# Presentation: 804606

## Data Integrity Issues:How to Proceed?

**Data Integrity Issues:****How to Proceed?**

- Engineering Node
- Elizabeth Rye

- August 3, 2006
- http://pds.nasa.gov

## PDS Requirements for Data Integrity

- The PDS has made a commitment to ensure the integrity of its data archives.  This commitment is primarily spelled out in the Level 3 requirement 4.1.2:
- “PDS will develop and implement procedures for periodically ensuring the integrity of the data.”
- Several other Level 3 requirements suggest additional implications for data integrity assurance.

- Data Integrity Issues

- 3 August 2006

## PDS Requirements for Data Integrity

- The PDS is responsible for assisting data providers in determining how to validate the data they provide:
- “PDS will provide criteria for validating archival products” (1.3.3)
- The PDS is responsible for ascertaining that the data we deliver to the NSSDC is valid:
- “PDS will meet U.S. federal regulations for the preservation and management of data.” (2.8.3)
- “PDS will meet U.S. federal regulations for preservation and management of the data through its Memorandum of Understanding (MOU) with the National Space Science Data Center (NSSDC)” (4.1.5)

- Data Integrity Issues

- 3 August 2006

## PDS Requirements for Data Integrity

- The PDS is responsible for enabling our users to verify the integrity of the data they receive from us:
- “PDS will develop and maintain online mechanisms allowing users to download portions of the archive” (3.2.1)
- “PDS will develop and maintain a mechanism for offline delivery of portions of the archive to users”( 3.2.2)
- “PDS will provide mechanisms to ensure that data have been transferred intact” (3.2.3)
- The PDS needs to ensure the maintenance of data integrity through the media refreshing process:
- “PDS will develop and implement procedures for periodically refreshing the data by updating the underlying storage technology” (4.1.3)

- Data Integrity Issues

- 3 August 2006

## PDS Requirements for Data Integrity

- The PDS has a stated goal of utilizing standardized procedures in areas that affect inter-node data transfers:
- “PDS will provide standard protocols for accessing data, metadata and computing resources across the distributed archive” (2.7.3)

- Data Integrity Issues

- 3 August 2006

## PDS Requirements for Data Integrity

- From the above requirements, we can derive several areas of concern for data integrity:
- Verifying the integrity of data stored on physical media
- Detecting errors introduced during transfer of data to newer media
- Detecting errors that occur during transmission of data:
- From data providers to the PDS
- Between PDS nodes
- From the PDS to the NSSDC
- From the PDS to end users

- Data Integrity Issues

- 3 August 2006

## PDS Requirements for Data Integrity

- There are two additional areas not derivable from existing PDS requirements where data integrity issues are involved:
- The re-delivery of non-archived data during the operations phase of a mission
- The potential updating of data to newer formats long after it has been archived

- Data Integrity Issues

- 3 August 2006

## Mitch Gordon Survey

- For each numbered item, do you think that it is an important issue for us to address?
- Section A - It is critical that the PDS be able to ascertain the integrity of its archive.  This includes (but is not limited to):
- detecting errors that occur during the transmission of data from providers to the PDS,
- detecting errors that occur during the transmission of data between PDS nodes,
- detecting errors that occur during the transmission of data from the PDS to end users.
- detecting errors that occur during the transmission of data from the PDS to the NSSDC
- verifying the integrity of data stored on various types of external physical media (all of which have finite life spans),
- detecting errors introduced during transfer of data to newer media,

- Data Integrity Issues

- 3 August 2006

## Mitch Gordon Survey

|  |  | Lyle | Steve | Ed | Steve | Todd | David | Anne | Dick | Mitch |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  | ATM | EN | GEO | PPI | PPI | SBN-Az | SBN-Md | RS | Rings |
| A | Archive Integrity |  |  |  |  |  |  |  |  |  |
| A1 | Providers to PDS | Y | Y | Y | Y | N | Y | Y | N | Y |
| A2 | PDS to PDS | Y | Y | Y | Y | N | Y | Y | N | Y |
| A3 | PDS to users | Y | Y | M | Y | N | Y | Y | N | Y |
| A4 | PDS to NSSDC | Y | Y | M | Y | N | Y | Y | Y | Y |
| A5 | Integrity on media | Y | Y | Y | Y | Y | Y | Y | Y | Y |
| A6 | Upgrading to new media | Y | Y | Y | Y | Y | Y | Y | Y | Y |
|  |  |  |  |  |  |  |  |  |  |  |
| B | Identify tool |  |  |  |  |  |  |  |  |  |
| B1 | Use MD5 | M | Y | N | M | N | Y | N | M | Y |
|  |  |  |  |  |  |  |  |  |  |  |
| C | Policies for use of tool | Y | Y | N | Y | Y |  | N | M | Y |

- Data Integrity Issues

- 3 August 2006

## Possible Solutions to the Problem

- Checksums are widely accepted in the broader community as a means for ensuring data integrity
- MD5 checksums, in particular, are well suited to this purpose
- There has been no mechanism beside checksums suggested by any of the nodes as a means for detecting changes in data
- There is no consensus within the PDS as to whether we should limit ourselves to the MD5 checksum algorithm
- There is little consensus within the PDS as to whether we should use a standardized approach to utilizing checksums to verify data integrity

- Data Integrity Issues

- 3 August 2006

## Mitch Gordon Survey

- Section B - Identify a tool that can help (not necessarily be        sufficient) with any, or hopefully all, of the above.
- Use a single tool, MD5, for generating and validating checksums
- Section C - Establish policies for the use of the tool in a variety of situations.

- Data Integrity Issues

- 3 August 2006

## Mitch Gordon Survey

|  |  | Lyle | Steve | Ed | Steve | Todd | David | Anne | Dick | Mitch |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  | ATM | EN | GEO | PPI | PPI | SBN-Az | SBN-Md | RS | Rings |
| A | Archive Integrity |  |  |  |  |  |  |  |  |  |
| A1 | Providers to PDS | Y | Y | Y | Y | N | Y | Y | N | Y |
| A2 | PDS to PDS | Y | Y | Y | Y | N | Y | Y | N | Y |
| A3 | PDS to users | Y | Y | M | Y | N | Y | Y | N | Y |
| A4 | PDS to NSSDC | Y | Y | M | Y | N | Y | Y | Y | Y |
| A5 | Integrity on media | Y | Y | Y | Y | Y | Y | Y | Y | Y |
| A6 | Upgrading to new media | Y | Y | Y | Y | Y | Y | Y | Y | Y |
|  |  |  |  |  |  |  |  |  |  |  |
| B | Identify tool |  |  |  |  |  |  |  |  |  |
| B1 | Use MD5 | M | Y | N | M | N | Y | N | M | Y |
|  |  |  |  |  |  |  |  |  |  |  |
| C | Policies for use of tool | Y | Y | N | Y | Y |  | N | M | Y |

- Data Integrity Issues

- 3 August 2006

## Issues to be Addressed

- Data Integrity Issues

- 3 August 2006

## Standardization Issue

- Should we have a standardized approach across the PDS for storing and accessing checksums or should each node be permitted to use whatever mechanism it chooses?
- Some flexibility needed to deal with variety of ways in which data providers deliver data to the PDS
- Standardization permits the development of tools for generating, accessing, and periodically validating against checksums
- Standardization permits the addition of checksum tools to existing interfaces (like PDS-D and NSSDC delivery mechanism) to utilize and validate against checksums

- Data Integrity Issues

- 3 August 2006

## Urgency

- Volume of data returned from missions is increasing exponentially every couple of years
- Going back and calculating checksums for every file already in the PDS holdings is currently feasible, but will become a significantly more difficult task with each passing year

- Data Integrity Issues

- 3 August 2006

## Policy Questions to be Answered

- At what level of detail should checksums be required?
- For what parts of the archiving process should checksums be required?
- To what degree should standardization among nodes be insisted upon?
- When should we begin requiring checksums?

- Data Integrity Issues

- 3 August 2006

## Current Proposal (SCR 3-1034, V9)

- Mandates generation of file checksums for every file on every archive volume
- Mandates standardized format and location for storage of checksums
- Is insufficient to solve all data integrity problems, but is a necessary part of the solution
- Required for all missions archiving to v3.8 or higher of Standards Reference (roughly missions starting process late this year)

- Data Integrity Issues

- 3 August 2006

## Most Recent Votes on Checksum SCR

|  |  | Version 5 (MC) | Version 9 (Tech) |
| --- | --- | --- | --- |
| Science Nodes | ATM | no | reject |
|  | GEO | no | reject |
|  | IMG | yes | recommend |
|  | PPI | yes | recommend |
|  | RINGS | yes | recommend |
|  | SBN | no | recommend |
| Support Nodes | EN | yes | recommend |
|  | NAIF | yes | recommend |
|  | RS | no | reject |

- Data Integrity Issues

- 3 August 2006

## Options for Next Step

- Proceed with MC vote on version 9 of SCR
- Form new working group to come up with a new proposal
- MC draft policy on data integrity to provide further guidance to Tech group
- Drop the issue (fails to meet our requirements)
- Other?

- Data Integrity Issues

- 3 August 2006