# Presentation: 777525

## Building the Archives of the Future: Self-Describing Records

**Building the Archives of the Future: ****Self-Describing Records**

- Kenneth Thibodeau
- Director, Electronic Records Archives Program
*National Archives and Records Administration*July 18, 2001

**Notes:**
Information and services survivability

Our Nation’s security, commerce, education, and well-being depend increasingly on our information infrastructure. It is thus critical to ensure the survivability of that infrastructure in the face of malicious attacks or viruses, equipment or software failures, and overload. Survivability means that services will be available when needed and information will be delivered in a timely fashion.  Services must operate correctly and the information they deliver must be of high quality or be identified if otherwise. Survivability includes long term preservation of information; a document of historical importance should be preserved, even as underlying storage technologies and information representation evolve and even if it is not accessed for hundreds of years.
P.41

## The Electronic Records Archives Vision

- Overcome technological obsolescence in a way that preserves demonstrably authentic records.
- Build a dynamic solution that incorporates the expectation of continuing change in information technology and in the records it produces.
- Find ways to take advantage of continuing progress in information technology in order to maintain and improve both performance and customer service

**Notes:**
• Concern with digital preservation  focuses on the twin problems of technology obsolescence and the fragility of digital media.  Based on 30 years experience in preserving digital information at NARA, we believe that the media problem is a manageable one, but obsolescence remains a major problem. {Give examples} In fact, a focus on obsolescence is misdirected.  Digital preservation is a dynamic problem.  It is not enough to keep the information around.  You must keep it in a way that enables you to use the best available technology to discover and deliver the information.  The National Archives is preserving electronic records from World War II.  Our customers would not be happy if they could only access these records by entering requests on punch cards, using 1940’s vintage computers, and getting computer printouts in all uppercase in response.  At NARA we want to preserve digital information in a way that takes advantage of continuing technological progress in order to provide continuing improvements in customer service.
• Even apart from concern with customer service, it is clear that a digital preservation strategy must be intrinsically dynamic simply because obsolescence is not a one-time problem. Even solving all the problems of obsolescence that we face today would not be enough.  Today’s progress....
• Furthermore, even if we succeed in solving existing problems of obsolescence, it is probable that continuing progress will produce better solutions to the same problems.

## Critical Challenge

- Proven methods for preserving digital information across generations of technology are limited to the simplest formats
- Available methods are increasingly inadequate
- The market has not delivered solutions.

## How will we develop the Electronic Records Archives? 

- How will we develop the Electronic Records Archives?

## ERA Infrastructure Concept

- Workbench

- Records
- Creator

- Public User

- Trusted
- Repository

- Records
- Creator

- Records
- Creator

- Public User

- Digital
- Library

- NARA
- User

- SCALABLE
- Gb/sec Internet
- Grid Security
- Distributed Processing
- Mediation among Systems
- Distributed, redundant Storage
- Infrastructure Independence

- Government
- User

- Workbench

- NARA
- User

## ERA Infrastructure 

- In NARA (using NARANET)
**Archival workstations for staff **
**Reference workstations for researchers **
- On the National Information Infrastructure, under NARA’s control
**ERA Ingest & Distribution portal ****(Internet & Media)**
**POP repositories ****(Normal, Trusted, Special)**
**Affiliated Archives**
- On the National Information Infrastructure
**Agency systems with access to NARA portal**
**Digital Libraries**
**Public Users**

## NARA Partnerships

**Open Archival Information System** (OAIS) Reference Model
- NASA, Consultative Committtee on Space Data Systems
**Distributed Object Computation Testbed**  (DOCT)
- Defense Advanced Research Projects Agency, U.S. Patent and Trademark Office
**National Partnership for Advanced Computational Infrastructure**  (NPACI)
- National Science Foundation
**Presidential Electronic Records Processing Operational System** (PERPOS)
- Army Research Laboratory, Georgia Tech Research Institute
**Archivist’s Workbench**
- NHPRC Grant to San Diego Supercomputer Center
**International research on Permanent Authentic Records in Electronic Systems** (InterPARES)
- 7 international, multidisciplinary research teams, 10 national archives

## ERA Functional ModelAn Open Archival Information System Implementation

- ERA Functional ModelAn Open Archival Information System Implementation

**Producer**

- Submission
- Information
- Packages

- OAIS

- Archival
- Information
- Packages

- Dissemination
- Information
- Packages

**Consumer**

- queries

- orders

- Result sets

## InterPARES: Preserve Electronic Records Model

## Information Management Architecture for Persistent Object Preservation

- Information Management Architecture for Persistent Object Preservation

- Attributes
- Semantics

**Knowledge**

**Information**

**Data**

**Ingest **
**Services**

**Management**

**Access**
**Services**

- (Topic Maps / Buckets / Model-based Access)

- (Data Handling System - SRB / FTP / HTTP)

- MCAT/HDF

- Grids

- XML DTD

- SDLIP

- XTM DTD

- Rules - KQL

- Information
- Repository

- Attribute- based
- Query

- Feature-based
- Query

- Knowledge or
- Topic-Based
- Query / Browse

- Knowledge
- Repository for
- Rules

- Relationships
- Between
- Concepts

- Fields
- Containers
- Folders

- Storage
- (Replicas,
- Persistent IDs)

## ERA Concept model

## ERA Processes

**ACCESSIONING**

## Accept an Accession? 

- Accessioning Workbench

**Transferred**
**Records**

- Transfer Documentation

- What should the agency have transferred?
- What did the agency say it transferred?
- What was transferred?

**Expected**
**Records**

**Notes:**
To accept an accession, we need to determine that the records transferred are records we are committed to preserving.
To do this, we need first to determine what records the agency should have transferred, then whether the records actually transferred meet our expectation.

We determine what records should be transferred from the appraisal report.  We determine when records should be transferred and what records should be included in each transfer from the related records schedule.
Once we articulate what records were expected, we compare our expectation with what the agency told us it transferred.
If the expected transfer differs from the agency description of what was transferred, we need to decide if the difference is significant.  
If the difference is not significant, we adjust our expectation.
If the difference is critical, we reject the transfer.
If the difference is significant, but not critical, what do we do?

If our expectation matches what the agency said it sent, we proceed to compare our expectation with what was actually transferred.
If the actual transfer is not what was expected, we need to decide if the difference is significant.

## SELF-DESCRIBING

- Records
- Files
- Series
- Record Systems

## Persistent Object Method

- Characterize significant properties of the things that are to be preserved.
- Express these properties in formal models
- Encapsulate objects in metadata defined in the models.
- Use software “mediators” to enable future technologies to interpret the models and metadata
- to rebuild and repopulate collections
- to re-present the records
- support information discovery and delivery.

## E-mail: Groupwise view

## E-mail: text editor view

## E-mail: MIME-aware view

## Tagged MIME E-mail Message

**<Message-Id> **p05010429b6e7e938e0b4@[10.2.68.205] **</Message-Id>**
**<X-Organization>**USC/Information Sciences Institute**</X-Organization>**
**<X-Phone>**(310) 822-1511  ext. 766**</X-Phone>**
**<X-Fax>**(310) 822-0751**</X-Fax>**
**<Date>**Wed, 28 Mar 2001 22:22:30 -0500**</Date>**
**<To>**Readers:;**</To>**
**<From>**Yigal Arens {arens@ISI.EDU}**</From>**
**<Subject>**Announcing DG Online, the magazine of digital government research**</Subject>**
**<Conten**t**-Type**: multipart/alternative; boundary="============_-1226283781==_ma============"
- --============_-1226283781==_ma============**>**
**<Content**-Type: text/plain; charset="us-ascii" ; format="flowed"**>**
**<Message_Body>**DG Online {http://www.dgrc.org/dg-online/}.
- ....

**Notes:**
This is a specific example of the application of the Persistent Object Preservation Method.  One of the demonstrations that San Diego did in 1999 was with a collection of 1 million USENET e-mail postings.  The emphasis in this demonstration was on the volume of objects (1 million).  The objects themselves are relatively simple: USENET messages are in plain ASCII, with no attachements, and there is a published standard (rfc1036) which defines the structure of the messages.
   This example shows the rfc1036 standard translated into an XML Document Type Definition. (Actually only part of the translation)  An USENET message is simple: it can contain only 2 elements: headers and a body.  But there can be as many as 3 types of headers: required, optional and other.  Required and optional headers are pre-defined, with the obvious difference any optional_header may not be present in a message, while all required_headers must be.
   The DTD on the slide shows that each of the required_headers consists of some data entered by the user (PCDATA).  It also shows that each element has a required attribute (ATTLIST), a sequence number (seqno).  This ensures that the message can be recreated with all of the elements in proper order.  The DTD goes on to specify the sequence number attribute for all required-headers, then includes the element definition and attributes for the optional-headers.
   Other_headers are created by individual users.  In a collection of 1 million messages, San Diego found over 1,000 user defined headers.  The DTD for this collection includes all of those elements.

## Structure of E-mail Message aka: Document Type Definition

**<!ELEMENT Email_Message (Header, Message Body, Attachment*)>**
**<!ELEMENT Header (Internal Header, External Header)>**
**<!ELEMENT Internal Header****	****(Message_Id, X-Organization, X-Phone, X-Fax)>**
**<!ELEMENT Message-Id>**
**<!ELEMENT X-Organization>**
**<!ELEMENT X-Phone>**
**<!ELEMENT X-Fax>**
**<!ELEMENT External Header****	****(Date, To, From, Subject)>**
**<!ELEMENT Date****	****(Weekday, Day_of_Month, Month, Time>**
**<!ELEMENT To****	****	**(#PCDATA)+**>**
**<!ELEMENT From****	**(#PCDATA)**>**
**<!ELEMENT Subject****	**(#PCDATA)***>**
**<!ELEMENT Message_Body****	**(#PCDATA)***>**

**Notes:**
This is a specific example of the application of the Persistent Object Preservation Method.  One of the demonstrations that San Diego did in 1999 was with a collection of 1 million USENET e-mail postings.  The emphasis in this demonstration was on the volume of objects (1 million).  The objects themselves are relatively simple: USENET messages are in plain ASCII, with no attachements, and there is a published standard (rfc1036) which defines the structure of the messages.
   This example shows the rfc1036 standard translated into an XML Document Type Definition. (Actually only part of the translation)  An USENET message is simple: it can contain only 2 elements: headers and a body.  But there can be as many as 3 types of headers: required, optional and other.  Required and optional headers are pre-defined, with the obvious difference any optional_header may not be present in a message, while all required_headers must be.
   The DTD on the slide shows that each of the required_headers consists of some data entered by the user (PCDATA).  It also shows that each element has a required attribute (ATTLIST), a sequence number (seqno).  This ensures that the message can be recreated with all of the elements in proper order.  The DTD goes on to specify the sequence number attribute for all required-headers, then includes the element definition and attributes for the optional-headers.
   Other_headers are created by individual users.  In a collection of 1 million messages, San Diego found over 1,000 user defined headers.  The DTD for this collection includes all of those elements.

## File of Electronic Records

**File of Electronic Records**

- REPORT OF INDEPENDENT ACCOUNTANTS
-  To the Board of Directors and Stockholders of Great Plains Software, Inc.
-      In our opinion, the consolidated financial statements listed in the
-  accompanying index present fairly, in all material respects, the financial position
-  of Great Plains Software, Inc. and its subsidiaries at May 31, 1999 and 1998,
-  and the results of their operations and their cash flows for each of the three
-  years in the period ended May 31, 1999, in conformity with generally
-  accepted accounting principles. In addition, in our opinion, the financial
-  statement schedules listed in the accompanying index present fairly, in all
-  material respects, the information set forth therein when read in conjunction
-  with the related consolidated financial statements. These financial statements
-  and financial statement schedules are the responsibility of the Company's
-  management; our responsibility is to express an opinion on these financial
-  statements and financial statement schedules based on our audits. We
-  conducted our audits of these statements in accordance with generally
-  accepted auditing standards, which require that we plan and perform the audit
-  to obtain reasonable assurance about whether the financial statements are free
-  of material misstatement. An audit includes examining, on a test basis, evidence
-  supporting the amounts and disclosures in the financial statements, assessing
-  the accounting principles used and significant estimates made by management,
-  and evaluating the overall financial statement presentation. We believe that our
-  audits provide a reasonable basis for the opinion expressed above.
-  /s/ PricewaterhouseCoopers LLP
-  PricewaterhouseCoopers LLP
-  Minneapolis , Minnesota
-  June 25, 1999

- Types of Records

**Notes:**
Suppose we had a file that contained the 3 documents shown above: a report, a form and an e-mail message. We would need 3 separate schemas, or models, to desribe the structure of each of these document types.

## eXtensible Business Reporting Language (XBRL)Example

- eXtensible Business Reporting Language (XBRL)Example

## XBRL DTD

## Structure expressed as Tree

- Internal Header

- External Header

- *

- *

- Electronic Mail Message

## Accept an Accession? 

- Accessioning Workbench

**Transferred**
**Records**

- Transfer Documentation

**Expected**
**Records**

**Notes:**
To accept an accession, we need to determine that the records transferred are records we are committed to preserving.
To do this, we need first to determine what records the agency should have transferred, then whether the records actually transferred meet our expectation.

We determine what records should be transferred from the appraisal report.  We determine when records should be transferred and what records should be included in each transfer from the related records schedule.
Once we articulate what records were expected, we compare our expectation with what the agency told us it transferred.
If the expected transfer differs from the agency description of what was transferred, we need to decide if the difference is significant.  
If the difference is not significant, we adjust our expectation.
If the difference is critical, we reject the transfer.
If the difference is significant, but not critical, what do we do?

If our expectation matches what the agency said it sent, we proceed to compare our expectation with what was actually transferred.
If the actual transfer is not what was expected, we need to decide if the difference is significant.

## How does ERA determine the dates of records?

- E-mail
- All e-mail contains a field indicating the date it was sent.  For the sender, that is the date of the record.  ERA needs to search the date-sent fields.
-   *(Technology solution)*
- Attachments to e-mail messages
- Attachments to a record are parts of that record.  The date of the message is the date of the record.
*(Archival principle)*
- Records forwarded, via e-mail, for filing in a recordkeeping system
- E-mail is used only to transmit a record to the system.  The date of the attached record depends on the record
*(Archival principle)*

## Defining models for electronic recordse.g. E-mail

- All E-mail
- Groupwise mail
- cc:mail
- USENET mail
- User defined fields
- .....

## Transformation 

- Accessioning
- Workbench

**Transferred**
**Records**

**Metadata**

**Persistent**
**Records**

**Exceptions**

**Mistakes**

**Archival **
**Exceptions**

## Aggregation 

- Accessioning
- Workbench

- Container
- Metadata

**Persistent**
**Records**

- Container

**Exceptions**

## Aggregation: risk management 

- Accessioning
- Workbench

**Persistent**
**Records**

- Container Metadata

- Container

**Exceptions**

## Risk Management:Multi-Valent Documents 

## ERA: Reference Process

- Metadata

**Archival**
**Repository**

- Query

- Rebuild

- Present

** ****Collection**

** ****Collection**

** ****Collection**

- User

## Process: Check metadata for the series to identify relevant DTDs

- Reference Workbench

- Repository

**Metadata**

_**Series Metadata**_
**Files**
**Records**

**Notes:**
To accept an accession, we need to determine that the records transferred are records we are committed to preserving.
To do this, we need first to determine what records the agency should have transferred, then whether the records actually transferred meet our expectation.

We determine what records should be transferred from the appraisal report.  We determine when records should be transferred and what records should be included in each transfer from the related records schedule.
Once we articulate what records were expected, we compare our expectation with what the agency told us it transferred.
If the expected transfer differs from the agency description of what was transferred, we need to decide if the difference is significant.  
If the difference is not significant, we adjust our expectation.
If the difference is critical, we reject the transfer.
If the difference is significant, but not critical, what do we do?

If our expectation matches what the agency said it sent, we proceed to compare our expectation with what was actually transferred.
If the actual transfer is not what was expected, we need to decide if the difference is significant.

## Translate E-mail DTD to Relational Database Structure

- Translate E-mail DTD to Relational Database Structure

**Reference Workbench**

## Process: Retrieve the records and place in the target structure

- Reference Workbench

- Repository

**Metadata**

**Notes:**
To accept an accession, we need to determine that the records transferred are records we are committed to preserving.
To do this, we need first to determine what records the agency should have transferred, then whether the records actually transferred meet our expectation.

We determine what records should be transferred from the appraisal report.  We determine when records should be transferred and what records should be included in each transfer from the related records schedule.
Once we articulate what records were expected, we compare our expectation with what the agency told us it transferred.
If the expected transfer differs from the agency description of what was transferred, we need to decide if the difference is significant.  
If the difference is not significant, we adjust our expectation.
If the difference is critical, we reject the transfer.
If the difference is significant, but not critical, what do we do?

If our expectation matches what the agency said it sent, we proceed to compare our expectation with what was actually transferred.
If the actual transfer is not what was expected, we need to decide if the difference is significant.

## Persistent Object Preservation

- Aims at independence of technological infrastructure
- Reduce threats to integrity and authenticity by minimizing changes over time.
- Embeds changes in a comprehensive information management architecture designed for preservation
- Inherently extensible
- Facilitates use of future, advanced technologies, without requiring change in what is preserved.
- Currently beyond state of the art of information technology.

## Self-describing Objects for Records Management

- Facilitate management, exchange, and disposition of records
- explicitly identify the content of records, files, series,...
- express how content is organized
- allow the content to be stored once and used in different documents
- separate, but link, management of content and presentation
- capture the relationships among documents and collections of documents
- and support multiple views of a collection of documents
- all in plain language

## Thank you.

- For more information:
- www.nara.gov/era