# Presentation: 541280

## A Daunting PREMIS:Implementing Preservation Metadata within the METS Framework

- Jerome P. McDonough
- Graduate School of Library & Information Science, UIUC
- ICDAT 2006
- Inst. of Information Science, Academia Sinica
- October 19, 2006

## One Great Loss for Mankind

**Notes:**
The original Apollo 11 television broadcasts were transmitted from the Moon using black & white slow-scan TV at a rate of 10 frames per second with 320 lines per frame.  The television transmission from the Lunar Module was received by two NASA tracking stations (a 64 meter dish at Goldstone, California, and a 26 meter dish near Canberra, Australia), as well as the CSIRO Parkes Radio Telescope in Australia.

## One Great Loss for Mankind

- Source: Sarkissian, John M. (21 May 2006). The Search for the Apollo 11 SSTV Tapes.
- Parkes, Australia: CSIRO Parkes Observatory.
- http://www.honeysucklecreek.net.nyud.net:8080/Apollo_11/tapes/Search_for_SSTV_Tapes.pdf

**Notes:**
The television signal received by the three monitoring stations were actually of higher quality than what was broadcast at the time.  The conversion process employed to change the SSTV signal into NTSC employed an RCA optical scan converter which reduced the frame resolution from 320 lines to 262.5 and introduced additional signal noise.  The scan-converted signal was transmitted to Mission Control in Houston (introducing more signal noise), which then fed the video signal to the television networks for broadcast. The NTSC broadcast version is on the left; on the right is a polaroid photograph that was taken of the SSTV display at Goldstone.  The original superior SSTV signal, however, was recorded on to 1” tape at all three tracking stations, and the tapes were mailed to Goddard Space Flight Center.   What happened to those tapes at that point is less clear.  They appear to have been transferred to the National Archives about 1970, and the National Records Center has records confirming that the tapes were returned to Goddard Space Flight Center at Goddard’s request in 1984.  But there are no records at Goddard of them having received the returned tapes.  The Data Evaluation Lab at Goddard Space Flight Center, possibly the only facility with the equipment and expertise to play back the tapes if they are found, is slated to be closed in October of 2006 due to budget cuts.  If the tapes are not located soon, we may never be able to recover the original signal transmitted on July 20 and 21, 1969.

## “Houston, we’ve had a problem here.”

- Loss of data due to format conversions
- Need to insure viable access to playback devices for media
- Inadequacy of traditional archival practice for insuring item-level access to media
- Need to detailed event history to document life-cycle/provenance of information

**Notes:**
As a cautionary tale, the story of the Apollo 11 SSTV tapes touches upon a number of issues that are of concern to all of us seeking to preserve audio/visual materials, including issues regarding lossy conversion processes and the need to insure access to playback equipment if we are to provide access to media.  Some of the more crucial lessons to be learned, however, revolve around metadata.
Goddard has no record of the tape’s location or disposition after their receipt in 1969.  The National Archives does have records to confirm the receipt of a large number of tapes from Goddard during the 1970 time period, but nothing to definitively confirm they received the Apollo 11 tapes, since tapes were accessioned without any reference to the mission number or program.  Tapes were also all accessioned as “magnetic tape” without any indication as to whether the tape contained audio, telemetry data, or video.  There is some evidence to suggest that one particular accession did contain the Apollo mission telemetry tapes, and that these tapes were all recalled by Goddard Space Flight Center for permanent retention.  However, the metadata recorded for the tapes submitted to the National Archive is insufficient to determine for certain whether or not the Apollo 11 tapes were ever submitted, or whether they were part of the Accession that appeared to contain much of the Apollo data.  The partial evidence available from the National Archives highlights two significant facts: 1. Traditional archival practice does not typically capture enough item-level information to allow unambiguous identification of the contents of a particular accession; 2. Even limited information about lifecycle events surrounding digital information can prove critical in trying to establish information’s provenance and history.

It is this critical need to standardize the recording of digital preservation and provenance metadata for digital media which led OCLC and RLG to launch the Preservation Metadata Implementation Strategies effort, better known as PREMIS.

## History of PREMIS

- OCLC/RLG Preservation Metadata Framework Working Group (2001-2002)
- “...to define the concept of preservation metadata...and evaluate the prospects for a community-wide, consensus-building activity....”
- Final Report: _Preservation Metadata for Digital Objects: A Review of the State of the Art_
- “...to develop a framework outlining the types of information -- i.e., metadata -- that should be associated with an archived digital object.”
- Final Report: _A Metadata Framework to Support the Preservation of Digital Objects_

**Notes:**
In 2001-2002, OCLC and RLG convened an international working group of experts, representing a variety of institutions and initiatives, to consider and discuss fundamental issues associated with the use of metadata to support digital preservation.

## History of PREMIS

- PREservation Metadata Implementation Strategies [PREMIS] (2003-2005)
- “Develop a core preservation metadata set, supported by a data dictionary, with broad applicability across the digital preservation community.”
- “Identify and evaluate alternative strategies for encoding, storing, and managing preservation metadata in digital preservation systems.”
- Final Report: _Data Dictionary for Preservation Metadata: Final Report of the PREMIS Working Group_
- _PREMIS Maintenance Activity_ at Library of Congress, including _XML Schema_

**Notes:**
In 2003, a follow-on working group to the Preservation Metadata Framework Working Group was created, also sponsored by OCLC and RLG.
Composed of international experts from institutions that had developed or were currently developing digital preservation capacity, including  representatives from a variety of domains interested in digital preservation, including libraries, museums, archives, government, and the private sector.

## PREMIS Data Model

**Notes:**
“To facilitate the logical organization of the PREMIS metadata elements, the group developed a simple model of five types of entities involved in digital preservation activities: Intellectual Entities, Objects, Events, Rights, and Agents.4 In the data model diagram, entities are drawn as boxes while the relationships between them are drawn as lines. The direction of the arrow shows the direction of the relationship link defined in the Data Dictionary; for example, the arrow from Rights to Agents means the metadata defined for Rights includes semantic units to identify the related agent(s). A double-headed arrow means reciprocal links are defined.”

* An Intellectual Entity is a coherent set of content that is reasonably described as a unit, for example, a particular book, map, photograph or database.  An Intellectual Entity can contain other Intellectual Entities....  An Intellectual Entity may have one or more digital representations

An Object, or Digital Object, is a discrete unit of information in digital form.  Objects can be one of three subtypes: Representation, File or Bitstream
Representation -- set of files, including structural metadata, needed for a complete and reasonable rendtion of an Intellectual Entity, e.g., for a page-image book an XML file giving structural metadata and all of the individual page image files.
File -- a named and ordered sequence of bytes that is known by an operating system.
Bitstream -- contiguous or non-contiguous data within a file that has meaningful common properties for perservation purposes
* An Event is an action that involves at least one object or agent known to the preservation repository.
* An Agent is a person, organization, or software program associated with preservation events in the life of an object.
* Rights, or Rights Statements, are assertions of one or more rights or permissions pertaining to an object and/or agent.

## PREMIS Data Dictionary: Object

- Object Identifier
- Preservation Level
- Object Category
- Object Characteristics
- Creating Application
- Original Name
- Storage
- Environment
- Signature Information

- Relationship
- Linking Event Identifier
- Linking Intellectual Entity Identifier
- Linking Permission Statement Identifier

- An Object can be associated with one or more Rights statements, can participate in one or more Events, and can be related to one or more Agents

## PREMIS Data Dictionary: Event

- Event Identifier
- Event Type
- Event Date & Time
- Event Detail
- Event Outcome

- Linking Agent Identifier
- Linking Object Identifier

- An Event must be related to one or more objects, and can be related to one or more Agents.

## PREMIS Data Dictionary: Agent

- Agent Identifier
- Agent Name
- Agent Type

- An Agent may hold or grant one or more rights, may  carry out, authorize, or compel one or more events, and may create or act upon one or more objects.

## PREMIS Data Dictionary: Rights

- Permission Statement Identifier
- Granting Agreement
- Permission Granted

- Linking Object
- Granting Agent

## <premis:object>

-      <premis:objectIdentifier>
-           <premis:objectIdentifierType>hdl</premis:objectIdentifierType>
-           <premis:objectIdentifierValue>loc.music/gottlieb.09611</premis:objectIdentifierValue>
-      </premis:objectIdentifier>
-      <premis:objectCategory>file</premis:objectCategory>
-      <premis:objectCharacteristics>
-           <premis:fixity>
-                <premis:messageDigestAlgorithm>MD5</premis:messageDigestAlgorithm>
-             <premis:messageDigest>36b0319.....</premis:messageDigest>
-                <premis:messageDigestOriginator>LocalDCMS</premis:messageDigestOriginator>
-           </premis:fixity>
-           <premis:size>20800896</premis:size>
-           <premis:format>
-                <premis:formatDesignation>
- 	     <premis:formatName>image/tiff</premis:formatName>
- 	     <premis:formatVersion></premis:formatVersion>
-                </premis:formatDesignation>
-           </premis:format>
-      </premis:objectCharacteristics>
- </premis:object>

**Notes:**
The PREMIS framework working group developed an XML schema for the PREMIS data model, so that
PREMIS information could be recorded in XML, and included in other XML records, such as those conforming to the METS standard.
Here’s an example of a PREMIS object record.

We have a CRNI Handle as an identifier for the object, with a value of loc.music/gottlieb.09611.
The object is a file.
It has an MD5 hash value of 36B0319....
It’s 20,800,896 bytes in size.
It’s in TIFF format, and we don’t know the version of TIFF.

## <premis:event>

-      <premis:eventIdentifier>
-           <premis:eventIdentifierType>LocalRepository</premis:eventIdentifierType>
-           <premis:eventIdentifierValue>e001</premis:eventIdentifierValue>
-      </premis:eventIdentifier>
-      <premis:eventType>ingestion</premis:eventType>
-      <premis:eventDateTime>2006-06-06T00:00:00.001</premis:eventDateTime>
-      <premis:linkingAgentIdentifier>
-           <premis:linkingAgentIdentifierType>AgentID</premis:linkingAgentIdentifierType>
-           <premis:linkingAgentIdentifierValue>na12345</premis:linkingAgentIdentifierValue>
-      </premis:linkingAgentIdentifier>
- </premis:event>
- <premis:agent>
-      <premis:agentIdentifier>
-           <premis:agentIdentifierType>AgentID</premis:agentIdentifierType>
-           <premis:agentIdentifierValue>na12345</premis:agentIdentifierValue>
-      </premis:agentIdentifier>
-      <premis:agentName>LC Repository</premis:agentName>
-      <premis:agentType>organization</premis:agentType>
- </premis:agent>

**Notes:**
Here’s an example of a PREMIS event record and a PREMIS agent record

We have a local repository identifier for the event of e001.
The event is an ingestion event where material has been brought into the repository.
The agent responsible for bringing the material into the repository has the AgentID of na12345.

We have a record for an agent with an AgentID of na12345.
The agent has the name of LC Repository, and is an organization.

## Overview of METS

- Digital Library Federation Initiative launched in 2001 as successor to _Making of America II __project_
- Goal: Create a single document format for encoding digital library objects which can fulfill roles of SIP, AIP and DIP within the OAIS reference model
- Scope limited to objects comprised of text, image, audio and video files (or combination thereof)
- _METS Maintenance Activity __at Library of Congress, including __XML Schema_

**Notes:**
One of the goals for the PREMIS framework working group was to define an implementation that would work with other forms of metadata that repositories would need to store for digital objects, including other forms of administrative metadata, as well as descriptive and structural metadata.  A key aspect of any digital library object repository is going to be a structural metadata standard that allows you to bind content, descriptive metadata, and administrative metadata such as PREMIS into a single coherent entity.  The METS standard, developed by the Digital Library Federation, was intended to provide just such a structural metadata standard.

The METS standard was developed by the Digital Library Federation as a successor effort to the Making of America II project, which sought to define a general service model for digital libraries, along with a common standard for digital library object encoding.  The intent of METS was to create a single, XML-based document format that could fulfill the roles of Submission Information Package, Archival Information Package, and Dissemination Information Package within the OAIS Reference model.

That being a rather tall order, it was decided to limit the scope of METS to digital library objects comprised of text, images, audio or video files, or some combination thereof.  METS has been publicly available for about 5 years now, and maintenance activity for the standard is on-going, and conducted under the auspices of both the METS editorial board and the Library of Congress in the US.

## METS Framework

- METS Document

- Header

- Descriptive MD

- Admin. MD

- File Section

- Link Structure

- Structural Map

- Behaviors

**Notes:**

A METS document is composed of seven major sections:
Header -- metadata about the METS file, separate from the digital library object
Descriptive Metadata Section -- one or more descriptive metadata records for the DL object, in any format.
Administrative Metadata Section: Technical, IP Rights, Analog Source and Digital Provenance Metadata for the object.
File Section -- an inventory of the content files for the DL object, including some file-specific technical metadata.
Link Structure -- structural metadata recording the existence of hyperlinks within the DL Object
Structural Map -- a hierarchical description of the DL object’s structure, either physical or logical (or both)
Behaviors -- API and location information for any software behaviors needed to work with the DL object’s content or metadata

## METS Structure

- Object modeled as tree (e.g. movie is composed of scenes, which are composed of one or more shots)
- Every node in tree structure can be associated with content files and descriptive & administrative metadata
- Every content file can be associated with descriptive & administrative metadata

**Notes:**
Structural Map -- heart and soul of a METS document, providing the grounding framework for all other linking Of content and metadata with in a METS object.

We are Aristotelian in METS: if you can’t model it using a tree, it’s obviously not worth worrying about.
So, a movie can be decomposed hierarchically as one or more scenes, including one or more shots.

Every node in a METS structural tree can be associated with both content and descriptive and administrative metadata.

Content files can be separately associated with descriptive and administrative metadata.  So, the root node in the structural Map might link to a MARC21 record describing a moview, and an MPEG2 file in the file section might link to a technical Metadata record providing information in the file format, compression ratio, bit-rate, colorspace, etc.

## METS Administrative Metadata

- 4 Types: Technical, Rights, Source Document, Digital Provenance
- Non-prescriptive/Multiple instances
- may be internal (XML or binary) or external (XLink) to METS document
- Internal XML reliant on extension schema (e.g., PREMIS) for support

**Notes:**
METS subdivides metadata into four different subtypes: technical metadata providing information about the technical characteristics of a file and its format; IP rights metadata regarding the object or its components; source metadata, being descriptive or administrative metadata about an analog source from which the digital object derives; and digital provenance metadata, which documents the life-cycle of both the content and metadata comprising the digital object.

You can use any form of administrative metadata you want, you can have as many instances as you want, and it can reside inside or outside of the METS document, in either XML or binary form.

METS does not define its own administrative metadata formats, leaving that to other standards efforts.
This means that METS is extremely reliant on other standards efforts like PREMIS to provide the additional building blocks needed to record the complete set of metadata needed for a DL object.

## METS & PREMIS

**Notes:**
Combination of METS & PREMIS is currently in use at Library of Congress, Stanford, National Library of Australia, Florida Center for Library Automation, MathArc Project (Cornell and Gottingen State & Univ. Library), with other experimental efforts underway.

This provides you with a graphical view of how the various bits and pieces of a METS object might fit together.

Orange is structural map, defining a logical structure for the object (a movie).  The root node links to a descriptive metadata record in dublin core format for the object, as well as an IP rights record for the object using the PREMIS rights schema.  One scene in the movie can be mapped to a set of MPEG2 files comprising the content, along with a subsection of a text file providing subtitling information.  Content files can be linked to multiple technical metadata records, in this case a PREMIS record for general technical metadata and a SMPTE RP210 record for video specific technical metadata, and to a variety of PREMIS records describing the file’s life-cycle.

## OAIS Information Package

**Notes:**
A few key aspects of OAIS relating to digital preservation:
Representation Information: The information that maps a Data Object into more meaningful concepts. An example is the ASCII definition that describes how a sequence of bits (i.e., a Data Object) is mapped into a symbol. 

Representation information includes:
	Structure Information: The information that imparts meaning about how other information is organized. For example, it maps bit streams to common computer types such as characters, numbers, and pixels and aggregations of those types such as character strings and arrays. 

and

	Semantic Information: The information needed to properly interpret the full meaning of various bits of structured information in the data object.  The difference between properly interpreting data as an ISO conformant date, and knowing the date represents the copyright date for the object, or knowing whether the string “Chou” is referring to the Dynasty or the pastry dough.

Preservation Description information, which includes:

Reference Information: stable identifiers for the object and its components;

Provenance Information: The information that documents the history of the Content Information. This information tells the origin or source of the Content Information, any changes that may have taken place since it was originated, and who has had custody of it since it was originated. Examples of Provenance Information are the principal investigator who recorded the data, and the information concerning its storage, handling, and migration. 

Context Information: The information .that documents the relationships of the Content Information to its environment. This includes why the Content Information was created and how it relates to other Content Information objects. 

And finally

Fixity information: the information that lets you know that the object is stable and unchanged.  

PREMIS exists to support recording representation information and preservation description information.

## On-going Issues 

- Architecting objects for performance, or the Metadata that Ate Cincinnati
- Organizing successful & complete representation networks
- Enabling trustworthy metadata
- Supporting ‘non-generic’ Event, Rights & Agent metadata
- Creating metrics & methods for evaluating digital preservation activities

**Notes:**
The combination of standards such as METS and PREMIS gives us the beginnings of a powerful framework for addressing long-term preservation of digital content, but I don’t think we’re done building that framework yet.  There are several lingering issues that we must confront before we can really claim to be doing a decent job of insuring digital information’s longevity.

## The Metadata That Ate Cincinnati

- Add a 300 page digitized book with TIFF page images, a TEI encoding and a METS wrapper to your repository:
- 302 PREMIS Object Records, 302 Other Technical Metadata Records, 1 Descriptive Metadata Record, 1 Rights Record, 1 PREMIS Event Record (Ingest), 1 PREMIS Agent Record (Ingesting Agent), 302 PREMIS Event Records (JHOVE Validation)
- Migrate TIFF to JPEG2000
- Add 300 PREMIS Event Records, 300 Additional Event Detail Records, 1 PREMIS Agent Record 300 PREMIS Object records, 300 Technical Metadata Records
- Run Fixity Check on Content Files
- Add 302 PREMIS Event Records, 1 PREMIS Agent Record
- Continue *ad infinitum....*

**Notes:**
2400 metadata records and counting.
Consider now the metadata needed to ingest a digital edition of the entire journal run of the Philosophical Transactions of the Royal Society (starting From 1655).

Librarians have lots of experience dealing with two or three metadata records per object.  We have no experience dealing with 2400 metadata records for one object.  
We do not know the specific types of user activities or relative levels of use that this metadata will have to support, because we’ve never done it before.  We *do* know there can be performance issues in parsing and manipulating extremely large, multi-schema XML documents.  
We need to consider the implications of moving from a world in which metadata is stored in relatively static databases to systems which are highly transactional In nature and think about how to best architect our objects, as well as our systems, to support the uses we deem likely to emerge both from our user communities and amongst ourselves.

## Representation Networks

- ISO/IEC 15444-1 + :2004/PDAM 1 (JPEG 2000 +Amendment 1/profiles for Dig. Cinema)
- SMPTE 384M (MXF)
- W3C XML 1.1
- SMPTE 372M
- EBU Standard N22 1997
- AES3-2003
- SMPTE 196E
- ISO/IEC 15948:2004 (PNG)

- Unicode version 4.0.01
- SMPTE 12M (auxiliary file format)
- SMPTE 336M (KLV)
- ISO 15706 (ISAN)
- SMPTE 330M-2004 (UMID)
- ITU-T Recommendation X.509
- ISO 3166 (language code)
- TIA-442 (RS-422)
- IEEE802.3

- Partial (first layer) representation
- network for Digital Cinema System Specification

**Notes:**
Above represents standards directly referenced by the Digital Cinema System specification.
It does not include a variety of other referenced materials, including RFCs for URNs, SHA1, & RSA,
CIE Colorimetry publication, etc.

Almost all of these standards reference other standards necessary to their interpretation *IN SOME INSTANCES*.  We cannot know in advance which parts of this network of standards and documentation will be needed to Interpret a particular electronic document.  

“As a practical matter, the OAIS needs to have enough Representation Information associated with the bits of the Content Data Object in the Content Information that it feels confident that the members of the Designated Community can enter the Representation Network with enough knowledge to begin accurately interpreting the Representation Information. This is a significant risk area for an OAIS, particularly for those with an expert Designated Community, because jargon and apparently widely understood terms may be short-lived. In such cases extra care needs to be exercised to ensure that the natural evolution of the Designated Community Knowledge Base does not effectively cause information loss from the Content Information. “ -- OAIS Reference Model

Various efforts trying to build representation information registries/repositories, including DCC, GDFR, PRONOM, but the appropriate extent of a representation network to maintain, the appropriate mechanisms for preserving representation information (including administrative/organizational/legal/economic) are not at all clear.

## Trustworthy Metadata

- Metadata from a known (and trusted) source
- Metadata that has not experienced unauthorized change
- Metadata that is accurate
- Metadata that is sufficient to need
- Metadata that is transparent

**Notes:**
If our users are going to believe us when we say we have preserved digital information, we have to have metadata that they consider trustworthy, which raises the very difficult question: What do we mean by ‘trustworthy’ metadata?

A known and trusted source -- which may not be known or trusted in the future.  How do we document sources?
Metadata that has not experienced unauthorized changed -- how do we demonstrate that, particular in an age where computer scientists keep inventing remarkable new ways to create hash collisions?
Metadata that is accurate -- accurate by what standard? what level of accuracy are we willing to guarantee?  What do we NEED to guarantee?
Metadata that is sufficient to need -- difficult when we cannot predict future needs, as the Apollo example demonstrates.  Apparently the need to identify a specific mission’s tapes didn’t occur to anyone.
Metadata that is transparent -- transparency implies interpretability, and interpretability implies retaining sufficient knowledge of the originating context to allow you to figure out what was meant.  How do we do that?

## Generic vs. Specific:Events, Rights & Agents

- Event Example -- Migrate SD DTV to HD DTV.  You may want to know:
- De-interlacing technique (motion-compensated or not, linear or non-linear)
- Colorspace conversion (gamma correction, luma equations for source and destination, primary chromaticities and white points for source and destination)
- Aspect ratio conversion technique
- Similarly, we may want to know more about Rights and Agents than the minimal generic information

**Notes:**
PREMIS Object provides technical metadata applicable to any digital object.
Z39.87, SMPTE RP210, AES-X098B all provide technical metadata specific to their respective formats.
While PREMIS provides generic Events, Rights & Agents metadata, scholars in the future may want more.  We need not only format specific technical metadata standards, we may also need situationally/format specific event records, agent records and rights records.  Metadata common to all formats/situations is key, but it is not the whole story.

## Evaluating Digital Preservation Programs

- What does it mean to preserve digital content?  Does the meaning of “preservation” vary with context?
- What metrics should we employ to evaluate the success of a digital preservation program?

## 謝謝!

- Jerome McDonough
- Graduation School of Library & Information Science
- University of Illinois at Urbana-Champaign
- 501 E. Daniel Street, MC-493
- Champaign, IL 61820
- jmcdonou@uiuc.edu