# Presentation: 520204

## OCLC’s Digital Archive – Disseminating with METS

**Jay Goodkin**
**Software Engineer**
**Digital Collection and Preservation Services**

## Today’s Presentation		

- OCLC Digital Archive Overview
- Dissemination Workflow
- OCLC’s METS Implementation for Dissemination

**Notes:**
To start with, I will provide a quick overview of the Digital Archive and its features. 

Next, I will provide a short walkthrough of the Dissemination Workflow of the Digital Archive.

Finally, I will concentrate on the structure of our Dissemination Package structure and the specifics of our METS implementation.

## OCLC Digital Archive - Overview

- Functions include:
- Harvest
- Ingest
- Content Group Management
- Rights Group Management
- Viewing
**Dissemination**
- Reports
- Periodic Audits of Objects in the Archive
- Frequent Backups and Disaster Prevention
- Dissemination Information Packages based on METS schema

**Notes:**
The primary focus of the Digital Archive (currently) is capturing web-based documents, storing them, and preserving them for the long term.

The current customer base of the Digital Archive consists mainly of State Libraries and Government Agencies. For example, The State Library of Connecticut and GPO, the Government Printing Office.

The Digital Archive supports Metadata Creation for each Digital Object and Content Management.

It Provides Discovery of archived objects in a variety of ways, including OCLC’s Connexion and FirstSearch services and local Catalogs. 

As you can see, the functions include Harvest (process of capturing document from web), Ingest, Content Group Management, Rights Group Management, Viewing, Dissemination, Reports and Periodic Audits and Backup/Disater Prevention Services.

Our Dissemination Packages are, of course, based on METS.

## Digital Archive Services

**Notes:**
The Digital Archive Services are shown here. As mentioned before, most of these services are made available through OCLC’s Connexion interface.

In addition to the Connexion interface, users with Admin privileges will access the Digital Archive Admin service directly. The Admin Service allows users to organize objects into Content Groups, create and assign Rights Groups to objects and create Rights Statements.

System Processes include an Auditor which runs period checks on the integrity of objects in the archive. We are currently working on an Annotator which extracts technical metadata from images and stores that metadata.

## Discovery Services

**Notes:**
One an object has been ingested into the Digital Archive, there are a variety of ways for users to discover them. 

Since each Digital Object in the Archive also has an associated Bibliographic Record in WorldCat, any means of searching WorldCat can be used to discover Digital Archive content. OCLC’s FirstSearch and Local OPACs fall into this category.

The bib record associated with a Digital Archive Object will contain a link to that object in the Digital Archive repository.

Links to Digital Archive objects can also be embedded on web pages.

## Digital Archive Record

- Contains the Metadata for each Digital Object in the Archive
- Extension of Dublin Core
- Entered and Maintained using OCLC’s Connexion Service
- Starting Point for Most Functions in the Archive
- Please see _[http://www.](http://www.oclc.org/support/documentation/digitalarchive/da_metadata_elements/)__[oclc](http://www.oclc.org/support/documentation/digitalarchive/da_metadata_elements/)__[.org/support/documentation/](http://www.oclc.org/support/documentation/digitalarchive/da_metadata_elements/)__[digitalarchive](http://www.oclc.org/support/documentation/digitalarchive/da_metadata_elements/)__[/](http://www.oclc.org/support/documentation/digitalarchive/da_metadata_elements/)__[da](http://www.oclc.org/support/documentation/digitalarchive/da_metadata_elements/)__[_metadata_elements/](http://www.oclc.org/support/documentation/digitalarchive/da_metadata_elements/)_ for a detailed description of our metadata elements

**Notes:**
The Digital Archive Record contains mimimal descriptive and full preservation metadata about the Digital Object. These records are only editable by the owning institution of a Digital Object. In that sense, this record is owned along with the Digital Object.

The descriptive potion of this record may be crosswalked from a bib record in WorldCat. This crosswalk also sets up the link between the Bib record and the Digital Archive record.

Most of the archive functionality is launched from the Digital Archive record, including harvest, ingest and dissemination.

## DA Record in Connexion

**Notes:**
This is a simple Screenshot of what a typical Digital Archive Record will look like.

Our metadata set was created by looking at a variety of existing metadata standards including NLA’s Pandora archive and CEDARS. In addition, internal OCLC Research contributed to this data set.

OCLC is currently participating in the PREMIS (Preservation Metadata Implementation Strategies). So our metadata will likely evolve along with any preservation metadata standards.

## Disseminating an Object

**Notes:**
Like I mentioned before, disseminating a Digital Object starts with a Digital Archive Record. 

As you can see, there are several Functions listed in the Drop Down. Depending on where the object is in the archival process, different functions will appear in the list. For example, this object has already been harvested from the web and ingested into the archive, so those functions do not appear here.

Once the Dissemination action is selected, a Dissemination Options Page will be presented.

## Dissemination Options

**Original Links** – Disseminate Original HTML Pages as Captured from the Web
**Relative Links** – Disseminate a Locally Viewable Object

**Notes:**
This Page Provides a Brief overview of the Object as a double-check.

In addition it shows the Logical Object Size and Number of Files.

The Link Type Option is interesting because it allows the user to specify whether to Disseminate the Pages in their Original Form as captured from the Web, or in a Locally Viewable form.

With Original Links, web pages will be disseminated exactly as they were captured from the web – with links unmodified.

With Relative Links, locally viewable derivatives of the Original Pages will be add to the DIP such that when Files are extracted, they can be viewed properly with a file browser.

Once the Disseminate button is clicked, the Dissemination process will start.

## Dissemination Confirmation

**Notes:**
This page indicates to the User that the Dissemination Process has started.

The Dissemination Process is asynchronous, and the user can periodically check on its progress in Connexion.

In the meantime, the user can continue working on other tasks in the Digital Archive or in the Cataloging System.

## Reports

- Each Object has a List of Associated Reports
- Harvest, Ingest and Disseminate Reports shown

**Notes:**
Each Dissemination will result in the creation of the Dissemination Package itself and a Dissemination Report.

All Reports for a Specific Object will show on this screen.

## Dissemination Report

**Notes:**
This is an example of the Dissemination Report. 

The Header contains familiar information from the Object that was Disseminated.

Each File included in the Dissemination Package is listed in the Report. An expand/contract button is provided to display or hide the detailed information about each file.

## Dissemination Manager

**Notes:**
All of the current DIPs created by an Institution will be available in the Dissemination Manager, with the most recent sorting to the top.

Currently, the Digital Archive packages each DIP as a ZIP file. 

Clicking the hyperlink will allow the DIP to be delivered to the local machine via HTTP. We decided to use HTTP to start with for its simplicity. We will likely add FTP access as the demand for it arises.

## Inside OCLC’s DIP

- Supports Multiple Objects
- Contains all Content Files for each Object
- Contains one METS Manifest
- Contains all Object-Level METS Documents

**Notes:**
Currently, the Connexion interface allows only one object to be disseminated at a time, but OCLC’s DIP structure was designed to handle multiple objects. 

We have an internal interface to create multiple-object DIPs, which I will show an example of later.

Each DIP will contain all of the Content Files for each Digital Object. All content and metadata for a Digital Object in a DIP will be organized into a folder for that Object. 

Also, each DIP will contain a Manifest File, encoded in METS.

In addition, each Object will contain a METS descriptor file.

## WinZIP View of the DIP

**Notes:**
This is a WinZIP view of the DIP created in our earlier example.

The two METS files mentioned earlier are highlighted. The mets_manifest will always appear in the root, and contains an inventory of all Digital Objects included in the DIP.

Each object-level mets.xml file will appear in the appropriate object folder within the ZIP.

## METS Manifest

- Uses **mptr** to point to all Object-Level METS Documents included in the DIP

**Notes:**
As you can see, the METS manifest is a very simple document that points to all of the Object-Level METS documents included in the DIP. It uses the mptr structure provided by METS.

## OCLC’s METS Profile

- Header - No extension
- Descriptive Metadata Section - OCLC descriptive schema _[http://](http://digitalarchive.oclc.org/schemas/oclc_dm.xsd)__[digitalarchive](http://digitalarchive.oclc.org/schemas/oclc_dm.xsd)__[.](http://digitalarchive.oclc.org/schemas/oclc_dm.xsd)__[oclc](http://digitalarchive.oclc.org/schemas/oclc_dm.xsd)__[.org/schemas/](http://digitalarchive.oclc.org/schemas/oclc_dm.xsd)__[oclc](http://digitalarchive.oclc.org/schemas/oclc_dm.xsd)__[_dm.](http://digitalarchive.oclc.org/schemas/oclc_dm.xsd)__[xsd](http://digitalarchive.oclc.org/schemas/oclc_dm.xsd)_
- File Section - No extension
- Structural Map Section - No extension
- Behavior Section - No extension

**Notes:**
In the next two slides, I’ll quickly cover the OCLC METS Profile for Dissemination.

We do have an OCLC-specific extension schema for Descriptive Metadata that encompasses the metadata on the DA Record shown earlier. 

We have no extensions for the  Header, File Section, Struct Map or Behavior Sections.

## METS Profile Continued

- Administrative Metadata Section –
- 	MIX schema _[http://www.loc.](http://www.loc.gov/standards/mix/mix.xsd)__[gov](http://www.loc.gov/standards/mix/mix.xsd)__[/standards/mix/mix.](http://www.loc.gov/standards/mix/mix.xsd)__[xsd](http://www.loc.gov/standards/mix/mix.xsd)_
- 	textMD schema _[http://](http://dlib.nyu.edu/METS/textmd.xsd)__[dlib](http://dlib.nyu.edu/METS/textmd.xsd)__[.](http://dlib.nyu.edu/METS/textmd.xsd)__[nyu](http://dlib.nyu.edu/METS/textmd.xsd)__[.](http://dlib.nyu.edu/METS/textmd.xsd)__[edu](http://dlib.nyu.edu/METS/textmd.xsd)__[/METS/](http://dlib.nyu.edu/METS/textmd.xsd)__[textmd](http://dlib.nyu.edu/METS/textmd.xsd)__[.](http://dlib.nyu.edu/METS/textmd.xsd)__[xsd](http://dlib.nyu.edu/METS/textmd.xsd)_
- 	OCLC provenance schema _[http://](http://digitalarchive.oclc.org/schemas/oclc_prov.xsd)__[digitalarchive](http://digitalarchive.oclc.org/schemas/oclc_prov.xsd)__[.](http://digitalarchive.oclc.org/schemas/oclc_prov.xsd)__[oclc](http://digitalarchive.oclc.org/schemas/oclc_prov.xsd)__[.org/schemas/](http://digitalarchive.oclc.org/schemas/oclc_prov.xsd)__[oclc](http://digitalarchive.oclc.org/schemas/oclc_prov.xsd)__[_](http://digitalarchive.oclc.org/schemas/oclc_prov.xsd)__[prov](http://digitalarchive.oclc.org/schemas/oclc_prov.xsd)__[.](http://digitalarchive.oclc.org/schemas/oclc_prov.xsd)__[xsd](http://digitalarchive.oclc.org/schemas/oclc_prov.xsd)_

**Notes:**
For Administrative Metadata, we use the MIX metadata schema for image Technical Metadata.

We have implemented the TextMD standard for Text Documents.

Also, we have created an OCLC-specific extension for handling Provenance Metadata – which in our case consists of Fixity and Virus Detection Metadata. 

## METS Header

- References OCLC Extensions
- References Industry Standard Extensions – e.g. Mix

**Notes:**
This is the METS header for the previous Dissemination. 

You can see from the highlighted text, that we have added some OCLC-specific Metadata extensions, as well as an industry standard extension, which is the Mix Image Metadata standard.

We created the oclcprov (Provenance) schema to acommodate metadata for our Fixity Checksums and Virus Checking statuses.

In addition, we include Mix metadata for images, when available.

## METS File Group

- Lists all Content Files for a given Object
- Points to Technical/Provenance Metadata for each File

**Notes:**
This is a Screenshot of the File Group portion of our METS document for the object previously disseminated.

You might notice that each file disseminated references an ID for a Provenance Metadata section and a Technical Metadata Section. I will show examples of these shortly.

Also, a link to the corresponding content file within the DIP package is included for each file using the Flocat subelement. This is only visible in the first expanded file element. We use “FILE” URLs to reference the content.

## TechMD Mix Example

**Notes:**
As mentioned before, each file references an ID to a Technical Metadata section. In this example, we show a technical metadata section for an image, encoded in the Mix standard.

## ProvMD Mix Example

**Notes:**
Similarly to the previous example, you can see that each file also has an OCLC-devised Provenance Metadata Section. Here you can see the metadata relating to Fixity and Virus Detection.

All content in the archive is checked for viruses at Harvest time. In addition, periodic Fixity and Virus checks are performed periodically after an object has been ingested into the archive. This is performed by the Auditor service mentioned before.

## StructMap Example

**Notes:**
This printscreen illustrates the Struct Map section of an OCLC Web Document.

This is the part of the METS creation that we struggled with the most. Trying to fit the complex structure of linked documents into the structMap has proven to be very challenging.

Currently, what we have done is list each of the HTML files with a separate structMap.
Then all of files that make up that page: The Page Itself and its Dependencies (I.e. Images, Stylesheets, Javascript files) are listed next.

Then the structLink section shows the pages that link together.

One idea would be to separate the main page, its dependencies, and its links, but we haven’t had any pressure to change this structure as of yet.

## StructMap & Web Documents

- Fitting Web Documents into **div** Structure
- Issues with representing Links and Offsets
**structLink** shows which Pages link together but not where the Links occur
- Considered using the **<par><area>** structure

**Notes:**
As mentioned previously, we had difficulty trying to fit our web document struture into the METS structMap.

In the Digital Archive, we maintain metadata about each HTML file, that contains a list of links, offsets and the ID for the file that each link represents.

The structLink allows us to show which pages link together, but not where on a given page each link occurs. 

The likely solution would be to have a standardized METS profile for web documents with an extension schema that handles this complex metadata.

## Multi-Object DIP

**Notes:**
This is an example of a DIP that contains two Digital Objects.

Again, you can see the METS manifest, and two Object-Level METS documents.

You can also see how the files are segregated into object folders.

## Multi-Object Manifest

**Notes:**
This is an example of a METS manifest for a DIP that contains two objects.

As you can see, there are two div sections with mptr references to each Object-Level METS document.

## Issues with Dissemination

- Interoperability of OCLC’s DIP
- Issues with encoding large Digital Objects into a METS package
- Standardization of Extension Schemas Web Documents

**Notes:**
When we first decided to develop a DIP, the idea was to try to create something that was as standards-based as possible.

Our use of METS was the first step in that direction. Now what we want to do now at OCLC is continue to evolve our DIP to the point where it is interoperable with other archives- an industry standard if you will.

We decided not to use METS for packaging the content itself, because we may eventually have VERY large Digital Objects, and parsing the METS XML would be too cumbersome. For example, one of our test batchloads had a digitized book that was 40GB in size for a single Digital Object.

So what we are interested in for the future is the development of a standardized METS profile for Web Documents, or an extension schema that accommodates the complex metadata for linked documents.

## Future Plans

- Use METS to Integrate with other Digital Repositories
- CONTENTdm
- Olive
- DSpace
- Etc.
- Redesign Digital Archive Storage Layer based on METS

**Notes:**
Our future plans include the ability to integrate with other Repositories using our DIP. OCLC currently has relationships with Olive and CONTENTdm, so those products would be a logical starting point.

Because of DSpace’s popularity, that would be another good possibility in the future.

Another idea in the works is that the next generation architecture for the Digital Archive would be completely file-based at the core. And we would use METS as the primary storage format for metadata. METS provides good mechanisms for linking to other metadata documents so we could make good use of that.

We are currently in the process of an Annotator Project that is based on this model.

## Contacts at OCLC     

- Please visit our Web site at:
- 	http://www.oclc.org/digitalarchive/
- Contacts:
- 	Pam Kircher – Product Manager
- 		_[kircherp@oclc.org](mailto:kircherp@oclc.org)_
- 	Shweta Rani – Software Developer
- 		_[ranis@oclc.org](mailto:ranis@oclc.org)_
- 	Jay Goodkin – Software Developer
- 		_[goodkinj@oclc.org](mailto:goodkinj@oclc.org)_

## Questions     

- Please see our Tutorial at:
- 	_[http://www.oclc.org/support/training/digitalarchive/](http://www.oclc.org/support/training/digitalarchive/)_
- Questions?