# Presentation: 100767

## Using METS and MODS to Create an XML Standards-based Digital Library Application

- Morgan CundiffNetwork Development andMARC Standards OfficeLibrary of Congress

**Notes:**
Out topic is...
METS and MODS to express data for digital objects
Library of Congress Presents Music, Theater, and Dance is the digital library web application
I will be talking mainly about the first part (the data) and Nate will be talking more about the
LC Presents digital library application

## Who are we?

- NDMSO Technical TeamMorgan Cundiff
- Nate Trail
- Betsy Miller
- Glenn Gardner
- Corey Keith

- LC Presents Team (Music Division)
- Karen Lund
- Pat Padua
- James Wolf
- Paul Fraunfelter
- Mike Ferrando

**Notes:**
Acknowledge colleagues who worked on the project

## XML is the lingua franca of the Web

-  Increasingly used for data exchange and messaging in the business world
-  Web pages are increasingly based on XHTML
-  Family of technologies to leverage (XML Schema, XSLT, XPath, and XQuery)
-  Software tools widely available (for storage, editing, parsing, validating, transforming and publishing XML) – much of this software is open source and free – and it is constantly and actively being improved.
-  Microsoft Office 2003 supports XML as document format (WordML and ExcelML)
-  “Web 2.0” heavily based on XML (AJAX, Semantic Web, Web Services, etc.)

**Notes:**
Background perspective: 
Using XML is the larger issue
More and more data is being stored in XML format.
When XQuery becomes a W3C recommendation, Xpath/Xquery will become the way to query XML. When that occurs there will quickly follow a wave of new products for storing and searching XML data. 

## XML

- Jane Hunter
- Working towards MetaUtopia - A Survey of Current Metadata Research

**Notes:**
Writing about metadata for Web resources, Jane Hunter made this statement back in 2003

## The Importance of Standards

- “In moving from dispersed digital collections to interoperable digital libraries, the most important activity we need to focus on is *standards*... most important is the wide variety of metadata standards [including] descriptive metadata... administrative metadata..., structural metadata, and terms and conditions metadata...”
- Howard Besser
- The Next Stage: Moving from Isolated Digital Collections to Interoperable Digital Libraries

**Notes:**
And speaking about standards (also in 2003) Howard Besser made this statement:

Howard was exactly right. We have heard various people speak against the creation of new “stovepipes”. By that they mean creating standalone applications that have no ability to communicate or “interoperate” with other applications. These are the “dispersed” collections that Besser refers to.

We also hear Dr. Billington and others speak about creating a World Digital Library. The World Digital Library is a vision for digital library interoperabilty on a very grand scale and it is very noble and powerful vision.  Again, I think Howard Besser was right when he said, the most important activity we need to focus on, to realize that vision, is metadata standards. And that is exactly what we are doing in our work to build digital library collections that are based on METS and the other members of the family of standards that have emerged from the digital library community.

## XML in the Digital Library Community

-  Family of XML data standards: METS, MODS,  MIX, PREMIS, TEI, and EAD
-  METS Implementations: LC, OCLC, RLG, California Digital Library, Harvard, Princeton, National Library of Portugal, National Library of Wales, University of Indiana, Stanford, New York University, University of Göttingen, Oxford University, etc etc...
-  METS Software Tools: Harvard METS Toolkit, Harvard DRS METS Archive Tool (Dmart) for Audio Deposit, CDL 7train METS Generation Tool, MEX Authoring Tools (Das Bundesarchiv), ContentE (Biblioteca Nacional Digital, Portugal), METS Navigator (Indiana University Digital Library Program) ResCarta Metadata Creation Tool (ResCarta Foundation) etc
-  METS listserv: 530 subscribers

**Notes:**
XML in our “domain” the digital library community
Since about 2002 there has been steady and increasing “uptake” of a family of XML standards for digital library metedata
NDMSO has been very involved in this development- maintenance agency for METS, MODS, MIX, MARCXML, PREMIS, and EAD

## XML Standards at LC: A little historical perspective ...

-  1995 – first American Memory web collections released (not XML-based)
-  1998 – XML 1.0 becomes a W3C Recommendation
-  2002 – METS and MODS released
-  2002 – Digital Audio-Visual Preservation Prototyping Project (led by Carl Fleischhauer, first use of METS, MODS, and MIX at LC)
-  2003 – “Patriotic Melodies” (first use of METS and MODS in production at LC)
-  2003 – Veterans History Project released
-  2004 – I Hear America Singing released (since renamed to LC Presents)
-  2004 – Justice Blackmun Papers collection released
-  2006 – National Digital Newspaper Project (LC and partners, first use of METS, MODS, MIX, PREMIS) as repository submission package at LC- September launch)
-  2006 – Ser2Dig (Digital Serials workgroup, METS for multi-volume monographs)
-  2006 – Draft METS profile for “article-level” historical newspapers

**Notes:**
Historical perspective at LC

## What is METS?(a quick primer)

- METS is an XML Schema designed for the purpose of creating XML document instances that express the hierarchical structure of digital library objects, the names and locations of the files that comprise those objects, and the associated metadata. METS can, therefore, be used as a tool for modeling real world objects, such as particular document types.

**Notes:**
METS is used to express the structure of the document and the associated metadata.

## What is MODS?(a quick primer)

- MODS is an XML Schema designed for expressing bibliographic data. MODS can be seen as an alternative to the MARC format. It is especially useful for XML-based digital library projects. MODS can be used as an “extension schema” to METS.
- Note to catalogers: MODS does not make you obsolete. The same knowledge and skills (mastery of cataloging rules and controlled vocabularies, subject knowledge, etc) are still necessary. It is just a different syntax (i.e. different from MARC) for making bibliographic data machine-readable.

**Notes:**
I do not know how fast the ILS world will move to XML though I think it is inevitable. In the meantime there continues to be an unfortunate divide between digital library projects and the cataloging operation here at LC. We are hoping to begin having music catalogers assist with LC Presents in the near future.

## 	What are the 7 Sections of a METS Document?

**<mets>**
**  ****<metsHdr/>**
**  ****<dmdSec/>**
**  ****<amdSec/>**
**  ****<fileSec/>**
**  ****<structMap/>**
**  ****<structLink/>**
**  ****<behaviorSec/>**
**</mets>**

## 	The Descriptive Metadata Section with mdWrap

**<mets>**
**  ****<dmdSec>**
**    ****<mdWrap>**
**      ****<xmlData>**
**        ****<!-- ****insert data from different namespace here**** -->**
**      ****</xmlData>**
**    ****</mdWrap>**
**  ****</dmdSec>****	****	****	**
**  ****<fileSec></fileSec>**
**  ****<structMap></structMap>**
**</mets>**

## The Descriptive Metadata Section with MODS as extension schema

**<mets:mets>**
**  ****<mets:dmdSec>**
**    ****<mets:mdWrap>**
**      ****<mets:xmlData>**
**        ****<****mods:mods****></****mods:mods****>**
**      ****</mets:xmlData>**
**    ****</mets:mdWrap>**
**  ****</mets:dmdSec>****	****	****	**
**  ****<mets:fileSec></mets:fileSec>**
**  ****<mets:structMap></mets:structMap>**
**</mets:mets>**

## The Descriptive Metadata Section with MODS and relatedItem elements

**<mets:mets>**
**  ****<mets:dmdSec>**
**    ****<mets:mdWrap>**
**      ****<mets:xmlData>**
**        ****<****mods:mods****>**
**           ****<****mods:relatedItem type=“constituent”****>**
**             ****<****mods:relatedItem type=“constituent”****></****mods:relatedItem****>**
**           ****</****mods:relatedItem****>**
**        ****</****mods:mods****>**
**      ****</mets:xmlData>**
**    ****</mets:mdWrap>**
**  ****</mets:dmdSec>****	****	****	**
**  ****<mets:fileSec></mets:fileSec>**
**  ****<mets:structMap></mets:structMap>**
**</mets:mets>**

## METS document with two hierarchies (logical and physical)

**<mets:mets>**
**  ****<mets:dmdSec>**
**    ****<mets:mdWrap>**
**      ****<mets:xmlData>**
**        ****<****mods:mods****>**
**           ****<****mods:relatedItem****>**
**             ****<****mods:relatedItem****></****mods:relatedItem****>**
**           ****</****mods:relatedItem****>**
**        ****</****mods:mods****>**
**      ****</mets:xmlData>**
**    ****</mets:mdWrap>**
**  ****</mets:dmdSec>****	****	****	**
**  ****<mets:fileSec></mets:fileSec>**
**  ****<mets:structMap>**
**    ****<****mets:div****>**
**      ****<****mets:div****></****mets:div****>**
**    ****</****mets:div****>**
**  ****</mets:structMap>**
**</mets:mets>**

## MODS relatedItem type=“constituent” element

- Child element to MODS
- relatedItem element has same content model as mods (titleInfo, name, subject, physicalDescription, note, etc)
- The relatedItem element makes it possible to create very rich analytic descriptions for contained works within a MODS records
- relatedItem element is repeatable and it can be nested recursively (thus making it possible to build a hierarchical tree structure)
- relatedItem elements make it possible to associate descriptive data with any structural element.

## 	<mods:mods>

**	****  ****<mods:titleInfo>**
**	****    ****<mods:title>Bernstein conducts Beethoven and Mozart</mods:title>**
**	****  ****</mods:titleInfo>**
**	****  ****<mods:name>**
**	****    ****<mods:namePart>Bernstein, Leonard</mods:namePart>**
**	****  ****</mods:name>**
**	****  ****<mods:relatedItem type="constituent">**
**	****    ****<mods:titleInfo>**
**	****      ****<mods:title>****Symphony No. 5****</mods:title>**
**	****    ****</mods:titleInfo>**
**	****    ****<mods:name>**
**	****      ****<mods:namePart>****Beethoven, Ludwig van****</mods:namePart>**
**	****    ****</mods:name>**
**	****    ****<mods:relatedItem  type="constituent">**
**	****      ****<mods:titleInfo>**
**	****        ****<mods:partName>****Allegro con moto****</mods:partName>**
**	****      ****</mods:titleInfo>**
**	****    ****</mods:relatedItem>**
**	****    ****<mods:relatedItem type="constituent">**
**	****      ****<mods:titleInfo>**
**	****        ****<mods:partName>****Adagio****</mods:partName>**
**	****      ****</mods:titleInfo>**
**	****    ****</mods:relatedItem>**
**	****  ****</mods:relatedItem>**
**	****</mods:mods>**

**Notes:**
Here is a simple illustration of a logical hierarchy created with relatedItems.

## Linking in METS Documents(XML ID/IDREF links)

- DescMD
- 	mods
- 		relatedItem
- 		relatedItem

- AdminMD
- 	techMD
- 	sourceMD
- 	digiprovMD
- 	rightsMD

- fileGrp
- 	file
- 	file

- StructMap
- 	div
- 	   div
- 	      fptr
- 		div
- 		   fptr

## DescMD
	mods
		relatedItem
		relatedItem

- 	mods
- 		relatedItem
- 		relatedItem

- AdminMD
- 	techMD
- 	sourceMD
- 	digiprovMD
- 	rightsMD

- fileGrp
- 	file
- 	file

- StructMap
- 	div
- 	   div
- 	      fptr
- 		div
- 		   fptr

- Linking in METS Documents(XML ID/IDREF links)

## DescMD
	mods
		relatedItem
		relatedItem

- 	mods
- 		relatedItem
- 		relatedItem

- AdminMD
- 	techMD (mix)
- 	sourceMD
- 	digiprovMD
- 	rightsMD

- fileGrp
- 	file
- 	file

- StructMap
- 	div
- 	   div
- 	      fptr
- 		div
- 		   fptr

- Linking in METS Documents(XML ID/IDREF links)

## DescMD
	mods
		relatedItem
		relatedItem

- 	mods
- 		relatedItem
- 		relatedItem

- AdminMD
- 	techMD (mix)
- 	sourceMD
- 	digiprovMD
- 	rightsMD

- fileGrp
- 	file
- 	file

- StructMap
- 	div
- 	   div
- 	      fptr
- 		div
- 		   fptr

- Linking in METS Documents(XML ID/IDREF links)

## DescMD
	mods
		relatedItem
		relatedItem

- 	mods
- 		relatedItem
- 		relatedItem

- AdminMD
- 	techMD (mix)
- 	sourceMD
- 	digiprovMD
- 	rightsMD

- fileGrp
- 	file
- 	file

- StructMap
- 	div
- 	   div
- 	      fptr
- 		div
- 		   fptr

- Linking in METS Documents(XML ID/IDREF links)

## DescMD
	mods
		relatedItem
		relatedItem

- 	mods
- 		relatedItem
- 		relatedItem

- AdminMD
- 	techMD (mix)
- 	sourceMD
- 	digiprovMD
- 	rightsMD

- fileGrp
- 	file
- 	file

- StructMap
- 	div
- 	   div
- 	      fptr
- 		div
- 		   fptr

- Linking in METS Documents(XML ID/IDREF links)

## What is a METS Profile?

- “METS Profiles are intended to describe a class of METS documents in sufficient detail to provide both document authors and programmers the guidance they require to create and process METS documents conforming with a particular profile.”
- A profile is expressed as an XML document. There is a schema for this purpose. The profile expresses the requirements that a METS document must satisfy.
- A sufficiently explicit METS Profile may be considered a “data standard”.
- Note: A METS Profile is a human-readable prose document and is not intended to be “machine actionable”.

**Notes:**
You might think of the METS Profile document as something analogous to the LCRI

## Slide 24

## METS Profiles in use in LC Presents

-  Sheet Music
-  Musical Score (may be a score, score and parts, or a set of parts only)
-  Print Material (books, pamphlets, etc)
-  Music Manuscript (score or sketches)
-  Recorded Event (audio or video)
-  PDF Document
-  Bibliographic Record
-  Photograph
-  Compact Disc
-  Collection

## Multiple Inputs to Common Data Format

- New digital items

- Legacy database

- Harvest of American Memory Collection

- Profile-based METS/MODS object (A common data set for searching and display)

- Web publication (LC Presents)

**Notes:**
One of the realities we face is that we have valuable data especially bib data in many different formats and systems. We also have bib data that is in no system at all- it is in an old procite database or a word perfect or an even older print document. 

## Example 1: New digital objectMETS musical score profile

- _[Library of Congress march / John Philip Sousa [musical score and parts]](http://memory.loc.gov/cocoon/ihas/loc.natlib.ihas.100010802/default.html)_

- Example of “routine” METS-making

## Example 2: New digital objectMETS Recorded Event Profile

- _[Juilliard String Quartet / Juilliard String Quartet [sound recording]](http://memory.loc.gov/cocoon/ihas/loc.natlib.ihas.200003878/default.html)_

- Example of “routine” METS-making

## Example 3: from “random” databaseMETS Bibliographic Record Object

- _[DUKE ELLINGTON AND HIS ORCHESTRA (1962) [motion picture]](http://memory.loc.gov/cocoon/ihas/loc.natlib.ihas.200015087/default.html)__[ ](http://memory.loc.gov/cocoon/ihas/loc.natlib.ihas.200015087/default.html)_

- Example of “database of bib data” sourceConversion from Filemaker Pro databasetoFilemaker XML dump (1 XML file)XSLT to14,000 METS/MODS recordsamd XSL to PDF (1 file)

## Example 4: American Memory CollectionMETS Photograph ObjectHarvest of William P. Gottlieb Collection

- _[Portrait of Louis Armstrong, Carnegie Hall, New York, N.Y., ca. Apr. 1947] / William P. Gottlieb [photograph]](http://memory.loc.gov/cocoon/ihas/loc.natlib.gottlieb.00151/default.html)__[ ](http://memory.loc.gov/cocoon/ihas/loc.natlib.gottlieb.00151/default.html)_

- File of 1600 MARC recordsmarc4j toXML modsCollection (1 file)XSLT toMETS photograph profile (1600 files)

##      <mets:div TYPE="photo:version" “ DMDID=“ver02“>

**  ****<mods:titleInfo>**
**    ****<mods:title>****Original**** ****Work (vesion 1)****</mods:title>**
**  ****</mods:titleInfo>**
**<****mods:relatedItem type=“otherVersion****" ID=“ver02">**
**    ****<mods:titleInfo>**
**      ****<mods:title>****Derivative ****Work 1****</mods:title>**
**    ****</mods:titleInfo>**
**</mods:relatedItem>**
**<****mods:relatedItem type=“otherVersion****" ID=“ver03">**
**    ****<mods:titleInfo>**
**      ****<mods:title>****Derivative ****Work 2****</mods:title>**
**    ****</mods:titleInfo>**
**</mods:relatedItem>**
**</mods:mods>**

- Logical (MODS)

- Physical (structMap)

-  **<mets:structMap>**
**    ****<mets:div TYPE="photo:photoObject" DMDID="MODS1">**
**      ****<mets:div TYPE="****photo:version****“DMDID="ver01”** **>**
**        ****<mets:div TYPE="photo:image">**
**          ****<mets:fptr FILEID="FN10081"/>**
**        ****</mets:div>**
**      ****</mets:div>**
**     ****<mets:div TYPE="****photo:version****"** **“ DMDID=“ver02**“**>**
**      ****<mets:div TYPE="photo:image">**
**       ****<mets:fptr FILEID="FN10090"/>**
**      ****</mets:div>**
**      ****<mets:div TYPE="****photo:version****"** **“DMDID="ver03**”**>**
**        ****<mets:div TYPE="photo:image">**
**          ****<mets:fptr FILEID="FN1009F"/>**
**        ****</mets:div>**
**       ****</mets:div>**
**      ****</mets:div>**
**    ****</mets:div>**
**  ****</mets:structMap>**

- mods element and relatedItem type =“otherVersion” elements
- Sequence of 3 nodes

- div TYPE=“photo:version” elements
- Corresponding sequence of 3 nodes linked to logical sequence by ID/IDREF

## METS Profiles makes possible 3 levels of validation for  METS objects

-  Valid XML (well-formed)
-  Valid METS/MODS (XML Schema)
-  Valid METS Profile

## Aggregation Example 1METS Collection Object

- _[http://lcweb2.loc.](http://lcweb2.loc.gov/cocoon/ihas/loc.natlib.ihas.200031146/default.html)__[gov](http://lcweb2.loc.gov/cocoon/ihas/loc.natlib.ihas.200031146/default.html)__[/cocoon/](http://lcweb2.loc.gov/cocoon/ihas/loc.natlib.ihas.200031146/default.html)__[ihas](http://lcweb2.loc.gov/cocoon/ihas/loc.natlib.ihas.200031146/default.html)__[/loc.](http://lcweb2.loc.gov/cocoon/ihas/loc.natlib.ihas.200031146/default.html)__[natlib](http://lcweb2.loc.gov/cocoon/ihas/loc.natlib.ihas.200031146/default.html)__[.](http://lcweb2.loc.gov/cocoon/ihas/loc.natlib.ihas.200031146/default.html)__[ihas](http://lcweb2.loc.gov/cocoon/ihas/loc.natlib.ihas.200031146/default.html)__[.200031146/default.html](http://lcweb2.loc.gov/cocoon/ihas/loc.natlib.ihas.200031146/default.html)_

## Aggregation Example 2MODS relatedItem type=“host”

- _[http://lcweb2.loc.](http://lcweb2.loc.gov/cocoon/ihas/search?query=%2BmemberOf:%22Baseball%20sheet%20music%20collection%22&start=0&view=thumbnail)__[gov](http://lcweb2.loc.gov/cocoon/ihas/search?query=%2BmemberOf:%22Baseball%20sheet%20music%20collection%22&start=0&view=thumbnail)__[/cocoon/](http://lcweb2.loc.gov/cocoon/ihas/search?query=%2BmemberOf:%22Baseball%20sheet%20music%20collection%22&start=0&view=thumbnail)__[ihas](http://lcweb2.loc.gov/cocoon/ihas/search?query=%2BmemberOf:%22Baseball%20sheet%20music%20collection%22&start=0&view=thumbnail)__[/search?query=%2BmemberOf:"Baseball%20sheet%20music%20collection"&start=0&view=thumbnail](http://lcweb2.loc.gov/cocoon/ihas/search?query=%2BmemberOf:%22Baseball%20sheet%20music%20collection%22&start=0&view=thumbnail)_

## Aggregation Example 3:“See also”MODS relatedItem (no type)

- _[http://memory.loc.](http://memory.loc.gov/cocoon/ihas/loc.natlib.ihas.200003800/default.html)__[gov](http://memory.loc.gov/cocoon/ihas/loc.natlib.ihas.200003800/default.html)__[/cocoon/](http://memory.loc.gov/cocoon/ihas/loc.natlib.ihas.200003800/default.html)__[ihas](http://memory.loc.gov/cocoon/ihas/loc.natlib.ihas.200003800/default.html)__[/loc.](http://memory.loc.gov/cocoon/ihas/loc.natlib.ihas.200003800/default.html)__[natlib](http://memory.loc.gov/cocoon/ihas/loc.natlib.ihas.200003800/default.html)__[.](http://memory.loc.gov/cocoon/ihas/loc.natlib.ihas.200003800/default.html)__[ihas](http://memory.loc.gov/cocoon/ihas/loc.natlib.ihas.200003800/default.html)__[.200003800/default.html](http://memory.loc.gov/cocoon/ihas/loc.natlib.ihas.200003800/default.html)_

## Administrative Metadata ExampleUse of PREMIS and MIXfor digital images

- _[louis](file:///louis.xml)__[.xml](file:///louis.xml)_

## Slide 37

## METS and MODS software tools (open source XML toolkit)

- Emacs (text editor) - edit MODS- nxml-mode (Emacs plug-in for schema-aware XML editing)- XML Schemas for METS, MODS, MIX, PREMIS- cygwin – bash shell command line and tools- Saxon (XSLT transformations)- Xerces (XML validation)- mysql-jdbc-connector (connect to natlib mySQL database)- SRU (retrieve records from ILS)- Cocoon facilities to retrieve and load records, retrieve xml version of a file system, etc- Ant – used to automate all of the above tasks and create pipelines of multiple tasks and run from Emacs

## Parting thoughts: Advantages of METS/MODS-based approach

-  Ability to model complex objects
-  Easy to change, extend (both the data and the application)
-  Use modern, non-proprietary software tools
-  Leverage use of XSLT for legacy data conversion, for batch METS creation and editing, and for Web displays and behaviors
-  Common syntax: XML for data creation/editing/storage/searching
-  Output to HTML, PDF for display
-  Easy to edit single records or selected batches of records
-  Ability to validate data
-  Ability to aggregate disparate data sources
-  Improved ability to manage data and publish data now and ...
-  Well positioned for Future: new web application (Web 2.0), repository submission, cooperative project (test interoperability), provide METS for OAI harvesting ...

## Slide 40

## Slide 41