# Presentation: 164760

## How to Build, Display and Find METS Objects

**Nate Trail**
**Digital Project Coordinator**
**Network Development and MARC Standards Office**
**Library of Congress**
**ALA - June 25, 2007**

**Notes:**

Morgan told you about the standard data structures we use.  I’m going to tell you about 
how we build those structures from existing data and objects, 
And how we use them to search for and display the resulting digital objects

## Outline of Topics

**Outline of Topics**

**A little about the technology **
**How we build and display digital objects**
**Searching **
**Other Tools **
**Future Directions**
**Conclusions**

**Notes:**
In the context of “Library of Congress Presents and Veterans History Project:
The Technology we use
How we use the technology
Searching
Web tools in building and maintaining the application
Future Directions
Conclusions about our standards prototyping

## Technologies Used

** ****XML documents**
**METS Files **
**MODS Files**
** ****XSLT processes **
- Transformation
- Display
** ****MySQL Database**
** ****Cocoon framework**
-  And a little Java

**Notes:**
XML is data marked up according to a schema, laid out with tags of elements and attributes (modifier of the elements) and data, all self defined.

METS, MODS, and HTML can be examples of XML.  We currently de

XSLT is also XML, but it is a set of processing instructions that transform an xml document from one format to another, or display it in a browser 

Cocoon is a web processing framework that allows us to work with URLs on the web so its accessible anywhere  

## XSLT Transformation

**XSLT Transformation**

- MODS XML for Beaux Arts Trio recording

- Stylesheet “transforms” it into a simpler structure of elements, labels and values.

**Notes:**
A little about XSLT.  

As a transformation tool, we use it to convert a mods record, with it’s layers and complexity, to a flattened structure that’s easier to display: elements with labels and values.

## XSLT for Display

**XSLT for Display**

- METS XML for Beaux Arts Trio recording

- XSLT “transforms” it into an HTML or XHTML display for the screen

**Notes:**

XSLT for display is really also just a transformation because HTML is a standardized xml structure that a browser reads and interprets.

## Web Framework Concepts

**User input via URL**

**Create small programs for easy-to-modify parts**
**Separate Data from Action and Design**

**Notes:**
Cocoon is the framework that allows us to take an XML data source, transform it one or more times, and produce it for the web or another process.

Users enter their desired action in the URL (view a particular object).

All our programs are broken up into a set of small parts that can be modified easily

In order to keep the data separate from action and design  we have different pipelines that deal only with 
data input and output from databases, 
 html creation
merging, sorting,  and manipulating data from various sources.

Each pipeline is independent from the others, and can be changed internally without affecting others in the chain.  I’ll illustrate this later.

## Key Elements in Describing an Object

**Bibliographic data**
**Files **
**Object type **
**(sheetMusic,printMaterial**
**	****audio,video, compactDisc**
**	****recordedEvent, bibRecord, **
**	****photo, Article, Score...)**

## Harvested from Voyager ILS (SRU/z3950)
Inserted into MySQL database				

**Inserted into MySQL database****	****	****	****	**

**Bibliographic data (1)**

**Notes:**

For Bibliographic data, we have to processes

We have automated processes that get MODS records from other sources, such as our Voyager ILS using  sru (z3950)

We insert those records back into our database. 

## Bibliographic Data(2)

**Direct Data entry for items not previously cataloged**
**	****	****	****	**

**Notes:**
Bib Data 2
The second way we get metadata is we have data entry screens for items never previously cataloged.  That data is stored in MySQL but available on the web immediately, through a process you’ll see next

## Bibliographic Data is extracted from the database and converted to MODS

**METS Maker Pipeline – Step 1**

**Notes:**
A major tool we’ve created is our METS Maker pipeline, which consists of several steps:
Our METS consist of file locations, bibliographic data, and logical/structural data indicating how to put the files together.

The previous slides showed getting data out of the database; another step is to convert that to MODS using a stylesheet

## Combine queries of:
Object type (SheetMusic, RecordedEvent etc.)
Files on the server
MODS data
Rights metadata
Creates a virtual file called “pre-mets”

- Object type (SheetMusic, RecordedEvent etc.)
- _[Files](http://lcweb2.loc.gov:8081/cocoon/natlib/files.xml?DigitalID=200003788)_ on the server
- MODS data
- _[Rights](http://lcweb2.loc.gov:8081/cocoon/natlib/rightsdb.xml?DigitalID=200003788)_ metadata
**Creates a virtual file called “pre-mets”**

**METS Maker Pipeline – Step 2**

**Web****Pipeline**

**Object**
**Type**

**MODS****MD**

**Rights****MD**

**Files ****on ****Server**

**pre-**
**mets**

**Notes:**
Then another cocoon pipeline combines a query of the object type, the files on the server, the mods data and rights metadata into a file called “data”

## METS Maker XSLT processes the file using the object’s profile and builds a METS File

**METS Maker Pipeline – Step 3**

- METS file for Beaux Arts Trio concert recording

**Notes:**
And our metsmaker stylesheet processes that file according to the object type to build a METS object.

## Within-object Navigation

**Display – Step 1**

**Notes:**
Having made our METS object, we need to use it to display the object for users to view or play it on the web.

We create a pipeline that sets up navigation behaviors for the object.

that gets transformed into our menu; that you see onscreen

## Descriptive information about the object

**Display – Step 2**

**Notes:**

Another section of our display HTML is the descriptive metadata; 

we have a stylesheet that takes our simplified label/value pairs and converts them to list items in HTML.

## Page-turning or Playlists

**Display – Step 3**

- (pre-display virtual file )

**Notes:**

When an object has pages to turn or sections to play, we transform the files listed in the mets first 
into simplified page elements, and 
then into HTML.

We combine the navigation menu, bibliographic data, and any pages into a pre-display virtual file, ready for final transformation

## Add page framing and breadcrumbs

**Display – Step 4**

- LC Presents header and high-level navigation

- Breadcrumb navigation

- LC Presents footer (not in view)

**Notes:**

Our final step is to add branding headers and footers, plus a breadcrumb trail.

The display and all the intermediate files are virtual pages; no HTML is actually stored on disk.

## Veterans History Project Displays

- Veterans History Project *Experiencing War* Header

- Standard Veterans History Project Header

**Notes:**
 in VHP, (another site we built and maintain), we take it a step further, and swap out the view based on the presentation we are creating, so the object can be at home in a different context

In VHP, some of our veterans stories are featured in “Experiencing War”, so they need to look like that in the main presentation, but when they are found as a result of the search page, they need to look like a regular VHP item.

While we’re on VHP:an interesting note: these are stories and collections of artifacts around an individual, not books, etc.  

But we can create a METS profile for a veteran’s story that allows for photos, creative works, interviews in audio, video, transcriptions of interviews, etc. [see Mimi Lesser: 11904 browse it]. [next is search/lucene]

## Searching

**Lucene is open source and scalable to large indexes: **_**[Australia ](http://ll01.nla.gov.au/index.jsp)**__**has 16 million items indexed**_
**Lucene indexes are easy to configure using XSLT: **_**[METS ](http://lcweb2.loc.gov/natlib/ihas/mets/lucene/loc.natlib.ihas.200003788.xml)**__**[](http://lcweb2.loc.gov/natlib/ihas/mets/lucene/loc.natlib.ihas.200003788.xml)**__**[ Indexed item](http://lcweb2.loc.gov/natlib/ihas/mets/lucene/loc.natlib.ihas.200003788.xml)**_

**Notes:**
So we’ve seen building and displaying digital objects; now lets talk about searching

We use another open source tool, Lucene, for indexing and search.

Lucene allows you to build your index files any way you want, and changing it is just making a change to an XSL stylesheet.

[next is search page]

## Searching in LC Presents

**Single box search offered on home page**
_**[Collection-level search](http://memory.loc.gov/cocoon/ihas/html/ragtime/index.html)**_** offered for subcollections**
**Searches may also be limited to certain fields ****or object types**

**Notes:**
For any search entered by the user, we translate the search into a query that Lucene understands.  We add word stemming, case insensitivity, and other functionality to the indexing as well as the searching. 

[Read the lines and go to advanced search]

[Next is Canned search ]

## Searching in LC Presents

**Canned search to create “virtual collections”**

**Notes:**

We have a  way of showcasing virtual collections; 

any new objects that are members of a set we designate are automatically included in a presentation, and the canned search results page grows automatically as items are added.

## Searching in LC Presents

**New **_**[browsing](http://lcweb2.loc.gov/cocoon/ihas/anybrowsesubjects?query=gottlieb)**__** capability**_

- Browse by
- Subject, Name or Title

**Notes:**

We are also developing new browsing capability for search results sets; 

delving into the set and extracting names, subjects and titles for sorting and grouping. 

 We’ve also made our subjects searchable links so that you can find related topics in other objects across all of LC Presents.

## Searching in Veterans History

- Limit by conflict, branch, gender, POW status, etc.

**Notes:**

In VHP, we do a lot of limiting by various aspects of a veteran’s status, 

so you can group whole categories of people or search more narrowly among them 

[WWII vets interviewed by rouse: goofein creative works]

## Browsing in Veterans History

- Browse by last name, state, race/ ethnicity and war/branch

**Notes:**

These browses are another way of making your own virtual collections;

 browse by war and service branch, or name, state, race.

## Searching in Minerva (across collections)

- Choose among collections harvested.

- MODS terms to search.

**Notes:**

In Minerva (our web harvesting archive), we have recently developed an index that combines the archives into one, so you can perform the same search across all archives.  

This is beginning to show us that even though we may use MODS to describe an object, the differences among the collections make unified searching difficult, unless you just want to find key words in objects, a la google.

[tools is next]

## Other Web Tools

**SQL – query any Oracle or MySQL tables from the browser**
- Holdings queries (beyond z3950/SRU)
**Site administration**
- Convert from MARC to MODS or MODS to MARC
- Convert _[HTML](http://en.wikipedia.org/wiki/List_of_books_by_P._G._Wodehouse#Chronological_list)_ to _[XML](http://lcweb2.loc.gov:8081/cocoon/test/tidy?source=http://en.wikipedia.org/wiki/List_of_books_by_P._G._Wodehouse)_
- Link checking

**Notes:**
We use Cocoon to build a lot of tools that aid our processes. 

We do a lot of SQL using Cocoon; besides getting MODS records from Voyager, but we can do ANY query from Voyager or from any MySQL database; these queries are useful in creating METS objects from voyager records.
[Read site administration]
Sometimes, the data we ingest is only in an HTML presentation page, so we need to convert that HTML to true XML first, so we can write stylesheets that transform it into a useful fielded format. 

## Future Directions

**New Behaviors, Profiles**
- _[Article OCR](http://lcweb2.loc.gov/cocoon/diglib/loc.news.sr.1003/default.html)_
- _[Multivolume](http://lcweb2.loc.gov/cocoon/test-ihas/loc.rbc.sr.50047442/default.html)__[ Monograph](http://lcweb2.loc.gov/cocoon/test-ihas/loc.rbc.sr.50047442/default.html)_
- _[Generating RSS data on the fly](http://lcweb2.loc.gov:8081/cocoon/test/natlibRSS)_
**Integrate JHOVE file inspection tool (and MIX metadata) into our METS objects**
**Move to JPEG2000 file format for images**

**Notes:**
We’re always looking for new behaviors and profiles to accommodate them.

Here’s a prototype of a full-text, article-level markup of a historical newspaper (Morgan).  Note the use of the coordinates in CONTENTS.  Note the full text is added to MODS (full text searching!)

Here’s a prototype for a multivolume monograph.  Actually two object types: a collection object that brings together all the volumes (showing a as a Table of Contents view), and the individual volume objects, which are much like a single book object, but with a link back to the parent.

The second draft of the multivolume monograph is a newspaper index, but we’ve added the ability to jump from volume to volume, not just page to page.

We’re playing with generating RSS feeds on the fly; it’s easy to write a SQL statement that selects new objects from the database, and transform it into the RSS xml structure.  After that, anyone can subscribe to the latest “LC Presents” objects, or the newest vet story processed, etc.  IN ADDITION: RSS can be dynamically extracted and from voyager records, too.  [latest authority records?]
Using JPeg2000 images will allow us to quit creating derivatives in various sizes for our displays; a single image file can be dynamically sized according to the given display. (Thumbnail, full size etc).

[next is conclusions]

## Conclusions

**Profiles**** in METS help define an object and it’s ****	****range of behaviors**
**METS and MODS play well together**

** ****The METS standard is flexible enough to ****	****describe multiple kinds of complex objects.**

**Notes:**
The way we do that is through rigorously defined profiles that lay out what an object looks like in METS.

## Conclusions, cont.

**MODS is extremely powerful as a structural tool, not just a bibliographic tool**
**Consistent, authoritative, structured metadata makes search and display for an object persistent into future software and hardware systems**

**Notes:**
MODS related Items provide a valuable tool in describing objects in a METS profile

If your data is well structured, you will be a huge step ahead in creating tomorrow’s displays.

## Conclusions, cont.

**  ****Standards matter.  Without standardized   ****	****structures, we can’t program a site to retrive or ****	****display items with any reliability.**
**Open Source software is a major step forward for ****	****the ability of libraries and other lower budget ****	****institutions to enhance the availability and use ****	****of their digital assets.**

**Notes:**

All of our object display pages don’t really exist.  We think the work of our designer, Betsy Miller, is key to the usability of our site, and we wanted to make sure that when she wants to update our look, we can do so with minimal impact on the site. 

The next view of our objects will be easier to design because the data will be ready to accommodate any new design layout or new behavior.  

We have set up our objects with a modular pipeline that has a separate design layer, so the object data can be reused without modification when the look changes.  

We believe open source, standard structures and tools enable future redesigns to be less painful, costly, and time consuming.

## Nate Trail  
ntra@loc.gov

_**[ntra](mailto:ntra@loc.gov)**__**[@loc.gov](mailto:ntra@loc.gov)**_

**www.loc.gov/lcp (LC Presents Music, Theater, and Dance)**

***Questions?***

** ****Library of Congress**

**www.loc.gov/vets (Veterans History Project)**