# Presentation: 944205

## How to Build, Display and Find METS Objects

- Nate Trail
- Network Development &
- MARC Standards Office
- 8/1/2006

- With help from Sydney Olive

**How to Build, Display and Find METS Objects**

## Outline

- A little about the technology
- How we use it
- Searching and Lucene
- Cocoon Tools
- Future Directions
- Conclusions

**Notes:**
In the context of “Library of Congress Presents and Veterans history project,
The Technology we use
How we use the technology
Searching
Cocoon tools in building and maintaining the application
Future Directions
Conclusions about our standards prototyping

## Technology Used

-  XML documents (METS, MODS, etc.)
- _[METS Object](http://memory.loc.gov/cocoon/ihas/loc.natlib.ihas.200003788/mets.xml)_
-  XSL processes
- Transformation or display
-  Cocoon framework (and a little Java)

**Notes:**
XML is data marked up according to a schema, laid out with tags of elements and attributes and data, but self defined.
METS and HTML can be examples of XML
XSL is also an XML file, but it is a set of processing instructions that transform an xml document from one format to another, or display it in a browser 
Cocoon is a web processing framework that allows us to work with URLs on the web and on UNIX instead of on PCs.  
Java is used to enhance our searching and breadcrumb tracking.

## XSL for Transformation

**Notes:**
A little about XSL.  
As a transformation tool, we use it to convert a mods record, with it’s layers and complexity, to a flattened structure that’s easier to display: elements with labels and values.

## XSL for Display

**Notes:**
XSL for display is really also just a transformation because HTML is a standardized xml structure that a browser reads and interprets.

## 
User input via URL matches

- User input via URL matches

- Cocoon Concepts (1)

- –<map:match pattern="**navigation/*****"****>**
- 	<map:generate src="cocoon**:/{1}/mets.xml**"/>
- 	<map:transform src="**navigation.xsl**"/>
- 	<map:serialize type="**xml**"/>
- –</map:match>

## 

Combine small matches to create
a pipeline

 Separate Data from Action and Design

- Combine small matches to create
- a pipeline
-  Separate Data from Action and Design

- Cocoon Concepts (2)

## Database SQL processing

						MySQL extract

						MODS (SRU)
						(Three masters LC  publication)

- 						_[MySQL](http://lcweb2.loc.gov:8081/cocoon/natlib/db.xml?DigitalID=100010476)__[ extract](http://lcweb2.loc.gov:8081/cocoon/natlib/db.xml?DigitalID=100010476)_
- 						_[MODS (SRU)](http://lcweb2.loc.gov:8081/cocoon/test/sru.xml?stdID=88602559&stylesheet=mods)_
- 						(Three masters LC  publication)

- How we use it (1)

**Notes:**
How we use the technology: 
SQL processing: 
We have a data entry form for manual entry of object descriptions.  We extract bibliographic data from MySQL 
We also have automated processes that get MODS records from other sources, and we insert those records back into our database using Cocoon. Here we have a little SRU pipeline that grabs Voyager records by LCCN, for insertion into the database.  We’ll see the object later in our search pages
Three masters

## METS making pipeline:
	MySQL extract        	MODS 
	ObjectType, Files, MODS, rights      data
	METSMaker stylesheet uses ObjectType to 	create links among files and descriptive 	metadata, based on METS profile: METS

- 	_[MySQL](http://lcweb2.loc.gov:8081/cocoon/natlib/db.xml?DigitalID=100010476)__[ extract](http://lcweb2.loc.gov:8081/cocoon/natlib/db.xml?DigitalID=100010476)_        	_[MODS](http://lcweb2.loc.gov:8081/cocoon/natlib/mods.xml?DigitalID=200003788)_
- 	ObjectType, Files, MODS, _[rights](http://lcweb2.loc.gov:8081/cocoon/natlib/rightsdb.xml?DigitalID=200003788)_      _[data](http://lcweb2.loc.gov:8081/cocoon/natlib/metsMaking.xml?DigitalID=200003788)_
- 	METSMaker stylesheet uses ObjectType to 	create links among files and descriptive 	metadata, based on METS profile: _[METS](http://lcweb2.loc.gov:8081/cocoon/natlib/mets.xml?DigitalID=200003788)_

- How we use it (2)

**Notes:**
A major tool is our METS Maker pipeline, which consists of several steps:
The previous slide shows getting data out of the database; another step is to convert that to MODS using a stylesheet
Then another cocoon pipeline combines a query of the object type, the files on the server, the mods data and rights metadata into a file called “data”
And our metsmaker stylesheet processes that file according to the object type to build a METS object.

## Cocoon for Display 
	Navigation, Description, Pageturning

	xml data for display
 add header, footer, 
	breadcrumbs

VHP presentations

- 	_[Navigation](http://lcweb2.loc.gov/cocoon/ihas/navigation/loc.natlib.ihas.200003788)_, Description, _[Pageturning](http://lcweb2.loc.gov/cocoon/ihas/pageturner/loc.natlib.ihas.200003788/recordedEvent)_
- 	xml data for display
-  add header, footer,
- 	breadcrumbs
- VHP _[presentations](http://lcweb2.loc.gov/cocoon/vhp-stories/loc.natlib.afc2001001.24003/)_

- How we use it (3)

**Notes:**
Given that we’ve built METS files for every object we have, we then build Display pipelines for the METS object profiles.
We combine together navigation, metadata, and page images (if any) to build an xml file that represents everything we’ll need from the object for  display.  
We then add branding and breadcrumbs to create the final display in a final step in the pipeline.
The display and all the intermediate files are virtual pages; no HTML is actually written.
Why add the header and footer separately? We want to have multiple views of the same object.  In VHP, some of our veterans stories are featured in “Experiencing War”, so they need to look like that in the main presentation, but when they are found as a result of the search page, they need to look like a regular VHP item.

(next is search/lucene)

## Searching and Lucene

- We have XML objects, so we need an XML based search system.
- Cocoon comes with Lucene.
- Lucene indexes are easy to configure and modify: _[METS ](http://lcweb2.loc.gov/natlib/ihas/mets/lucene/loc.natlib.ihas.200003788.xml)__[](http://lcweb2.loc.gov/natlib/ihas/mets/lucene/loc.natlib.ihas.200003788.xml)__[ Indexed item](http://lcweb2.loc.gov/natlib/ihas/mets/lucene/loc.natlib.ihas.200003788.xml)_

**Notes:**
So we have this set of 12-14 object types, and thousands of objects, and we needed a search tool. Cocoon comes with Lucene, and it’s XML based, so we have a natural fit.
Lucene allows you to build your index files any way you want, and changing it is just making a change to an XSL stylesheet.
(next is search page)

## Searching

- Search page
- limit search to certain object types,
- search certain fields only, or
- search all (default)
- Canned _[search](http://lcweb2.loc.gov/cocoon/ihas/search?query=%2BmemberOf:%22Library%20%2BCongress%20%2Bpublications%22&view=thumbnail&start=0)__ as collection presentation tool (LC Publications, baseball)_
- New _[browsing](http://lcweb2.loc.gov/cocoon/ihas/anybrowsesubjects?query=gottlieb)__ capability_

**Notes:**
Our advanced search page (we have a one-box search on the main LCP page) allows filtering and some more advanced selection. [search for cd with subject piano] We are pondering adding fields, formats, special searching functions.  For any search entered by the user, we translate the search into a query that Lucene understands.  We add word stemming, case insensitivity, and other functionality to the indexing as well as the searching. 
Canned search is home, LC Publications: a new way of showcasing virtual collections.
(next is cocoon tools)

## Cocoon Tools (1)

-  _[METS Profile Validation](http://lcweb2.loc.gov:8081/cocoon/natlib/metsCDProfileTest.xml/200003782)_ (fs01)
-  _[SQL ](http://lcweb2.loc.gov:8081/cocoon/test/query?db=ilssv4&SQL=select%20*%20from%20vendor_types)_: query any Oracle, MySQL tables from the browser to get back XML

**Notes:**
We use Cocoon to build a lot of tools that aid our processes.
Morgan mentioned profile validation.  I have a validator that works interactively for the CD Profile, but we have not yet integrated it into the METS creation process. See fs01.  We state all the rules, and the violation, so that you can tell the context of your error.  For integration, we’d filter it down to a report of pass/fail, with this document as backup.  We’ll write a validator for each METS profile as we have time. 
(more cocoon tools)

We do a lot of SQL using Cocoon; I showed how we extract MODS from Voyager, but we can also do any query from Voyager or from any MySQL database (the rights metadata isn’t in MODS, but we retain it in a table in MySQL)

## Cocoon Tools (2)

-  Site administration, enhancement of processes
- Get IDs for some set of the database for extraction, cleanup
- Convert from MARC to MODS (input) or MODS to MARC:  (extract)
- Convert _[HTML ](http://en.wikipedia.org/wiki/List_of_books_by_P._G._Wodehouse#Chronological_list)__to __[XML](http://lcweb2.loc.gov:8081/cocoon/test/tidy?source=http://en.wikipedia.org/wiki/List_of_books_by_P._G._Wodehouse)_
- Detect deletions of objects
- Link checking
- Search for MARC records using SRU

**Notes:**
For application administration, I have built a lot of tools.  
We often need to query or update a whole set of our MODS records, so I have a utility that gets those IDs.  
We convert data among various standard formats.  
Sometimes, the data we ingest is only in an HTML presentation page, so we need to convert that HTML to true XML first, so we can write stylesheets that transform it into a useful fielded format. (Next is Future)
We found we needed to track not only the presence of a METS file but it’s absence when an item was withdrawn or mistakenly added.
We have set up cocoon pipelines to check URLs on our sites.
I created a tool that gets MARC records and converts them to whatever format the user wants, or leaves it as an SRU result set. 
88602559 

## Future Directions

- New Behaviors, Profiles (_[Article OCR](http://lcweb2.loc.gov/cocoon/diglib/loc.news.sr.1003/default.html)_) (_[Multivolume Monograph](http://lcweb2.loc.gov/cocoon/test-ihas/loc.rbc.sr.50047442/default.html)_ / _[rarebook](http://lcweb2.loc.gov/service/rbc/rbc0001/2004/2004rosen1957volume1/pilkington.html)__[ source](http://lcweb2.loc.gov/service/rbc/rbc0001/2004/2004rosen1957volume1/pilkington.html)_)
- _[RSS](http://lcweb2.loc.gov:8081/cocoon/test/natlibRSS)_
- Integrate profile validation with METS building
- Integrate JHOVE file inspection tool (MIX)
- Investigate using JPEG2000 file format for image manipulation enhancement
- Submit objects to a repository
- Exchange objects with another entity

**Notes:**
We’re always looking for new behaviors and profiles: Here’s a prototype of a full-text, article-level markup of a historical newspaper.  Note the use of the coordinates (supplied by the OCR vendor.).  Note the full text is added to MODS, with links to coordinates.
Here’s a prototype for a multivolume monograph.  Actually two object types: a collection object that brings together all the volumes (showing a as a Table of Contents view), and the individual volume objects, which are much like a single book object, but with a link back to the parent.
Read the rest.
(next is conclusions)

## Conclusions

- What are we learning from building this application?

## Conclusions

- The METS standard is flexible enough to describe multiple kinds of complex objects

## Conclusions

- METS Profiles help define an object and it’s range of behaviors

**Notes:**
The way we do that is through rigorously defined profiles that lay out what an object looks like in METS.

## Conclusions

- METS and MODS play well together.

## Conclusions

- METS files can be built automatically.

## Conclusions

- It may look hard, but METS objects can be parsed, validated and used without difficulty

## Conclusions

- MODS is extremely powerful as a structural tool, not just a bibliographic tool
- (but good, consistent, authoritative cataloging makes search and display
- 	much more reliable)

## Conclusions

- Display is for today only; well structured data lasts.

**Notes:**
All of our object display pages don’t really exist.   So redesign is a comparatively easy task, if we got our data right.  We take the data from the stage where we know everything about the object, and apply a new design to that.

## Conclusions

- Without standardized structures, we end up with egg on our face.

**Notes:**
Last one.

## Nate Trail
ntra@loc.gov
7-2193

- Nate Trail
- _[ntra@loc.gov](mailto:ntra@loc.gov)_
- 7-2193

- Photos © 2006 Nate Trail

- Any Questions?

## Get back to work!!

- Photos © 2006 Nate Trail