# Presentation: 814820

## 8. Metadata Interoperability and Crosswalks

- Metadata Standards and Applications Workshop

**Notes:**
”Review exercise if done as homework.

## Goals of Session

- Understand interoperability protocols (OAI-PMH, OpenURL for reference)
- Understand crosswalking and mapping as it relates to interoperability

## Tools For Sharing Metadata/Interoperability

- Protocols
- OAI-PMH for harvesting
- OpenURL for reference linking
- Good practices and documentation
- Crosswalking

**Notes:**
Interoperability 
the ability of two or more systems or components to exchange information and to use the information that has been exchanged 
A repository is a collection of resources that can be accessed to retrieve information. 

## What’s the Point of Interoperability?

- For users, it’s about resource discovery (user tasks)
- What’s out there?
- Is it what I need for my task?
- Can I use it?
- For resource creators, it’s about distribution and marketing
- How can I increase the number of people who find my resources easily?
- How can I justify the funding required to make these resources available?

**Notes:**
Interoperability is a kind of buzzword but stands for something that traditional libraries have taken for granted.  The metadata world is still exploring what it means.
Users are trying to figure out certain things (see the questions on the slide)
The slide asks important questions for resource creators

## OAI-PMH

- Open Archives Initiative Protocol for Metadata Harvesting (_[http://www.openarchives.org/](http://www.openarchives.org/)_)
- Roots in the ePrint community, although applicability is much broader
- Mission: “The Open Archives Initiative develops and promotes interoperability standards that aim to facilitate the efficient dissemination of content.”
- Content in this context is actually “metadata about content”

**Notes:**
Interoperability is a particular issue when we talk about distribution outside the source of the metadata.  One important resource for interoperability is OAI-PMH
Began with preprints in computer science and physics but the applicability is broader than that.
The four-element description they used to begin with was abandoned in favor of DC.
Metadata is created not just to convey bibliographic information (for example, author information) but to serve a number of functions within digital libraries.
The essence of the open archives approach is to enable access to Web-accessible material through interoperable repositories for metadata sharing, publishing and archiving. It arose out of the e-print community, where a growing need for a low-barrier interoperability solution to access across fairly heterogeneous repositories lead to the establishment of the Open Archives Initiative (OAI). The OAI develops and promotes a low-barrier interoperability framework and associated standards, originally to enhance access to e-print archives, but now taking into account access to other digital materials. As it says in the OAI mission statement "The Open Archives Initiative develops and promotes interoperability standards that aim to facilitate the efficient dissemination of content." 
The OAI-Protocol for Metadata Harvesting (OAI-PMH) defines a mechanism for harvesting records containing metadata from repositories. The OAI-PMH gives a simple technical option for data providers to make their metadata available to services, based on the open standards HTTP (Hypertext Transport Protocol) and XML (Extensible Markup Language). The metadata that is harvested may be in any format that is agreed by a community (or by any discrete set of data and service providers), although unqualified Dublin Core is specified to provide a basic level of interoperability. Thus, metadata from many sources can be gathered together in one database, and services can be provided based on this centrally harvested, or "aggregated" data. The link between this metadata and the related content is not defined by the OAI protocol. It is important to realise that OAI-PMH does not provide a search across this data, it simply makes it possible to bring the data together in one place. In order to provide services, the harvesting approach must be combined with other mechanisms

## OAI-PMH in a Nutshell

- Essentially provides a simple protocol for “harvest” and “exposure” of metadata records
- Specifies a simple “wrapper” around metadata records, providing metadata about the record itself
- OAI-PMH has been about the metadata, not about the resources
- 	ARTstor cdwa-Lite experiment http://www.artstor.org/index.shtml

**Notes:**
Protocol for interaction between a harvester and information exposed for harvesting.
Provides administrative metadata about the resource (DC has no administrative metadata) such as ID for the metadata record.

## Metadata

-  About the
- Resource

- OAI Wrapper

- Meta-Metadata

**Notes:**
A graphic view of the relationship of OAI with the descriptive metadata about the resource.

## What was OAI-PMH designed for?

- Way to distribute records to other libraries
- Low barrier to entry for record providers
- Based on
- Records must be in XML
- OAI-PMH supports any metadata format encoded in XML—Simple Dublin Core is the minimal format specified
- Not Z39.50
- Not a way to support federated search
- No “on-the-fly” sets.
- More like CDS service, but it’s free,
- users “pull” records when they want, at intervals that are convenient for them (every day, every hour, on any schedule, or ad hoc)

**Notes:**
More like LC record distribution service
From Diane’s notes on deleted slide: Technical support varies between institutions, so barriers to participation are a big issue. Important to note that servers do not need to be dedicated to OAI support, and many software packages are available via the OAI website.  A predictable flow of data maximizes the machine flow of information.

More from another deleted slide:
In harvesting metadata, you set the package you’re looking at.  In crawling, you’re going out on the open Web with the intent to collect certain things.  Harvesting has an inherent permission.  The harvester just has to have a harvesting program.

There’s a registry of OAI data providers with a listing of what is there.  Sometimes there is a description, sometimes there is not much.

An “item” is the lowest granularity of a resource.  In a Word document with two different drafts, the item level could include both drafts, or each one would be a separate item.  The data provider decides.

Flow control:  if you have thousands of records and don’t want to have to harvest them in one chunk, define the chunks and use them in chunk-order.  Get them in small enough chunks that it is easy to manage.

Q/A:  Can you lose control of your data?  You can’t control how people use your data after harvesting.  You can specify what they are allowed to do with them (for example, a rights statement about no commercial use), but there is no enforcement of that.  Sometimes a data provider does not even keep track of who uses the metadata.

## OAI-PMH: Data Provider

- Has records to share
- Runs system that responds to requests
- following protocol
- Advertises base URL from which records are harvestable
- Just leaves system running
**No human intervention needed to service requests**
**Can control level of activity to protect performance for primary users**

**Notes:**
[“runs system”  “just leaves system running?”]
System responds to requests...

## OAI-PMH: Service Provider

- Assumed to be providing “union catalog” service
- OAIster  { _[http://www.](http://www.oaister.org/)__[oaister](http://www.oaister.org/)__[.org/](http://www.oaister.org/)_ }
- or a specialist, value-added service
- Sheet Music Consortium {http://digital.library.ucla.edu/sheetmusic/ }
- Harvests records, with ability to select limited to
- Records updated in a certain timespan
- Predetermined sets of records (like CDS)
- Known records by identifiers (OAI identifiers, not LCCNs)

**Notes:**

[assuming all are familiar with cds]???

A data provider has metadata to offer.
DATA PROVIDER ======= SERVICE PROVIDER
Effort occurs at two poles—OAI seeks to mediate in the middle, with reasonable expectations at both ends.
LC is a both: a data provider & a service provider—

## Inside OAI Repositories

*repository* - A repository is a network accessible server that can process requests. A repository is managed by a data provider to expose metadata to harvesters
*resource* - A resource is the object or "stuff" that metadata is "about,” whether physical or digital, stored in the repository or a constituent of another database
*item* - An item is a constituent of a repository from which metadata about a resource can be disseminated.
*record* - A record is metadata in a specific metadata format

**Notes:**
[I vote to delete this slide?]

Note difference between an item and a record, and how those definitions are different than what traditional librarians might expect.

Metadata for an item may be disseminated on-the-fly from the associated resource, cross-walked from some canonical form, actually stored in the repository, etc.

Note the separation of content from metadata in the definition of a “resource.”  In our diagram on Slide 7, we had more than one record per item.  With the OAI-PMH protocol, one of these records must be in simple Dublin Core, but there can (and should) be others that address the needs of the community served.

## Protocol has 6 requests

- Identify
- Facts about the data provider service
- ListMetadataFormats
- ListSets
- Predetermined sets of records
- ListIdentifiers
- Refine by set, date range for last update
- Good way to count records
- GetRecord
- By record identifier
- ListRecords
- Like ListIdentifiers but with full records in specified format
- More info at _[http://www.](http://www.openarchives.org/)__[openarchives](http://www.openarchives.org/)__[.org/](http://www.openarchives.org/)_

**Notes:**
[I vote to delete this slide too?]

From Diane’s slide about this:
Demo the Protocol Request Forms.  Start with GetRecord. Use one of the following URLs (note that the first part of the id “oai:nsdl.org” is already there).  Use nsdl_dc as the metadataPrefix (that’s the NSDL version of qualified DC)
Try: oai:nsdl.org:nsdl.nsdl:496367 this is a collection record
Or:
oai:nsdl.org:dlese.org:oai:dlese.org:DLESE-000-000-001-991  this is an item record

{[need to follow this}]???
[be careful about cut&paste introduced spaces, which will result in no record returned]

Go through some of the requests:  ID request, Response Date, base URL, adminEmail.  Note what the repository does with the deleted records.  In this case, the deleted records are transient (which means that the deleted records are not retained in the database).

“Get Record.”  nsdl.nsdl:00126  gets an error message.  Try nsdl.nsdl:1007936  This is definitely machine to machine.
“List MD formats.”  Give a schema for each schema used.  nsdl_dc is actually a qualified DC element.  oai_dc is the simple DC scheme.

A harvester can reorganize data.  It can use all the information, not just the way the data provider has used it.  There is no notion of one authoritative version of metadata.  Over time, you will want to know where stuff came from!

## http://www.oaforum.org/tutorial/english/page3.htm#section3

## OAI at LC

- LC as OAI Data Provider for historical collections
- http://memory.loc.gov/ammem/oamh/lcoa1_content.html
- Adding new collections steadily
- MARC source (so far)
- Handles rather than regular URLs in 856 $u
- Records upgraded from PREMARC (minimally, not AACR2)
- Records available as MARCXML, MODS or DC
*Available and Useful: OAI at the Library of Congress*
- Library Hi Tech, Volume 21, No. 2, 2003, pp. 129-139 [DOI:10.1108/07378830310491899]
- http://memory.loc.gov/ammem/techdocs/libht2003.html

**Notes:**
[define handles]

In Fall 2002, LC made its OAI-harvestable records available in the MODS format in addition to MARC21 and simple Dublin Core. These records provide richer semantics than Dublin Core and will support experimentation with the new format. The implementation uses an XSLT stylesheet to convert MARCXML records to MODS. NDMSO intends to maintain a suite of such tools for manipulating and transforming records between MARC and MODS.
The Library of Congress has registered its OAI server for American Memory collections in the central OAI registry, which means that others can harvest and re-purpose records without LC being aware of it. This is nothing new for LC. Its records are available for copying from bibliographic utilities, such as OCLC and RLG; users of Z39.50 have been able to download records directly from LC's main catalog for well over a decade. Indeed, one of the key benefits to LC from the harvesting protocol is that no special arrangements have to be made when others want to get the records and use them in another service. Since the records are updated infrequently, there is little re-harvesting and the processing load on the server has been unnoticeable.
LC has a particular interest in 
Metadata harvesting will be an important part of the interoperability toolkit for libraries. But it is just a first step. Communities that have close affinities with libraries, such as museums and other archival institutions, have good reason to use different descriptive practices 
A resource that is of potential use in many contexts may need to be described in several ways and using several vocabularies. The Library of Congress hopes that by making the intellectual assets in its catalog records more available, a valuable objective in itself, it can also learn how to make them more useful to other communities and in other contexts.

## A Simple OAI Example

**Notes:**
These examples may be redundant, if you can access the real server in the slide previous.
This example from the OAI documentation, note that the values DO NOT follow recommended Simple DC practices, particularly use of embedded labels in subject and multiple values in one statement.

## Demo:

- Straight from browser
- Sample queries on LC site
- _[http://memory.loc.gov/ammem/oamh/](http://memory.loc.gov/ammem/oamh/)_
- Harvesters need to:
- Work with XML
- Pick a metadata format
- Understand how to harvest complete set
- Could easily give a bit more help online
- More or different canned options (but not to harvest whole set)
- Explanation of how to harvest a whole set
- Point to OAI-harvesting software libraries

**Notes:**
[where did this slide come from—don’t think it is needed] ?

## OAI Best Practices Activities

- Sponsored by Digital Library Federation (DLF)
- Guidelines for data providers and service providers
- http://oai-best.comm.nsdl.org/cgi-bin/wiki.pl
- Not just DLF, also NSDL
- Best Practices for Shareable Metadata
- http://oai-best.comm.nsdl.org/cgi-bin/wiki.pl?PublicTOC
- Workshops to encourage DLF members to make records for their digitized content harvestable
- Also sponsored by IMLS

**Notes:**
Define initials,,,
National Science Digital Library
IMLS

## Slide 18

## OAIster http://www.oaister.org/

- A **union catalog of digital resources**. Provides access to digital resources by "harvesting" their descriptive metadata (records) using OAI-PMH.
- Currently provides access to 14,900,092 records from 939 contributors.

**Notes:**
One of the hopes was that a pool of metadata would support experimentation with interfaces and navigation. The interface for RLG's cultural materials service allowed users to switch easily between different structural views of the current result set enabling different browsing strategies and different approaches to successive refinement of a search. Every item is represented by a thumbnail for visual browsing; explicit modeling of parent-child relationships facilitates navigation from collection records to item records and vice versa. How will users respond to this type of interface in comparison to the more traditional interface of OAIster, the UIUC Cultural Heritage Repository, and American Memory itself. Will thumbnails become a standard component of metadata records? 

## http://www.oaister.org/

**Notes:**
OAIster is a union catalog of digital resources. We provide access to these digital resources by "harvesting" their descriptive metadata (records) using OAI-PMH (the Open Archives Initiative Protocol for Metadata Harvesting). The Open Archives Initiative is not the same thing as the Open Access movement.

## What’s an OpenURL?

- The OpenURL provides a standardized format for transporting bibliographic metadata about objects between information services
- Provides a basis for building services via the notion of an *extended service-link*, which moves beyond the classic notion of a *reference link* (a link from metadata to the full-content described by the metadata)

## OpenURL Characteristics

- Protocol operates between an information resource and a service component
- Service component is called a “link server” or “link resolver”
- Link server defines the user context
- Takes source citation and determines whether a user has access

**Notes:**
The link resolver sits at the institution locally or is contracted out.  The link resolver keeps track of what you have access to.

## Distinguishing Users

- Uses information stored in a cookie (the CookiePusher mechanism)
- Uses information contained in a digital certificate, such as the one proposed by the DLF digital certificates prototype project
- Identifies a user's IP address
- Obtains user attributes via the Shibboleth framework

**Notes:**
The system has to understand or know if the users have the right to use info......Shibboleth is an attempt to do this (authenticate users) across institutions, but it still has a long way to go ...

Shibboleth is based on a model whereby the user is authenticated closest to where they are known.  It is an authentication protocol.

The way identity is tracked by an institution as a whole is not always done at the level of granularity that is needed to authenticate users for access to information resources.

## Additional Open URL Services

- Link from a record in an abstracting and indexing database (A&I) to the full-text described by the record
- Link from a record describing a book in a library catalogue to a description of the same book in an Internet book shop
- Link from a reference in a journal article to a record matching that reference in an A&I database
- Link from a citation in a journal article to a record in a library catalogue that shows the library holdings of the cited journal

**Notes:**
Examples:

You can use citation information to see if the resource is available in print at the local library.  Or you can populate an ILL form.

Or you can send the citation information to a commercial vendor.

## OpenURL Examples & Demo

- _[http://sfxserver.uni.edu/sfxmenu?issn=1234-5678&date=1998&volume=12&issue=2&spage=134](http://sfxserver.uni.edu/sfxmenu?issn=1234-5678&date=1998&volume=12&issue=2&spage=134)_
- An OpenURL demo:
- _[http://www.ukoln.ac.uk/distributed-systems/openurl/](http://www.ukoln.ac.uk/distributed-systems/openurl/)_

## Crosswalking

- “Crosswalks support conversion projects and semantic interoperability to enable searching across heterogeneous distributed databases. Inherently, there are limitations to crosswalks; there is rarely a one-to-one correspondence between the fields or data elements in different information systems.”
- -- Mary Woodley, *“Crosswalks: The Path to Universal Access?”*

**Notes:**

[should we keep this and the next slide?---RG says keep this one]

The vast majority of metadata comes down to beginning with the form you have it in and what form you want to turn it into.
Mapping (conceptual) === transformation (practice, albeit in an automated way)
At least map from the local (usually richer) scheme to simple DC.
Metadata work is not a one-person activity.  You might work with someone who has scripting skills.
There are various attempts to have lists of crosswalks.  Most are not being maintained and versioned.

## Crosswalks

- Semantic mapping of elements between source and target metadata standards
- Metadata conversion specification: transformations required to convert metadata record content to another
- Element to element mapping
- Hierarchy and object resolution
- Metadata content conversions
- Stylesheets are created to transform metadata based on crosswalks

**Notes:**
Crosswalks require constant maintenance considering that metadata standards are continuously revised.

## Problems With Converted Records

- Differences in granularity (complex vs. simple scheme)
- Some data might be lost
- Differences in semantics
- Differences in use of content standards
- Properties may vary (e.g. repeatability)
- Converting may not always be the solution

## Example:mapping MODS:title  to  DC:title

- Includes attribute for type of title
- Abbreviated
- Translated
- Alternative
- Uniform
- Other attributes: ID,authority,displayLabel,xLink
- Subelements: title, nameOfPart, numberOfPart, nonSort
- Title definition reused by: Subject, Related Item

**Notes:**
[need to provide list of element sets for both?]

## Mapping mods:title to dc:title

- DC has one element refinement: alternative
- DC title has no substructure; MODS allows for subelements for partNumber, partName
- Best practice statement in DC-Lib says include initial article; MODS parses into <nonSort>
- MODS can link to a title in an authority file if desired

## Metadata Crosswalks

- Dublin Core-MARC
- Dublin Core-MODS
- ONIX-MARC
- MODS-MARC
- EAD-MARC
- EAD-Dublin Core
- Etc.

**Notes:**
And vise-versa...
15 elements have been mapped to MARC.

Not an official mapping in other direction; many to one is harder

FGDC: proposal in 1994 to define any elements from CSDGM that didn’t map to a defined MARC field; MARC used for catalog access

SGML DTD for elements in MARC

Need to standardize for interoperation

## Crosswalks

- Library of Congress
- _[http://www.loc.gov/marc/marcdocz.html](http://www.loc.gov/marc/marcdocz.html)_
- MIT
- _[http://libraries.mit.edu/guides/subjects/metadata/mappings.html](http://libraries.mit.edu/guides/subjects/metadata/mappings.html)_
- Getty
- _[http://www.getty.edu/research/conducting_research/standards/intrometadata/crosswalks.html](http://www.getty.edu/research/conducting_research/standards/intrometadata/crosswalks.html)_

## Slide 33

**Notes:**
This crosswalk and the one following are from the Dublin Core website.  Look at these in terms of the Exercise.  What else do you need to do besides a crosswalk?
[where is the 246, 300]?

[

This slide shows a crosswalk for unqualified Dublin Core; the following slide is for Qualified Dublin Core.  What do you lose, going from MARC to DC?  (Example, Creator:  some 700s are for related records.  You could have a creator value on this record for someone who is actually the creator on another resource.)

This crosswalk tells people to use Source.  Source is a specific kind of relationship, so you should probably use Relation instead.  This crosswalk is dated because the MARC relators have been “dumbed down” to “Contributor.”

Note that the crosswalks don’t’ tell you how to put values into the other metadata scheme, for example, to put parts of a subject into different subfields.

## Slide 34

**Notes:**
With Qualified Dublin Core, we’ve gained some things but it still does not address a lot of fields.
Those libraries that use MARC relators such as “author” or “composer” are dumbed-down to Contributor.  This is an unintended consequence.

Mappings have two columns:  “mappings in context” with examples for each scheme.

## Metadata Good Practices

- Adherence to standards
- Planning for persistence and maintenance
- Documentation
- Guidelines expressing community consensus
- Specific practices and interpretation
- Vocabulary usage
- Application profiles
- Without good metadata and good practices, interoperability will not work

**Notes:**

These are good practices for metadata in general.  If there is a standard that applies to your metadata, use it.  It enhances interoperability, the ability to share your metadata.

Libraries have a long history of thinking of metadata use over time.  The need to document standards and practices has probably come up already, repeatedly.

The visual resource community has a syntax, for example VRA Core and Cataloging Cultural Objects (CCO).

Use vocabularies consistently and document them (more about how to do this later)

## NISO’s Metadata Principles

**1**: Good metadata conforms to community standards in a way that is appropriate to the materials in the collection, users of the collection, and current and potential future uses of the collection.
**2**: Good metadata supports interoperability.
**3**: Good metadata uses authority control and content standards to describe objects and collocate related objects

## NISO’s Metadata Principles Continued

**4**: Good metadata includes a clear statement of the conditions and terms of use for the digital object.
**5**: Good metadata supports the long-term curation and preservation of objects in collections.
**6**: Good metadata records are objects themselves and therefore should have the qualities of good objects, including authority, authenticity, archivability, persistence, and unique identification.

## Exercise

-  Examine the MARC to DC crosswalk and the DC to MARC crosswalk, determining where data loss occurs

- http://www.loc.gov/marc/marcdocz.html

**Notes:**
Instructions in the Trainer documentation.  This can be done as both a large and small group exercise.

Talk about conversion tools:
Nate’s tool
MARC Edit?
Some points that might be brought out:
MARC takes longer, record is longer than DC
Application profiles are important to set standards for content
Importance of becoming comfortable w/ XML syntax”]