# Presentation: 272767

## the OAI Protocol for Metadata Harvesting

## The Open Archives Initiative has been set up to create a forum to discuss and solve matters of interoperability between preprint solutions, as a way to promote their global acceptance. 

*“...**the joint impact of these and future initiatives can be substantially higher when interoperability between them [e-print archives] can be established...”*

*Paul Ginsparg, Rick Luce & Herbert Van de Sompel*

## Luce, Van de Sompel, Ginsparg

## federated services

## metadata harvesting via OAI-PMH

- metadata

- FTXT

## federated services via OAI-PMH

- metadata

## Santa Fe

- convention

- OAI-PMH
- v.1.0/1.1

- OAI-PMH
- v.2.0

## the OAI-PMH

- service provider

- data provider

**6**

**Notes:**
The terms data provider and servidde provider may somehow be misleading; the reasons that they are there is that we do indeed think of the harvester as being a system that wants to provide services for data collected from multiple repositories.  Still, it is perfectly imaginable that the protocol would only be used as aa meaans to sync metadata between 2 sys; as such no real notion of service provision would be involved.

## Core concepts in OAI-PMH

-  low-barrier interoperability
-  data-provider & service-provider model
-  metadata harvesting model

- OAI-PMH

- Dublin Core

- HTTP based

- Community specific,
- oai-rights

-  shared metadata format and parallel, community-specific metadata formats

-  acceptable use

##  OAI-PMH data model

- entry point to all records pertaining to the resource

- metadata pertaining
- to the resource

-  OAI-PMH data model

## OAI-PMH harvesting tools

- Supporting protocol requests:
-  Identify
-  ListMetadataFormats
-  ListSets

- service provider

- data provider

## supporting protocol requests

- ListMetadataFormats

- ListMetadataFormats / Time / Request
-  *REPEAT*
-  Format prefix
-  Format XML schema
*/REPEAT*

- service provider

- data provider

## Purpose
Return general information about the archive and its policies (e.g., datestamp granularity)
Parameters
None
Sample URL
http://memory.loc.gov/cgi-bin/oai2_0?verb=Identify

- Return general information about the archive and its policies (e.g., datestamp granularity)
- Parameters
- None
- Sample URL
- http://memory.loc.gov/cgi-bin/oai2_0?verb=Identify

- Supporting : Identify

##  Identify

- <Identify>
-     <repositoryName>Library of Congress 1</repositoryName>
-     <baseURL>http://memory.loc.gov/cgi-bin/oai</baseURL>
-     <protocolVersion>**2.0**</protocolVersion>
-     **<adminEmail>**r.e.gillian@larc.nasa.gov</adminEmail>
-     **<adminEmail>**rgillian@visi.net</adminEmail>
**    ****<earliestDatestamp>1990-02-01T00:00:00Z</earliestDatestamp>**
-     **<deletedRecord>transient</deletedRecord>**
**    ****<granularity>YYYY-MM-DDThh:mm:ssZ</granularity>**
**    ****<compression>deflate</compression>**
**</Identify>**

##  harvesting granularity

-  mandatory support of YYYY-MM-DD
-  optional support of  YYYY-MM-DDThh:mm:ssZ
-  other granularities considered, but ultimately rejected
-  granularity of from and until must be the same

-  harvesting granularity

## Purpose
List metadata formats supported by the repository as well as their schema locations and namespaces
Parameters
identifier – for a specific record (O)
Sample URL
http://memory.loc.gov/cgi-bin/oai2_0?verb=ListMetadataFormats

- List metadata formats supported by the repository as well as their schema locations and namespaces
- Parameters
- identifier – for a specific record (O)
- Sample URL
- http://memory.loc.gov/cgi-bin/oai2_0?verb=ListMetadataFormats

- Supporting : ListMetadataFormats

## Purpose
Provide a listing of sets in which records may be organized (may be hierarchical, overlapping, or flat)
Parameters
None
Sample URL
http://memory.loc.gov/cgi-bin/oai2_0?verb=ListSets

- Provide a listing of sets in which records may be organized (may be hierarchical, overlapping, or flat)
- Parameters
- None
- Sample URL
- _[http://memory.loc.gov/cgi-bin/oai2_0?verb=](http://memory.loc.gov/cgi-bin/oai2_0?verb=ListSets)__[ListSets](http://memory.loc.gov/cgi-bin/oai2_0?verb=ListSets)_

- Supporting : ListSets

## OAI-PMH harvesting tools

- Supporting protocol requests:
-  Identify
-  ListMetadataFormats
-  ListSets

- Harvesting protocol requests:
-  ListRecords
-  ListIdentifiers
-  GetRecord

- service provider

- data provider

## OAI-PMH harvesting tools

- service provider

- data provider

- Datestamp
- Identifier
- Set

- Records

## harvesting requests

-                      * from=a
-                      * until=b
-                      * set=klm
- ListRecords  * metadataPrefix=dc

- ListRecords / Time / Request
-  *REPEAT*
-  Identifier
-  Datestamp
-  Metadata
*/REPEAT*

- service provider

- data provider

## Purpose
Returns the metadata (specific format) for a single item in the form of a  record
Parameters
identifier – unique id for item (R)
metadataPrefix – identifier of metadata format for the record (R)
Sample URL
http://memory.loc.gov/cgi-bin/oai2_0?verb=GetRecord&identifier=oai:lcoa1.loc.gov:loc.gdc/lhbcb.00835&metadataPrefix=oai_dc

- Returns the metadata (specific format) for a single item in the form of a  record
- Parameters
- identifier – unique id for item (R)
- metadataPrefix – identifier of metadata format for the record (R)
- Sample URL
- http://memory.loc.gov/cgi-bin/oai2_0?verb=GetRecord&identifier=oai:lcoa1.loc.gov:loc.gdc/lhbcb.00835&metadataPrefix=oai_dc

- Harvesting : GetRecord

## Purpose
Retrieves the metadata (specific format) for multiple items in the form of records
Parameters
from – start datestamp (O)
until – end datestamp (O)
set – set to harvest from (O)
resumptionToken – flow control mechanism (X)
metadataPrefix – metadata format (R)
Sample URL
http://memory.loc.gov/cgi-bin/oai2_0?verb=ListRecords&metadataPrefix=oai_dc

- Retrieves the metadata (specific format) for multiple items in the form of records
- Parameters
- from – start datestamp (O)
- until – end datestamp (O)
- set – set to harvest from (O)
- resumptionToken – flow control mechanism (X)
- metadataPrefix – metadata format (R)
- Sample URL
- http://memory.loc.gov/cgi-bin/oai2_0?verb=ListRecords&metadataPrefix=oai_dc

- Harvesting : ListRecords

## Purpose
List headers for all items corresponding to the specified parameters
Parameters
from – start datestamp (O)
until – end datestamp (O)
set – set to harvest from (O)
metadataPrefix – metadata format to list identifiers for (R)
resumptionToken – flow control mechanism (X)
Sample URL
http://memory.loc.gov/cgi-bin/oai2_0?verb=ListIdentifiers&metadataPrefix=oai_dc

- List headers for all items corresponding to the specified parameters
- Parameters
- from – start datestamp (O)
- until – end datestamp (O)
- set – set to harvest from (O)
- metadataPrefix – metadata format to list identifiers for (R)
- resumptionToken – flow control mechanism (X)
- Sample URL
- http://memory.loc.gov/cgi-bin/oai2_0?verb=ListIdentifiers&metadataPrefix=oai_dc

- Harvesting : ListIdentifiers

##  header contains set membership of item

-  header

-  <record>
-    <header>
-     <identifier>oai:arXiv:cs/0112017</identifier>
-     <datestamp>2001-12-14</datestamp>
**    ****<setSpec>cs</setSpec> **
**    ****<setSpec>math</setSpec> **
-    </header>
-    <metadata>
-     .....
-    </metadata>
-   </record>

##  ListIdentifiers returns headers

-  ListIdentifiers

- <?xml version="1.0" encoding="UTF-8"?>
- <OAI-PMH>
- <responseDate>2002-0208T08:55:46Z</responseDate>
- <request verb=“...” ...>http://arXiv.org/oai2</request>
- <ListIdentifiers>
-    **<header>**
**    ****<identifier>oai:arXiv:hep-th/9801001</identifier>**
**    ****<datestamp>1999-02-23</datestamp>**
**    ****<setSpec>physic:hep</setSpec>**
**   ****</header>**
**   ****<header>**
**    ****<identifier>oai:arXiv:hep-th/9801002</identifier>**
**    ****<datestamp>1999-03-20</datestamp>**
**    ****<setSpec>physic:hep</setSpec>**
**    ****<setSpec>physic:exp</setSpec>**
**   ****</header>**
-     ......

## Not (necessarily) identifier of the resource
Each item must have a globally unique identifier
identifiers must follow rules for valid URIs
Example:
 oai:<archiveId>:<recordId>
 oai:etd.vt.edu:etd-1234567890
Each identifier must resolve to a single item and always to the same item 
Can’t reuse OAI item identifiers 

- Each item must have a globally unique identifier
- identifiers must follow rules for valid URIs
- Example:
-  oai:<archiveId>:<recordId>
-  oai:etd.vt.edu:etd-1234567890
- Each identifier must resolve to a single item and always to the same item
- Can’t reuse OAI item identifiers

- OAI-PMH identifiers

## Needed for every OAI record to support incremental harvesting
Must be updated when addition or modification or deletion made in order to ensure changes are correctly propagated to harvesters
Also for dynamically generated metadata formats
Different from dates within the metadata – OAI datestamp is used only for harvesting
Can be either YYYY-MM-DD or YYYY-MM-DDThh:mm:ssZ (must be GMT timezone)

- Must be updated when addition or modification or deletion made in order to ensure changes are correctly propagated to harvesters
- Also for dynamically generated metadata formats
- Different from dates within the metadata – OAI datestamp is used only for harvesting
- Can be either YYYY-MM-DD or YYYY-MM-DDThh:mm:ssZ (must be GMT timezone)

- OAI-PMH datestamps

## requests must be submitted using the GET or POST methods of HTTP
repositories must support both methods

- repositories must support both methods

- OAI-PMH request format

## formatted as HTTP responses
content type must be text/xml
status codes (distinguished from OAI-PMH errors)e.g. 302 (redirect), 503 (service not available)

- content type must be text/xml
- status codes (distinguished from OAI-PMH errors)e.g. 302 (redirect), 503 (service not available)

- OAI-PMH response format

## response format: well formed XML:
XML declaration (<?xml version="1.0" encoding="UTF-8" ?>)
OAI-PMH root element
three child elements
 responseDate (UTC datetime)
 request (request that generated this response)
 a) error (in case of an error or exception condition) b) element with the name of the OAI-PMH request

- XML declaration (**<?xml version="1.0" encoding="UTF-8" ?>**)
**OAI-PMH** root element
- three child elements
-  **responseDate** (UTC datetime)
-  **request** (request that generated this response)
-  a) **error** (in case of an error or exception condition) b) element with the name of the OAI-PMH request

- OAI-PMH response format

## OAI-PMH response, no errors

**<OAI-PMH>**
**<responseDate>**2002-0208T08:55:46Z</responseDate>
**<request verb=“GetRecord”... ...>**http://arXiv.org/oai2</request>
** ****<GetRecord> **
-   <record>
-    <header>
-     <identifier>oai:arXiv:cs/0112017</identifier>
-     <datestamp>2001-12-14</datestamp>
-     <setSpec>cs</setSpec>
-     <setSpec>math</setSpec>
-    </header>
-    <metadata>
-     .....
-    </metadata>
-   </record>
** ****</GetRecord>**
**</OAI-PMH>**

##  OAI-PMH response, error

**<OAI-PMH>**
**<responseDate>**2002-0208T08:55:46Z</responseDate>
**<request>**http://arXiv.org/oai2</request>
**<error code=“badVerb”>**ShowMe is not a valid OAI-PMH verb</error>
**</OAI-PMH>**

-  OAI-PMH response, error

## repositories must indicate OAI-PMH errors
inclusion of one or more error elements 
defined errors:
badArgument
badResumptionToken
badVerb
cannotDisseminateFormat
idDoesNotExist
noRecordsMatch
noMetaDataFormats
noSetHierarchy

- inclusion of one or more **error** elements
- defined errors:
**badArgument**
**badResumptionToken**
**badVerb**
**cannotDisseminateFormat**
**idDoesNotExist**
**noRecordsMatch**
**noMetaDataFormats**
**noSetHierarchy**

- OAI-PMH errors

## flow control on two protocol levels
HTTP (503, retry-after)
OAI-PMH, resumptionToken
HTTP “retry-after” mechanism can be used in order to delay requests of clients
resumptionTokens are used to return parts (incomplete lists) of the result.
client receives a resumptionToken which can be used to issue another request – in order to receive further parts of the result

- HTTP (503, retry-after)
- OAI-PMH, resumptionToken
- HTTP “retry-after” mechanism can be used in order to delay requests of clients
- resumptionTokens are used to return parts (incomplete lists) of the result.
- client receives a resumptionToken which can be used to issue another request – in order to receive further parts of the result

- Flow control

## four of the request types return a list of entries
three of them may reply ‘large’ lists
OAI-PMH supports partitioning response
decision on partitioning: repository 
response to a request includes
incomplete list
resumption token + expiration date, size of complete list, cursor (optional)
new request with same request type 
resumption token as parameter
all other parameters omitted!

- three of them may reply ‘large’ lists
- OAI-PMH supports partitioning response
- decision on partitioning: repository
- response to a request includes
- incomplete list
- resumption token + expiration date, size of complete list, cursor (optional)
- new request with same request type
- resumption token as parameter
- all other parameters omitted!

- Flow control

## Flow control

**Harvester**

**Repository**

**Service ****Provider**

**Data ****Provider**

## record-level “about” container

- <about>
-  **<provenance>**
**  ****<originDescription> **
**   ****<baseURL>http://an.oa.org</baseURL>         **
**   ****<identifier>oai:r1:plog/9801001</identifier> **
**   ****<datestamp>2001-08-13T13:00:02Z</datestamp>   **
**   ****<metadataPrefix>oai_dc</metadataPrefix> **
**   ****<harvestDate>2001-08-15T12:01:30Z</harvestDate>   **
- 	<originDescription>
-   	 ... ... ...
-   	</originDescription>
**  ****</originDescription>**
-  **</provenance>**
- </about>

## record-level “about” container

- W3C XML schema defines format for **<rights> **package to be included in **<about> **container

## repository-level “description” container

- <description>
** ****<friends>**
**  ****<baseURL>http://cav2001.library.caltech.edu/perl/oai</baseURL>**
**  ****<baseURL>http://formations2.ulst.ac.uk/perl/oai</baseURL>**
**  ****<baseURL>http://cogprints.soton.ac.uk/perl/oai</baseURL>**
**  ****<baseURL>http://wave.ldc.upenn.edu/OLAC/dp/aps.php4</baseURL>**
** ****</friends>**
- </description>

## more info

- _[http](http://www.openarchives.org/OAI/openarchivesprotocol.html)__[://www.openarchives.org/OAI/openarchivesprotocol.html](http://www.openarchives.org/OAI/openarchivesprotocol.html)_
-  Validation tool
- 	_[http://re.cs.uct.ac.za/](http://re.cs.uct.ac.za/)_
-   Repository and harvesting tools
-  _[http://www.openarchives.org/tools/tools.html](http://www.openarchives.org/tools/tools.html)_
-  Registries of public OAI-PMH repositories
-  _[http://re.cs.uct.ac.za/](http://re.cs.uct.ac.za/)_
-  _[http://gita.grainger.uiuc.edu/registry/](http://gita.grainger.uiuc.edu/registry/)_
-  _[http://](http://www.openarchives.org/Register/BrowseSites)__[www.openarchives.org/Register/BrowseSites](http://www.openarchives.org/Register/BrowseSites)_

**Notes:**
Harvesting deep web: cfr Springer putting static pages out in order for them to be indexed by Google and the likes

## The OAI-PMH protocol is a low-barrier interoperability specification for the recurrent exchange of metadata between systems

-  Things become really cool when allowing flexibility re the interpretation of metadata.
-  Indeed: in OAI-PMH metadata is XML-formatted data pertaining to the resource