# Presentation: 290353

## Tools for a Preservation-Ready Web 

- Joan A. Smith & Michael L. Nelson
- Old Dominion University
- Department of Computer Science
- {jsmit, mln}@cs.odu.edu
- NDIIPP Digital Preservation Partners Meeting
- July 9, 2008

## What is Preservation?

- We will define preservation of a web site *W* to be:
- refreshing
- copying the bits from place to place
- R(W) = Wr
- migrating
- converting the bits from format f1 to format f2
- M(W) = Wm
- emulation
- simulating the original context for the bits
- E(W) = We

- putting it all together:
- E(M(R(W))) = Wrme

- NDIIPP Partners Meeting

- July 9, 2008

## Preservation Function P

- We define a preservation function *P*
- P(W) = Wp
- Intuition is that P makes other functions easier:
- 	M(Wp) is easier to implement than M(W)
- 	E(Wp) is easier to implement than E(W)
- 	R(Wp) is probably easier to implement than R(W)

- NDIIPP Partners Meeting

- July 9, 2008

## Web Site Preservation: 2 Problems

- The counting problem
- What are the members of W?

- The representation problem
- How do we define P(W)?

- NDIIPP Partners Meeting

- July 9, 2008

## Preservation & the Counting Problem

- To preserve a site, we need to enumerate the full set of a web site’s resources:
- W = {w1 ,w2 ,w3 , w4 ... wn }
- For non trivial web sites:
- The membership of W depends on who is asking
- W is unknown (unknowable?)
- W can only be approximated
*There is no HTTP mechanism to define* W
- Sitemaps are a method to convey locally-held knowledge about W to web crawlers

- NDIIPP Partners Meeting

- July 9, 2008

## Preservation & the Representation Problem

- % telnet foo.edu 80
- Trying 82.165.199.160...
- Connected to foo.edu.
- Escape character is '^]'.
- GET /jackJill.jpg HTTP/1.1
- Host: foo.edu
- HTTP/1.1 200 OK
- Date: Mon, 11 Jun 2007 16:49:25 GMT
- Server: Apache/1.3.33 (Unix)
- Last-Modified: Mon, 29 Aug 2005 12:01:40 GMT
- ETag: "5800535-3e72-4312f924"
- Accept-Ranges: bytes
- Content-Length: 15986
- Content-Type: image/jpeg
- ÿØÿà
- "#2s¡35Rq‘±3ÁÂ$%Ccruƒ“¢ÃÒÿÄ

- HTML metadata

- HTTP metadata

- NDIIPP Partners Meeting

- July 9, 2008

## P(W) Involves the Output of Forensic Metadata Utilities

- 58a54e8638db432f4515eedf89f44505

- EXIF:
- File Name	103_0315.JPG
- Camera Model Name	Canon EOS DIGITAL REBEL
- Date/Time Original	2003:09:30 13:37:51
- Shooting Mode	Sports
- Shutter Speed	1/2000
- Aperture	7.1
- Metering Mode	Evaluative
- Exposure Compensation	0
- ISO	400
- Lens	75.0 - 300.0mm
- Focal Length	300.0mm
- Image Size	3072x2048
- Quality	Normal
- Flash	Off
- White Balance	Auto
- Focus Mode	AI Servo AF
- Contrast	+1
- Sharpness	+1
- Saturation	+1
- Color Tone	Normal
- File Size	1606 kB
- File Number	103-0315

**Standard HTTP Headers** --
- 	Last-Modified: Mon, 29 Aug 2005 12:01:40 GMT
- 	ETag: "5800535-3e72-4312f924"
- 	Content-Length: 15986
- 	Content-Type: image/jpeg

- JHOVE:
- Date: 2007-06-18 14:35:50 EDT RepresentationInformation: /home/crate/apache/htdocs/jackJill.jpg
- ReportingModule: JPEG-hul, Rel. 1.2 (2005-08-22) LastModified: 2007-01-16 23:09:07 EST Size: 27750
- Format: JPEG Version: 1.00 Status: Well-Formed and valid SignatureMatches: JPEG-hul
- MIMEtype: image/jpeg Profile: JFIF JPEGMetadata:  CompressionType: Huffman coding, Baseline DCT
- Images:  Number: 1 Image:  NisoImageMetadata:  MIMEType: image/jpeg ByteOrder: big-endian
- CompressionScheme: JPEG ColorSpace: YCbCr  SamplingFrequencyUnit: inch  XSamplingFrequency: 33
- YSamplingFrequency: 26  ImageWidth: 172  ImageLength: 146  BitsPerSample: 8, 8, 8  SamplesPerPixel: 3
- Scans: 1  QuantizationTables: QuantizationTable:  Precision: 8-bit  DestinationIdentifier: 0
- Comments: LEAD Technologies Inc. V1.01 ApplicationSegments: APP0

- File/Magic:
- JPEG image data
- JFIF standard 1.00
- resolution (DPI)
- "LEAD Technologies Inc. V1.01“
- 33 x 26

- P(W) Involves the Output of Forensic Metadata Utilities

- NDIIPP Partners Meeting

- July 9, 2008

## Experiments & Evaluation

- Counting problem
- Web crawler behavior on decaying web sites (D-Lib 2006)
- Web crawler behavior on deep and wide web sites (D-Lib 2008)
- Defining W on a departmental web site (unpublished)
- Representation problem
- Dissemination time preservation metadata (JCDL 2007, IWAW 2007, D-Lib 2008)
- Performance evaluation of metadata utilities (ECDL 2008)
- Reference implementation: mod_oai, an Apache module
- uses Sitemaps, OAI-PMH resource harvesting for counting problem
- uses “CRATE” -- base64’d resource + metadata output as the OAI-PMH metadataPrefix for representation problem

- NDIIPP Partners Meeting

- July 9, 2008

## Decaying Web Sites(D-Lib 2006)

-  Lots of pages die in between crawler visits
-  IA never came in 3+ months

- Decaying Web Sites(D-Lib 2006)

- NDIIPP Partners Meeting

- July 9, 2008

## Deep & Wide Web Sites 

- “Buffet” = Level 1 links to
- levels 2, 3, 4 ... n

- “Bread Crumb” = Level 1 links to level 2,
- level 2 links to level 3, etc.

- Deep & Wide Web Sites
- (D-Lib 2008)

- NDIIPP Partners Meeting

- July 9, 2008

## Coverage of www.cs.odu.edu

| Source | Files | URLs |
| --- | --- | --- |
| Self-Crawl | 406 | 538 |
| External Crawl | 406 | 761 |
| File System | 2,052 | 2,052* |

- Notes:
-  Departmental snapshot
-   (no ~user URLs; CGI
-   files removed; spotty http logs)
-  Google Python Sitemap
- script crashed on ill-formed
- log data
-  100% defined in terms of
- file system count
-  Results written in a Sitemap
- file for mod_oai processing
- (more later)

- NDIIPP Partners Meeting

- July 9, 2008

## Integrate OAI-PMH functionality into the web server itself...
Use mod_oai
an Apache 2.0 module
automatically answers OAI-PMH requests for an http server
written in C
respects values in .htaccess, httpd.conf
Install mod_oai on http://www.foo.edu/   
Define baseURL: http://www.foo.edu/modoai

Result: web harvesting with OAI-PMH semantics (e.g., from, until, sets)

- Use **mod_oai**
- an Apache 2.0 module
- automatically answers OAI-PMH requests for an http server
- written in C
*respects values in .htaccess, httpd.conf*
- Install mod_oai on _[http://www.foo.edu/](http://www.foo.edu/)__   _
- Define baseURL: _http://www.foo.edu/__modoai_
**Result: web harvesting with OAI-PMH semantics (e.g., from, until, sets)**

- mod_oai implementation(JCDL 2007, IWAW 2007, D-Lib 2008)

**http://www.foo.edu/****modoai****?verb=ListRecords****&metdataPrefix=oai_crate****&from=2007-09-21****&set=mime:video:mpeg**

- Uses a public (or private) Sitemap for the definition of W (used to be dynamic file system inspection);
- create the Sitemap as the union of multiple tools

- NDIIPP Partners Meeting

- July 9, 2008

## CRATE

- <crate>
-   <http request headers>
-   <http response headers>
-   <metadata>
-       <name>
-       <version>
-       <output>
-   </metadata>
-   <metadata>
-       <name>
-       <version>
-       <output>
-   </metadata>
-   <metadata>
-       <name>
-       <version>
-       <output>
-   </metadata>
-   <base64’d resource>
- </crate>

- JVBERi0xLjIKJcfsj6IKNiAwIG9iago8PC9MZW5ndGgg....

- Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
- Accept-Language: en-us,en;q=0.5
- ...

- Content-Length: 101702
- Content-Type: image/jpeg
- ...

- If you like to look at lots of XML, go to:	http://www.cratemodel.org/modoai/?verb=ListRecords&metadataPrefix=oai_crate
- Intuition: now is the best time to describe a file, and the local user has a good idea as to what should be said (does not preclude further analysis by harvester)

- NDIIPP Partners Meeting

- July 9, 2008

## CRATE and the OAIS Information Model

- Base64-encoded resource

- CRATE

- Metadata from plug-ins:
- Summary, index, format analysis...

- OAI-PMH MPEG-21 DIDL Metadata Format

- MIME /
- GDFR Type
- Copyright
- Originator
- Description

**SIP**: original web resource as it exists on the web site
**AIP**: resource processed by mod_oai for metadata &
**DIP**: disseminated to crawler; to other repositories; to  an information archeologist for research/extraction

- NDIIPP Partners Meeting

- July 9, 2008

## CRATE: Apache Configuration File

- NDIIPP Partners Meeting

- July 9, 2008

## Tested CRATE Plug-Ins for mod_oai

| Name | Description |
| --- | --- |
| Exif | Image/video metadata extractor |
| Jhove | Image analysis |
| DC | dcTag html extactor |
| Droid | Pronom registry info |
| MetaX | Meta-extractor |
| OTS | Open Text Summarizer |
| wc | unix word count utility |
| file | unix file utility (magic cookie) |
| md5, sha | unix md5sum, shasum utilities |

- NDIIPP Partners Meeting

- July 9, 2008

## Time Required to CRATE Web Site(ECDL 2008)

- Server response time to other web requests: < 2% throughput delta

- Time Required to CRATE Web Site(ECDL 2008)

- NDIIPP Partners Meeting

- July 9, 2008

## Future Work

- OAI-ORE support
- CRATEs as Resource Maps
- Defining CRATEs as an http encoding format
- like gzip, zip, etc.
- can return a CRATE in response to a regular http request with appropriate q values (not just OAI-PMH harvest request)
- Third party metadata
- how can my web server use your installation of Jhove?
- Tighter http log / Sitemap integration:
- “Sitemap strict” -- don’t serve a file unless it appears in a Sitemap
- “Sitemap synch” -- in real-time, add/delete entries in Sitemap based on 200 / 404 responses

- NDIIPP Partners Meeting

- July 9, 2008

## For more information

- More info, code:
**http://www.modoai.org/**
**http://code.google.com/p/modoai/**
- A joint research project between:
- Old Dominion University and
- LANL Digital Library Research & Prototyping Team
- Research supported by the Andrew Mellon Foundation & the Library of Congress

- NDIIPP Partners Meeting

- July 9, 2008