# Presentation: 415696

## Web-at-Risk Project

- A Distributed Approach to Preserving our Nation’s Political Cultural Heritage
- Kathleen Murray – University of North Texas
- Tracy Seneca – California Digital Library
- Fall Federal Depository Library Conference &Depository Library Council Meeting
- October 2005

## Contents

- Project Overview
- CISA Path
- Project Status

**Notes:**
This is just the contents of the presentation. Can speak to who will cover what sections.

##  Web-at-Risk Goal

- 	Web Archiving Service (WAS): Develop web archiving tools that will be used by libraries to capture, curate, and preserve collections of web-based government and political information

**Notes:**
In a nutshell, the goal of this grant is to develop tools that librarians can use to create collections of web-based government information.  We're focusing on government and political information in this project--information produced by U.S. state, local, and federal governments and by local political activities and movements, such as the California gubernatorial recall election of 2003.

As a result of this project, the curators involved in using these tools will create discrete collections while also assessing the collection planning guidelines and actual archiving tools that we develop.  Better understanding of the issues involved in web archiving is another outcome of the project.  At the end of the 3-yr grant, we will look at extending these tools beyond the curators involved under this grant to other institutions.

## The Library of Congress

**The Library of Congress**
- Martha Anderson, Agreement OfficerTechnical Representative

**California Digital Library**(Lead Partner)
- Dan Greenstein, P.I.Peter Brantley, Director of TechnologyLynne Cameron, Manager, Ingest GroupPatricia Cruse, Director John Kunze, Technical LeadShifra Raffel, Web Archive ProgrammerTracy Seneca, Web Archive CoordinatorStu Sugarman, Arch. and Insfrast. Manager

**University of North Texas**
- Cathy Hartman, Fellow, Texas Center for Digital KnowledgeKathleen Murray, Assessment Analyst

**New York University**
- David Ackerman, Executive Director of eServicesRasan Rasch, Web Archive Programmer

- San Diego
-  Supercomputer Center

- Computer ScienceDept. Stanford University

- Sun Microsystems

**Selection and Curator Agents**
**Arizona State Library****New York University****Stanford University****University of California:** UC Berkeley, UC Davis, UC Irvine, UCLA, UC Riverside, UC Santa Barbara, UC Santa Cruz, UC San Diego**University of North Texas**

- NDIIPP: Web-at-Risk Project

**Notes:**
We have two main partners in the project: the University of North Texas and NYU.  
UNT brings experience with their Cyber Cemetery project,a service that works to guarantee permanent public access to the Web sites and publications of defunct U.S. government agencies and commissions. 
NYU brings web archiving experiences to the project (CRL’s political communication project), experience with METS, and experience collecting political information. =
Also involved are 10 curatorial partners, including government information librarians from the UC system, UNT and NYU, and the University of Arizona.  These curators will be building collections and helping us develop our curatorial tool kit by providing valuable feedback.  
Our technical partners--SDSC, sun, and Stanford Computer Sci dept--will help us as we grapple with some of the issues involved with storage and other technical components. 

## Four overlapping paths of activity:

- Content Identification and Selection CISA)
- Content Ingest, Retention and Transfer (CIRT)
- Content Harvest and Analysis (CHA)
- Partnership Building (PB)

**Notes:**

Start by reviewing how the grant is laid out  
The goal stated earlier is somewhat simplified... the grant overall aims to do more than just build the tools.  These four paths also cover developments in standards, user needs assessment work and more.

The grant looks at web archiving and preservation in totality, from gaining a theoretical understanding to building practical tools.

## Content Identification, Selection, &    Acquisition

- Needs assessment: surveys, focus groups and interviews
- Curator (librarian) needs
- End user needs
- Analysis of test crawls based on curators’ content
- Case studies, best practices, and collection development guidelines

**Notes:**
Kathleen will cover the details of this work later in the presentation.

This path began working intensively early on to provide data to inform the other project paths.

## Content Ingest, Retention, and Transfer

- Modify CDL’s Digital Preservation Repository
- Data model for Web Archive Digital Objects (WADO)
- Development and testing of remote replication strategies

**Notes:**
The Web Archiving Service is being built on CDL’s existing Digital Preservation Repository.

(Explain DPR – link to screen?)

Describe the new kinds of data we will need to account for unique to web archiving needs

Need to transfer data to Library of Congress, and to other archives with potentially quite different architectures.   

## Content Harvest and Analysis

- Tools to conduct and support curators' web crawling: Curator User Interface
- Crawler
- Collection Management
- Reports
- Export/Import Handler
- File formats for web crawling: WARC

## Partnership Building

- Develop model agreements (partner MOUs)
- Assess costs of sustaining a distributed approach to web archiving

**Notes:**

Data retention agreements: These agreements will outline the parties’ roles and responsibilities with respect to those collections and the terms and conditions applying to the collection for deposit, withdrawal, and use.
Data exchange agreements: will be signed by the California Digital Library and each of the partner institutions (New York University and University of North Texas) that back up the archival collections managed in the repositories maintained by the other two partner libraries.  Agreements will document roles and responsibilities; also define the terms and conditions of access to backup collections.
Data capture agreements will be required by partner libraries who use crawler technologies that allow data producers to “push” their web-based information using crawler hints tools to archiving institutions. The agreements will outline the parties’ roles and responsibilities in supplying and using the push technologies. They will also deal with the terms and conditions of access that the data owner attaches to archived content supplied in this way
Challenges and Risks: 
Different institutions with different practices. Mitigation – we will be clear, succinct and avoid language specific to UC.
Rapidly changing costs in digital preservation arena. Mitigation -- cost assessment will cover approaches and analytical information as well as specifics to allow for as much extrapolation as possible.

## CISA Path: Needs Assessment

- Activities
- Status
- Future Activities

**Notes:**
This is a transition slide between speakers: Tracey ends & Kathleen begins here.

## Needs Assessment Activities

**Collection**
**Development**
**Process**

**Needs Assessment **
**Toolkit**

**Data Collection       **

**6/05**

**2/05**

**Common Collection**
**Planning Framework**

**11/05**

**3/06**

**1/05**

## Collection Development

| Policy Setting
 | Political mandates, organizational mission, financial parameters, & technical capabilities. |  |
| --- | --- | --- |
|  | Selection | Factors: Focus of the collection, unit of selection, web boundaries, copyright obligations, and authenticity of materials.  |
|  | Acquisition | Requirements for crawling tools: Global or selective capture.  |
|  | Description | Baseline metadata: Machine-generated
Enriched metadata: Specific to an organization; both human-generated & machine-generated metadata.  |
|  | Organization | Considerations: Retain or modify the organizational structure of the materials as they existed on the web.  |
|  | Presentation | Considerations: Mirror the web at the time of their capture or selectively present (searching & browsing).  |
|  | Maintenance | Functions: Training, hardware and software maintenance, performance optimization, backups, upgrades, & duplicate detection.  |
|  | Deselection | Reasons: Duplication, errors, legal or social considerations. |
|  | Preservation | Challenges: Persistent naming, format migration and/or emulation, inventory management, volatility, replication, re-validation, & storage.  |

## Needs Assessment Toolkit

- Responsibilities
- Timeframes
- Participants
- Guidelines
- Consent Forms
- Instruments
- Survey Instrument
- Focus Group Guide
- End User Interview Guide
- Content Provider Interview Guide

- Available:
- _[http://web2.unt.edu/webatrisk](http://web2.unt.edu/webatrisk)_

## Data Collection Activities: 2005

|  |  | Jun | Jul | Aug | Sep | Oct |
| --- | --- | --- | --- | --- | --- | --- |
| 1 | Survey | X | X |  |  |  |
| 2 | Focus Group: National | X |  |  |  | X |
| 3 | Focus Group: Local |  |  | X | X |  |
| 4 | Interviews: End Users |  |  |  | X | X |
| 5 | Interviews: Content Providers |  |  |  | X | X |

## Survey Status

- Purpose
- End Users’ Needs
- Curators’ Needs
- Functional Requirements
- Web Crawler Tool
- Online Survey
- 5 Sections
- 59 Questions

- Respondents (N = 16)
- Curatorial Partners
- University of CA
- New York University
- University of North TX
- Stanford University
- AZ State Library

**Notes:**
Kathleen will add her slides here.

## Focus Group Status

| PARTICIPANTS | LOCATION | N = 44 |
| --- | --- | --- |
| Special & Academic Libraries | Chicago, IL | 8 |
| University of North Texas Libraries | Denton, TX | 7 |
| University of California & 
Stanford University Libraries | Oakland, CA | 10 |
| New York University Libraries | New York, NY | 8 |
| Public, State, & Academic Libraries  | Washington, DC | 11 |

## Interview Status

| INTERVIEWS | PARTICIPANTS | PURPOSE |
| --- | --- | --- |
| Face-to-Face | Researchers (N = 9–15)
Faculty
Students | End User Access Needs |
| Telephone | Web Publishers (N = 6-9)
Government Agencies
State, Regional, & Local
Political & Labor Organizations | Needs
Concerns |

## CISA: Future Activities

## Project Status 

- Test crawls being conducted
- Government sites provided by curators
- Hurricane Katrina
- Rights management
- Web-at-Risk rights management protocol

**Notes:**
Add link to Katrina crawl results?

Discuss current rights management plans

## Rights Management

- 3 rights management schemes
- A: consent assumed
- B: consent sought
- C: consent required
- Tools for recording rights contact activity

## 24 Sample Sites for Test Crawling

- 10 are .gov
- 3 federal
- 3 local govt
- 4 state agencies
- 3 are .com
- 1 of these is for a county agency
- 11 are .org
- One .org site contains warning that material may not be reproduced in any format without permission.

## Rights Oddities

- 3 different curators selected sites that are associations of local governments.  Two of those (San Diego and Sacramento) are .org sites, and one (Southern California) is a .gov site.
- http://www.ocsd.com  City agency with a .com address that states that all of its material is in the public domain.
- http://www.city.davis.ca.us: A city site that aggressively asserts copyright and demands permission for any reproduction.

## Rights Classification

- In 5 cases, I changed the original classification of the site after I read the copyright warnings/info on the site itself.
- I determined that if I were curating these sites, I would classify their rights as follows:
- Scheme A: Consent implied: 6 sites
- Scheme B: Consent sought: 14 sites
- Scheme C: Consent required: 4 sites

## Crawling Tools 

- Heritrix
- Contributing code back
- WERA for display

**Notes:**
Show Heritrix user interface

## More info:

- http://wiki.cdlib.org/WebAtRisk