# Presentation: 290349

- Digital Preservation at Scale

- NARA Partnerships in Innovation II

## Digital Preservation at Scale:A Case Study from Portico	

- Evan Owens, Chief Technology Officer
- www.portico.org

## Scale Matters

*Scale is a mysterious phenomenon -- processes that work fine at one scale can fail at 10 times that size, and processes that successfully handle a 10-times scale can fail at 100 times. [...] Institutions offering tools and systems for digital preservation should be careful to explain the scale(s) at which their systems have been tested, and institutions implementing such systems should ideally test them at scales far above their intended daily operation, probably using dummy data, in order to have a sense of when scaling issues are likely to appear. *
- Clay Shirkey, Library of Congress Archive Ingest and Handling Test (AIHT) Final Report, June 2005, page 26
- http://www.digitalpreservation.gov/library/pdf/ndiipp_aiht_final_report.pdf

- Digital Preservation at Scale

- NARA Partnerships in Innovation II

**Notes:**
Scale Matters.   

Quote to start us off, by Clay Shirkey from the AIHT final report.   
I would add that these comments apply to human as well as automated processes

Conference organizer asked us to talk about lessons learned from operating at scale. 
I’m going to do that in two ways:  getting to scale and running at scale

## Portico Business Summary

- A permanent archive of scholarly literature in electronic form
- All preservation and access rights secured by irrevocable contractual agreements
- Initial content area is E-Journals
- Start-up funding by Andrew W. Mellon Foundation, JSTOR, Ithaka, and Library of Congress NDIIPP
- Portico Stats (as of 9/22/08)
- 61 participating publishers
- 7,979 journal titles committed
- 29,000 e-book titles committed
- 469 participating libraries from 13 countries
- 8,073,180 articles archived; >14M articles committed
- 85,352,259 files
- 132 file formats
- 670 GB of preservation metadata
- Currently capacity is up to ~1 million articles per month
- Generating up to 90 GB of METS/PREMIS/JHOVE metadata

- Digital Preservation at Scale

- NARA Partnerships in Innovation II

**Notes:**

61 participating publishers; depends on how you count actual number is very much higher; e.g., BioOne is 1 in our list because there is one contract; other such as CrossRef might count BioOne as 60 publishers

## Portico Technology Summary

- OAIS-compliant repository designed for managed preservation
- Key influences:
- OAIS, GDFR, PreMIS, METS, MPEG-21, ARK
- Key technologies:
- XML, XML schema, Schematron, JHOVE, NOID
- Documentum, Oracle, Java, JMS, LDAP
- Format Registry
- Archive design goals:
- Content preserved in application-neutral content using open standards
- METS, PREMIS, JHOVE
- A “Bootstrapable Archive”
- XML plus digital objects
- Ingest system design goals:
- Pluggable tools to facilitate new providers and replacement tools
- Configurable workflows for different content types
- Scalable to very high content volumes...*in theory and now in practice*

- Digital Preservation at Scale

- NARA Partnerships in Innovation II

**Notes:**
NOTE: TWO SEPARATE SYSTEMS --- CONPREP   and ARCHIVE

Designed in 2004-2005; operation March 2006

Designed/built while PreMIS was being invented

Disadvantages to being early . . . Ahead of standards 

## Portico Preservation Policies

- Format-based migration strategy
- Driven by Portico Format Registry
- Preservation policies:
- Fully supported
- Reasonable effort
- Byte-preserve only
- Preservation policies based on
- Format validity
- File format action plans and archive capabilities
- Business rules such as publisher preferences
- Archive must also preserve supporting information
- Required files such as DTDs and entity files
- Documentation
- Contracts
- Archive policy documents
- Archival actions documents

- Digital Preservation at Scale

- NARA Partnerships in Innovation II

## Portico Systems Overview

- Digital Preservation at Scale

- NARA Partnerships in Innovation II

**Notes:**
Losely coupled systems.

SCALING CONCERN WAS IN CONTENT PREPARATION SYSTEM

AND TO A LESSER EXTENT IN THE INGEST MODULE OF THE ARCHIVE

ARCHIVE ITSELF WAS NOT THE BOTTLENECK

## Initialization and 

**Layer Removal**

**Content Unit Identification**

**Apply Policies**

**Content Component Identification**

**Metadata Curation**

**Characterization & Validation**

**Receive Content**

**Create Batches**

**Schedule Batches**

**Automated Processing**

**Quality Assurance**

**SIP Creation**

**Archive Ingest**

**Load into Archive**

**Verify Contract ID**

**Validate Checksums**

**Check Format  ID & Preservation Level**

**Validate Asset Inventory**

**Add Ingest Event to Portico METS**

- Digital Preservation at Scale

- NARA Partnerships in Innovation II

**Notes:**
Drill down into the content processing steps.

Orange is ConPrep
Automated steps on left are really 30+ workflow activities

Ingest is in blue on the right

## Portico Content Processing

- Inputs
- Per unit of content (e.g., article):
- one text or metadata file, zero or more other files
- Arbitrary (provider-specific) collections of data
- Standard or proprietary file & directory naming conventions
- Standard or proprietary formats
- Undocumented business rules hidden in the data
- Outputs
- Content packaged in Portico METS
- Metadata: technical, descriptive, events
- ~100 GB of metadata for every 1 TB of content
- E-Journal content averages 10 million files per 1 TB
- Content restructured to Portico content model
- Article component structure documented
- Content normalized as per preservation plans
- Proprietary publisher formats converted to NLM Archival DTD
- PDF created from TIFF as needed
- Processing Requirements
- Vary according to formats and number of files in batch

- Digital Preservation at Scale

- NARA Partnerships in Innovation II

## Result: 100s of Gigabytes of Preservation Metadata(NB: syntax will change in 2009)

- Digital Preservation at Scale

- NARA Partnerships in Innovation II

**Notes:**
In this case, 4449 lines of XML to describe one e-journal article. 

And that is with the JHOVE technical metadata already seriously trimmed . . . 

## Scaling Portico Ingest: August 2006 to May 2007

- As of Summer 2006
- System live since March
- Capacity around 75K articles / month on existing hardware
- In theory scalable but hitting performance roadblocks
- Business Requirements
- Signing publishers faster than expected
- Signing more large publishers than expected
- Goals: 4 to 6 million new articles (40 to 60 million files) ingested into the archive
- Increase total capacity to 10M articles / 100M files per year
- How to scale the system?
- Raise the speed limit
- Add more lanes to the highway
- Reduce traffic jams
- Software? Hardware? A combination of the two?
- Test data to the rescue!
- See http://www.diglib.org/forums/spring2008/presentations/Owens.pdf

- Digital Preservation at Scale

- NARA Partnerships in Innovation II

## High-Level Content Preparation System Overview

- Documentum
- CMS

- Workflow Instance

- Workflow Instance

- Workflow Instance

- Workflow Instance

- Workflow Instance

- Workflow Instance

- Oracle
- Server

- Workflow Instance

- Workers

- Workers

- Workers

- Workers

- Workers

- Workers

- Workers

- JMS
- Queue

- Digital Preservation at Scale

- NARA Partnerships in Innovation II

**Notes:**
Many details omitted:

UI, format registries, enterprise services, etc. 

Only showing pieces involved in the scaling . . . And pieces left after we simplified the architecture

## Software Plan

- Prioritized by bang for buck
- Rewrite of data persistence
- 45-50% performance improvement and allows further hardware scaling
- Replace Documentum attributes / folders with Oracle tables
- Original design abstracted Documentum at cost of performance
- New design is idiomatic Documentum application architecture
- Major shift in system resource demands from Documentum to direct Oracle connections
- Simplify system design
- Eliminate distributed tool processing in favor of embedded mode
- Move all tools into memory space of worker
- Eliminated 50% of JMS messages
- Eliminate trivial use of web services

- Digital Preservation at Scale

- NARA Partnerships in Innovation II

**Notes:**

Fast versus flexible --- early design more elegant but at cost of performance

## Hardware Plan

- Workers: T2000
- 32 hardware threads; in practice only ~15 usable
- Variable memory requirements for XML parsing, JHOVE, etc.
- Limited by Documentum 32bit libraries to 4GB of memory
- Limited by forking problems in Sun JVM
- Oracle: X4600
- Oracle load went from 90% to 50-60% while executions went up 20 fold
- Ramped from 250 executions / second to 5,000 / second
- 6 weeks of DBA time to tune
- Documentum: V440 (no upgrade)
- Repository scaled to 100M objects without any hiccups
- Limitation was performance on adds and deletes
- We add and delete up to 1M objects per day
- We rewrote the deletion routines

- Digital Preservation at Scale

- NARA Partnerships in Innovation II

**Notes:**
XML memory problems are discussed in the AIHT report

## Testing, Tuning, and Declaring Victory

- Balancing all the pieces
- # of Workflow instances in Documentum
- # of Workers (CPUs, JVMs, Threads)
- Performance peaked and then declined when too many threads were added
- Testing, testing, and more testing
- Staggered start, mixed content sizes and types essential
- Create random traffic effects
- Test both good and bad data
- Test on production equipment
- Can you afford to run a really large test?
- Tested > 1M articles, but not in a continuous run
- Found additional problems once system went live and ran for extended period
- How much tuning is enough?
- Cost/benefit analysis based on unknown future costs
- Hard to walk away from great ideas for further improvements

- Digital Preservation at Scale

- NARA Partnerships in Innovation II

**Notes:**
In parallel with oracle tuning

## System Impact of Scale

- Other Pieces of the System
- All the pieces of the system must be scaled up
- On ramps & off-ramps, not just the highway
- We had to rewrite our scripts, loaders, all the pieces around the edges of the system
- More than first expected
- System Admin
- New cleanup and maintenance jobs were necessary
- New deletion and purging routines
- New logging and log management
- User Interface
- Usability: scale may well prompt redesign
- Raw performance
- Storage / Backup / Recovery
- Differentiation becomes desirable, cost effective, and even essential
- Different types of disk for work areas, holding areas, archive
- Different backup strategies for each part of the system

- Digital Preservation at Scale

- NARA Partnerships in Innovation II

## Human Impact of Scale

- Human roles in our system
- Loading of content
- Quality Assurance for new content streams
- Problem Resolution during processing
- Scale changes everything
- Human tasks don’t scale gracefully
- Can’t look at 1,000 objects and see a pattern
- Can’t push a button 1,000 or 100,000 times
- Problem solving becomes much more technical
- New User Interface Requirements
- Reports, summary view, classifications
- Facilities to execute actions against sets of objects
- “My Batches”
- Hard for users to imagine the impact of scale
- Particularly for people used to the existing system
- Easier with an entirely new system
- Bring in UI consultants

- Digital Preservation at Scale

- NARA Partnerships in Innovation II

## Organizational Impacts of Scale

- It isn’t easy to keep a dragon well fed!
- Lead time in getting content organized and ready for ingest
- Challenges of mixing small and large content streams
- Can work for 2 months on 10K articles
- Or a week on 200K articles
- Note to charts on next two slides: Assumption is system would run at an average 50-75% of peak capacity, not 100%

- Digital Preservation at Scale

- NARA Partnerships in Innovation II

**Notes:**
With the processing capacity problem resolved,  the pressure moves to the operations staff and the logistics.

Fortunately we have a new director of operations with experience in running a complex and varied mix of projects, exactly what is needed to feed this monster.

- Digital Preservation at Scale

- NARA Partnerships in Innovation II

- Digital Preservation at Scale

- NARA Partnerships in Innovation II

## More Thoughts about Scale

- Scale affects everything before, after, and around the system
- Nothing is simple or quick in units of millions
- Batch processing redux:
- Schedule, wait, evaluate results, schedule again
- PC mode versus old mainframe batch processing mode
- What % of idle capacity is acceptable?
- Make sure you know how you are going to feed your dragon
- All parts of the organization have to on board with scaling up
- Technology alone cannot solve the problem
- Expect surprises!

- Digital Preservation at Scale

- NARA Partnerships in Innovation II

## Portico Lessons Learned

- Content is not perfect
- Software is not perfect
- People are not perfect
- Audit trail is essential
- Tool versions in event metadata helps trace problems
- Trial by fire early in life of archive
- CRL trial Audit report commended this feature of Portico systems
- Expect the unexpected!
- Fixity check revealed 30 minute gap due to hardware failure
- Recovered from copies and backups

- Digital Preservation at Scale

- NARA Partnerships in Innovation II

**Notes:**

Content is not perfect:
	Typesetter friend: question is not whether you have an error rate or not, it is what is it and how to keep it as low as possible.

## Portico Next Steps

- New Content Types
- E-books
- Digitized historical content
- Locally created content
- Portico Metadata 2.0
- Catch up with evolving standards
- Migrate 10M metadata records, 700M event records
- Support new content and metadata types
- System Enhancements
- Truly generic workflows
- Policy-driven preservation strategies
- Automated replication, item and collection fixity check
- Collection Management
- Assessing completeness of content received and ingested
- Preparing for TRAC Audit in 2009

- Digital Preservation at Scale

- NARA Partnerships in Innovation II