# Presentation: 623597

## Library of Congress National Digital Information Infrastructure and Preservation Program (NDIIPP)

- GDFR Governance Meeting at NARA
- November 14, 2007
- Caroline Arms (_[caar@loc.gov](mailto:caar@loc.gov)_) and Carl Fleischhauer (_[cfle@loc.gov](mailto:cfle@loc.gov)_)

## Background

- National Digital Information Infrastructure and Preservation Program established in 2000
- Originally, $175 million available with matching
- Using the money to fund partnerships
- Report to Congress in 2009
- Have 100+ partners

**Notes:**
SECTION ONE: OUR CURRENT PARTNERSHIPS

The National Digital Information Infrastructure and Preservation Program--NDIIPP--is defined for a decade, 2000-2009.  We are planning out the “concluding” report we will present to congress in 2009, but we have no doubt that the underlying activity will continue.

During this period, we have assembled a group of partners, about 100 total, organized in about 30 projects with a lead participant in each.

We tell this story since aspects may bear on today’s topic.  Some things are similar, some are different, and we need to compare and contrast.

## Leaf node picture

**Notes:**
The partners’ projects represent a mix of activities: policy and standards, including consideration of rights issues; examining how to sustain preservation (organizationally, financially); selecting and preserving at-risk digital content; developing supporting technology.  Any given project may tackle one or more of these activities.

## US map

**Notes:**
These are _partnerships_: we provide some funding, they also commit to providing funds and/or in-kind.  Two elements emerged when were setting them up. 

First, there was the challenge of establishing formal relationships, most with universities or other agencies in the federal or state government, several with non-profits, trade groups, etc.  There was a very high level of effort to get to the point where the papers were signed. Many are cooperative agreements, a few take a form more like a procurement contract.

## Slide 5

**Notes:**
Second, we had foreseen a dynamic in which the partners’ own energies and inclination led them to coalesce into one or more networks.  There has been a bit of that, but it is also the case that each partner had its own motivations to participate, which may have been only partially shared with others.  Regarding collective action, we found that we, as convener (and funder), encountered an expectation that the Library of Congress would lead in pulling together a organizational network.

## International Internet Preservation Consortium

- Enable the collection, preservation and long-term access of Internet content around the world.
- Foster development and use of common tools, techniques and standards for the creation of  international archives.
- Advocacy and outreach
-  _http://netpreserve.org/_

**Notes:**
The partnerships do vary: here is one that we call an “organizational alliance”:  the International Internet Preservation Consortium (IIPC).  This a group of national libraries and archival institutions established to
-- collect, preserve, and provide long-term access to Internet content
and to
-- foster the development and use of tools, techniques and standards
and
-- to do some advocacy and outreach.
It may have some interesting parallels to the GDFR.

## Deliberately adopted phased approach to organization

Phase 1
12 institutions (national libraries and archives)
lightweight agreement
work together on tools and standards 
3-year horizon
1 organization (BNF) provided administration through single staff member
 coordinated budget,  technical program, and communications 
salary paid by BNF 
As little funded centrally as possible (projects, some meeting costs) 

- Phase 1
- 12 institutions (national libraries and archives)
- lightweight agreement
- work together on tools and standards
- 3-year horizon
- 1 organization (BNF) provided administration through single staff member
-  coordinated budget,  technical program, and communications
- salary paid by BNF
- As little funded centrally as possible (projects, some meeting costs)

- International Internet Preservation Consortium

**Notes:**
The IIPC developed its organization– to address needs already identified as shared -- in deliberate phases:
During Phase 1, the dozen institutions were bound by a lightweight agreement.  They worked together on tools and standards with a 3-year horizon.  During this period, the BNF—the national library of France--provided administration through a single BNF-salaried staff member, who coordinated budget, technical program, and communications.  There was very little that was funded centrally apart from commissioning some third parties to work on technical tasks.

## Phase 2
more formal structure and agreement
same scope, same limited central funding
opening up to new members
Chair
1-year term to allow participation by smaller national libraries
Two office positions to be filled (and funded) by member institutions for 3-year terms
technical officer, 3/4 time
communications officer, 1/3 time
Fees based on institution’s budget

- more formal structure and agreement
- same scope, same limited central funding
- opening up to new members
- Chair
- 1-year term to allow participation by smaller national libraries
- Two office positions to be filled (and funded) by member institutions for 3-year terms
- technical officer, 3/4 time
- communications officer, 1/3 time
- Fees based on institution’s budget

- International Internet Preservation Consortium

**Notes:**
During Phase 2, there is a more formal structure, with the same scope and objectives.  It was opened up to new members.  The chair was set to be a 1-year term to allow participation by smaller national libraries.  Two office positions were established, to be filled (and funded) by institutions for 3-year terms.  These consisted of a technical officer at 3/4 time, and a communications officer, at 1/3 time.  Institutional membership fees are on a sliding scale based on total institutional budget.

## Technical underpinnings
not building an interoperable system
copyright laws in most jurisdictions are deemed to prohibit public access
the dependencies of members is tied to reliance on the tools they build
this makes them comparable to open software projects like DSpace, Fedora, and LOCKSS
DSpace, Fedora, and LOCKSS have membership organizations to fund ongoing development
governance models are different

- not building an interoperable system
- copyright laws in most jurisdictions are deemed to prohibit public access
- the dependencies of members is tied to reliance on the tools they build
- this makes them comparable to open software projects like DSpace, Fedora, and LOCKSS
- DSpace, Fedora, and LOCKSS have membership organizations to fund ongoing development
- governance models are different

- International Internet Preservation Consortium

**Notes:**
Meanwhile, thinking of technical underpinnings, the IIPC is not building an interoperable system, at least not yet, since copyright laws in most jurisdictions are deemed to prohibit public access to the web archives its members are accumulating.  The dependency of IIPC members on their consortium’s activities is tied to the reliance on jointly produced tools.  This is comparable to the dependency that other institutions might have on open software projects like DSpace, Fedora, and LOCKSS.  These latter three projects formed membership organizations to fund ongoing development once their success put them out of scope for research or seed funding.  Each has a distinct governance model.

## Partners and format registries

**National Geospatial Data Archive (UC Santa Barbara and Stanford University)**
- Developing a registry, wiki-based, human readable
- http://ngda.library.ucsb.edu/format/
**Portico, archiving service for scholarly literature published in electronic form**
- System functionality includes list of formats used to drive automated validation
- Accommodates fine granularity of versioning for publisher DTDs
- _[http://www.portico.org/](http://www.portico.org/)_, especially relevant is http://www.portico.org/news/Portico_DLF_Fall2005.pdf

**Notes:**
Some other NDIIPP partnerships provide good testimony to the need for a format registry.  At least four have made registry moves of their own, developing interim capabilities until the GDFR comes along.  Here are two specialized examples: for geospatial data and for a particular class of textual content.  These two archives are different in architecture, content scope, and indeed purpose, but share the need to identify and refer to formats and their associated specifications.  The teams offer essential domain-specific expertise on certain formats but would like to build on the work of others where the scope does overlap, e.g., if an article is accompanied by GIS data, or for widely used formats such as PDF. 

## Partners and format registries

**Drexel University, College of Engineering**
- Aiming at registry of CAD and other formats for engineering design repositories
- Designed for system access, using RDF
- Project activities connect to NIST (LTKR)
- _[http://digitalpreservation.wikispaces.com/LTKR+2007+Call+for+participation](http://digitalpreservation.wikispaces.com/LTKR+2007+Call+for+participation)_
**FOrmat CUration Service (FOCUS) **
- Demonstration project from University of Maryland Institute for Advanced Computer Studies (UMIACS),
- Designed for system access, using LDAP
- Drives automatic services based on tools (readers, editors, converters) that can process particular formats
- http://www.umiacs.umd.edu/research/adapt/focus/

**Notes:**
And here are two more: one is a design for a specialist registry for engineering – BTW: Josh Lubell, who is here from NIST, is familiar with this activity.  The other—FOCUS--was mentioned earlier by Steve Abrams, and it is a demonstration project that incorporated ten commonly used formats into its registry.  These registries are intended as components in a suite of curation or archiving tools.  A key to this curatorial work is the provision of different services supported by the ability to refer to formats unambiguously through a format registry.  

## Are other models instructive? 

- Specialist data archives & access services
- ICPSR -- social science data
- Institutional members (paying) and board
- International Virtual Observatory Alliance
- Loose association of funded national virtual observatories – corpora of sky surveys

**Notes:**
Although only one of these is an NDIIPP partner, we found ourselves thinking about two specialist data archiving & access services, organizations that play an important role in their respective domains.  One--the partner--is the ICPSR, devoted to social science data.  Based at the University of Michigan, it has paying institutional members and a board.  The other is the International Virtual Observatory Alliance, a loose association of funded national virtual observatories.  Both activities have active researchers willing to justify the associated costs.

[The challenge for GDFR is that the real beneficiaries are in the future.]

## Analysis of Digital Formats

- Provide inventory of information about formats
- Identify and describe the formats promising for long-term sustainability
- Synergy with Global Digital Formats Registry
- http://www.digitalpreservation.gov/formats/

**Notes:**
[SECTION TWO: ABOUT THE NDIIPP FORMATS WORK.]

As we in NDIIPP saw that the ability to manage formatted content would be critical to our partners, we saw added value for the format sustainability Web site we had been compiling.  It was originally launched to support planning within the Library of Congress, and we have begun the process of assessing formats in terms of their fitness for long-term sustainability.  

## Sample format description, top

**Notes:**
Our online description pages start off with a general description and indications of relationships to other formats.  In this example the MPEG-4 file format is a subtype of the ISO Base Media File Format, and it has a number of subtypes -- files that contain one or another video encoding, most of which are profiled in the specifications.

We want this work to complement GDFR, and we have been working with Steve Abrams, nowhere more closely than in the matter of relationships.  The GDFR data model allows for links to resources like ours, describing them as assessments to aid digital-content curators.

## Sample format description, middle

**Notes:**
Then our pages go on to highlight the factors that we think appropriate for comparing and contrasting the formats, to help curators assess their fitness.  These rows provide information about our seven sustainability factors (and one “sub-factor”).

## Sample format description, middle

**Notes:**
These rows provide information about quality and functionality.  This is where domain expertise can be essential.

## Sample format description, bottom

**Notes:**
And, at the bottom, the page lists relevant specifications or other information sources.

## A Network of Networks

- NDIIPP aims to form a national community of practice for digital stewardship
- Organization sense of *network*
- Best approach: engage effective domain networks and leverage their success

**Notes:**
SECTION THREE: THE FUTURE EVOLUTION OF A STEWARDSHIP NETWORK

An NDIIPP goal for the future is to expand and evolve the current network of partners into something larger, more formal, and more permanent.  There is no official name but we sometimes call this a stewardship network: it is first about content preservation and organizational, and secondarily about technology.  (Of course, the GDFR is viewed as a technology that will be important to the stewardship network.)

## Bulls eye diagram

**Notes:**
We and our partners have been discussing some ideas about this future network, one of which is governance, which can be seen as a part of the more general matter of roles and responsibilities.  The conversations yielded two diagrams.  This set of concentric circles enumerates some of the possible actors and the roles they may play, and suggests some of the elements that will be needed to support the actors in their work.  We are not sure if network management is the right name for the red circle in the middle -- sometimes we think of content in the middle and illustrating network management as an outer ring . . . .   but nevertheless, you can see that the NDIIPP group, like the GDFR group, is working on governance.  How will policy be determined?  How centralized do management functions need to be?

## Sample Layers in a Stewardship Network 

- Committed Content Custodians

- Information & Expertise, Development & Dissemination

- Services

- Capacity Building

**Sample Layers in a Stewardship Network**** **

**Notes:**
The layers diagram was developed to show how different organizations may play rather different roles at different scales.  Each organization is a cylinder, in a vertical orientation, penetrating the layers.  There are big cylinders and small cylinders, i.e., big and small organizations . . .  And some appear on one layer, some on two or three or four.

## Stakeholders in GDFR oversight

- Committed content custodians relying on GDFR services
- Experts in formats for a variety of domains
- Entities running GDFR nodes or interoperating services
- Capacity builders - other funding sources
- GDFR developers and central team
- How much or how little needs:
- a new governance structure
- exchange of $ rather than contribution in kind or of effort

**Notes:**
Studying these diagrams led us to think a bit about the GDFR and its stakeholders.  The GDFR is different from most standards organizations and most software development activities because it needs to provide a reliable set of _services_ and must continue to exist over the long term.

The layers for the stewardship network suggested varied stakeholders for GDFR:
-- Committed content custodians who rely on GDFR services
-- Experts in a variety of domains
-- Entities running GDFR nodes (or interoperating registry services)
-- Capacity builders, i.e., other funding sources

Then there are the GDFR developers and the central team, in effect what follows on from the development work by Harvard and OCLC and the resulting system.  All of these might have an interest in participating in oversight.  Some might be willing to contribute effort; others to support the service financially.  This meeting is a chance for us to talk about management and financial scaling: How much or how little will be needed to have an appropriate governance structure?  To what degree might exchanges of funds be needed (with the resulting need for formal agreements) rather than staff time or in-kind contributions absorbed by participants?

## A few final observations

- From varied experience
- Easy for pursuit of the perfect to get in the way of the “good enough to start with”
- Cannot ignore the politics of institutions or the motivations of individuals
- Need energy at the center or from a few individuals
- Find the individuals building systems that will take advantage of the GDFR and the institutions that are funding those systems
- Eventually, firm rules for resolving differences and procedures for proposing change will be needed, but that will be a sign of success
- Can’t help thinking that a deliberately phased approach might be appropriate

**Notes:**
If it were easy to envision the perfect governance structure for GDFR, it would have been done already.

Some observations from varied experience . . .
	Easy for pursuit of the perfect to get in the way of the “good enough to start with”
	Cannot ignore the politics of institutions or the motivations of individuals
	Need energy at the center or from a few individuals 
	Find the individuals building systems that will take advantage of the GDFR and the institutions that are funding those systems
	Eventually, firm rules for resolving differences and procedures for proposing change will be needed, but that will be a sign of success
	We can’t help thinking that a deliberately phased approach might be appropriate

## Thank you very much . . . .