![](media/image1.jpeg){width="6.945138888888889in"
height="1.2152777777777777in"}

University of California, Santa Barbara

Stanford University

The Library of Congress

An Interim Progress Report

September 2004 -- October 2007

**Table of Contents**

Introduction to NGDA and NDIIPP

NGDA Principles

Technical Architecture

UCSB Implementation: Ingest, Server, and Clients

Stanford University Systems: Access, Ingest, and Storage

Format Registry

Metadata Treatments

Collections

Collection Development Policy

Rights Management and Contracts

Content Provider Agreement

Node to Node Agreement

Procedural Manual

**Introduction to NGDA and NDIIPP**

The National Geospatial Digital Archive (NGDA) was formed in 2004 as
part of the National Digital Information Infrastructure and Preservation
Program (NDIIPP). Concerned that millions of nationally important
digital information resources are in danger of being lost or corrupted,
the Library of Congress established a groundbreaking preservation
network, made up of public and private sector organizations, to save our
nation\'s digital heritage. The NGDA is one of the NDIIPP projects
working at archiving geospatial data and images that are at risk of
disappearing.

Digital geospatial imagery is now a critical tool for state and federal
researchers of complex matters, including disaster relief, census data
and land use management. It is used by federal agencies in their
antiterrorism work as well. The University of California at Santa
Barbara and Stanford University are working together to as the NGDA, a
collecting network for the archiving of geographic information produced
with spatial software. They have partnered to construct a nationwide
digital collection and preservation system. The NGDA has been preserving
geospatial imagery and aging maps. The NGDA recognizes that the
geospatial information provides critical tools for state and federal
researchers pursuing greater understandings of complex matters
including: disaster relief, census data, and land use management. Other
features of the project include a geospatial format registry, common
best practices for archiving digital geospatial data and policy
agreements among the partners.

The project utilizes the developed hardware for retention and
distribution of cartographic material of both universities: The
Alexandria Digital Library (ADL) and Stanford Digital Repository (SDR).
Both institutions also house significant collections of material for
inclusion. The NGDA system is designed to archive collections at UCSB,
Stanford, and elsewhere rather than at a single location.

The NGDA partnership brings a breadth to the archive's collection.
UCSB's focus is largely born-digital collections, such as those it
currently collects, e.g., LANDSAT imagery and other born-digital content
from university, corporate and government sources, and images and
reference data accessible at independent web sites. Stanford focuses on
historically-oriented geospatial collections and outreach to
organizations (particularly professional societies and state and local
agencies) unlikely or unable to assure preservation of their own
resources, including digital materials converted from analog sources.

NGDA initiated the project by defining an overall project goals
associated with preservation of digital geospatial information. Other
first steps in the project include: identifying the issues that need to
be addressed regarding long-term preservation in general, and
preservation of geospatial data in particular; specifying requirements
that must be met to achieve the overall project goal; and determining
where the scope of the project could be safely and most effectively
limited. The result of this work is encapsulated in the \"NGDA
First-year Roadmap"
([http://www.alexandria.ucsb.edu/\~gjanee/ngda/roadmap.html](http://www.alexandria.ucsb.edu/~gjanee/ngda/roadmap.html )).

The project's initial goals were to:

-   Create a new national federated network committed to archiving
    geospatial imagery and data.

-   Investigate the proper and optimal roles of such a federated
    archive.

-   Collect and archive major segments of at-risk digital geospatial
    data and images.

-   Develop best practices for the presentation of archived digital
    geospatial data.

-   Develop partner communication mechanisms for the project and then
    ongoing.

-   Develop a series of policy agreements governing retention, rights
    management,\
    obligations of partners, interoperability of systems, and exchange
    of digital objects.

The NGDA project has both research and development components. Research
topics include considerations for long-term preservation; collection
development, including prioritization and scope; architectural and
economic models; rights issues; and best practices. The two libraries
are developing prototype archives for housing the data and jointly
creating a geospatial format registry to describe the data being stored.
During the second year of the grant the two archives will be federated
using the Alexandria Digital Library (ADL) software interface (see
Figure 1).This interim report details NGDA's pursuit of these objectives
and related achievements and discoveries from first three years of the
project.

**NGDA Principles**

To better understand the nature of geospatial data and how it is
produced and managed in practice, and to gauge the extent to which and
the ways in which data providers can work with archive providers in
archiving their content, NGDA hosted a workshop of data providers.
Attendees (<http://www.ngda.org/meetings.php>) representing institutions
attended along with several storage vendor representatives. Out of this
workshop emerged the following principles:

-   Archival must be extremely cheap and easy to be achieved on any
    scale. Providers simply do not have the funding, staffing, and other
    resources (and frankly, do not have the mandate) to address many of\
    the issues surrounding long-term preservation.

-   Archival content will long outlast the archival system that
    currently contains it, and therefore archival systems must be
    designed to be easily migratable as a whole. Furthermore, archival
    systems should offer a complete, base-level representation of
    content that requires minimal to no migration to handle the
    inevitable situation in which resources are not available to perform
    a proper migration (e.g., the situation when an archive provider
    goes out of business).

-   Geospatial data differs from other types of content, such as textual
    documents and audiovisual materials, in that there is often
    information required for its proper understanding that is not, in\
    practice, stored along with the data or even the metadata. This
    information (e.g., sensor platform documentation in the case of
    remote-sensing imagery) must be archived along with the data.

-   The format registry, which is needed to identify and store archival
    object semantics, has emerged as a major piece of work that is not
    currently satisfied by any external efforts despite being required
    by virtually every other NDIIPP awardee. We have begun creating a
    registry that is itself an archive of format specifications and
    related materials.

**Technical Architecture**

NGDA looked closely at how UCSB and Stanford could best collaborate, and
best take advantage of each other\'s work and leverage other, ongoing
efforts. Major points of technical interaction will occur at two levels:
use of ADL as an access mechanism and unified interface across
federations of archives, with UCSB and Stanford being initial nodes in
the federation; and development and population of a shared registry of
geospatial formats and products.

The two repositories are being built using similar technologies while at
the same time meeting the specific needs of each institution. Both
architectures contain standards-based interfaces, clearly defined
metadata formats, an underlying format registry, a goal of end-to-end
automation of the systems, and exploration into open source front ends.
UCSB has developed a repository specifically to house geospatial
information, with tools and templates designed around common data
structures. Stanford is building a repository to hold all of its digital
content no matter what its nature; the goal is to determine if a general
digital repository can adequately handle the complexities of geospatial
data formats using standard metadata and a content transfer manifest,
which include provisions for geospatial information. As of the end of
December 2005, both repositories were complete through their first
stages and had ingested geospatial data.

![](media/image2.jpeg){width="4.854166666666667in"
height="3.4791666666666665in"}

*Figure 1.* The overarching project activities for the three-year life
of the NGDA contract with the Library of Congress
(<http://www.alexandria.ucsb.edu/~gjanee/ngda/project-activities.pdf>)\
\
UCSB\'s prototype archive is being built on an Archivas storage array.
UCSB has developed a simple data model in which digital objects (as
defined in the Kahn/Wilensky framework) are represented as directories
of files tied together by manifests represented as XML documents. Some
preliminary data model diagrams can be found at:
<http://www.alexandria.ucsb.edu/~gjanee/archive/2005/ngda-arch-update.pdf>.

Access to the archive will initially be provided by making archive
objects and archive components directly Web-accessible and crawlable by
search engines; later access mechanisms will include ADL and an OAI-PMH
metadata server to support creation of third-party access mechanisms.\
An ingest crawler and archival server are being developed with an
initial focus on ingestion by bulk crawling of large collections that
are sufficiently homogeneous as to allow description of their content\
by ingest templates.

Stanford\'s archive is being built on the Archivas storage array as
well, with a robust ingest process based on the development of the
Transfer Manifest, a METS-based specification for transferring
descriptive, technical, and structural metadata along with digital
objects. Objects will be packaged with their metadata within Archivas,
with a tape robot writing out copies to be stored at geographically
separate locations for redundant backup. Metadata is also being stored
separately within Archivas to allow full search capabilities, and to
allow the archiving of format specifications as determined by the format
registry and the Type C team. A reconstructor is being created to
control access (authorization, authentication, and bandwidth), allowing
those with the authority to be able to search and retrieve items from
the archive through a web-based, scriptable interface.

Stanford and UCSB will both be including ADL and OAI-PMH access layers
to allow federation of archives. Stanford will create an abstraction
layer within the reconstructor to allow multiple federation and
distribution interfaces, applying security restrictions as appropriate
to all access.

Finally, Stanford will also be doing format migration where possible, to
keep playability at a maximum for ingested content. This is codified by
a contract system, where contributors agree on a preservation contract
which dictates the level of preservation, and steps to be taken, to
ensure satisfaction on both sides of the transaction.

**UCSB Implementation: Ingest, Servers, and Clients**

**Ingest**

From the outset of the project, it was clear that data loading, or
\'ingest\', was an area that would require serious work and
investigation. Early on, several approaches to the problem were modeled
and considered. As a result of this work, it became clear that a
proactive \'pull\' approach, in which the majority of the work and
responsibility for ingest was placed on the archive instead of the data
provider, was best suited to the project goals of preserving at-risk
data.

With an approach firmly established, design and implementation began.
Design work focused on a bulk ingest tool, which seemed the most
efficient way to handle the types of data that had at that point been
acquired for the archive. A prototype bulk ingest tool was built based
on these designs. This prototype served as a proof of concept and gave
insight into what a fully featured ingest system should look like.
Development began on a new, more modular version of the bulk ingest
tool. This tool proved very successful, allowing for the ingest of
several test collections into the archive.

By this point, it had become clear that the original NGDA data model was
too sparse to meet the needs of the archive. A revised version of the
data model was drafted and adopted. This required all of the archive
software to be updated, and when coupled with hardware issues, required
all data loaded to be re-ingested. With a solid basis in the new data
model, more ingest utilities were developed. Prototype single-item
ingest tools were created. A workflow suite was created to allow for
user oversight and feedback during the ingest process. These tools are
now live, and the archive has enlisted the help of a metadata librarian
to catalog and assist in the ongoing ingest process.

**Server**

One of the first ideas seized upon during the NGDA project was that an
archive should be hardware independent; that it should be able to
migrate forward as technology changes. To achieve this goal, it was
necessary to separate the various components of the archive from the
underlying storage system. The NGDA server was designed to meet this
need. The server was designed and implemented as a intermediary between
data manipulators (like the ingest system) and the redundant, secure
storage (Archivas). The server was also designed to handle some system
capability checking tasks, such as ensuring data integrity by comparing
md5 checksums. The server was implemented and successfully deployed. It
was necessary to refactor some of the design when the NGDA object data
model was revised, but this was handled successfully. The server is
currently operational and is essential to the operation of the archive.

**Clients**

The NGDA vision of the modern archive places access on the same level
(or very near the same level) as preservation. Given the focus of NGDA
on geospatial data, it was decided that some serious investigation
should go into providing and improving a geospatial search interface.
UCSB\'s Alexandria Digital Library project was used as a starting point
for this investigation. Planning and design cycles were spent
investigating what would make a geospatial user interface truly
intuitive. As a result of this work, the \'Glide\' interface was
developed. The streamlined Glide interface met with very positive
reviews during user testing. However, Glide was constructed as a
prototype, and as such had fundamental issues with scaling and
maintainability. Taking the lessons learned from Glide, work was started
on a new interface, dubbed \'Globetrotter\'. The Globetrotter interface
was designed to take the best aspects of the Glide prototype and build
them on a solid base. Globetrotter is now available at:

<http://clients.alexandria.ucsb.edu/globetrotter/>

While spatial-based searching has been the main focus of user interface
work under NGDA, more traditional types of access have not been
neglected. A filesystem-based view of the archive, called
\'ArchiveView\', was created. This view into the archive was created to
serve two purposes. The first was to maintain a consistent look and feel
for the archive, despite any shifts in underlying storage. The second
was to provide a \'crawlable\' interface into the archive, so that
search engines could index the textual information that objects posses.
The ArchiveView service is currently operational.

**Stanford University Systems: Access, Ingest, and Storage**

**\
Stanford NGDA Technical Information**

The Stanford Digital Repository (SDR) was built on the premise that to
be successful, the NGDA would be required to accept varied storage
systems as nodes. Testing with identical nodes, and then bringing in a
new node of a completely different architecture was a daunting task.
Since UCSB was building a single-purpose storage system for data
downloaded for NGDA, Stanford built a robust, general archival
preservation repository to house the data collected by the Stanford
node.

To demonstrate federation, Stanford and UCSB agreed on UCSB's Alexandria
Digital Library (ADL) interface, a single-purpose interface for viewing
geospatial data. Since ADL requires metadata in a different format than
either NGDA node is using, Stanford created a system to populate a
database with ADL-format data. UCSB is currently working on doing the
same with their data, as well as extending ADL to make these
transformations easier. Because of this transformation, any location
with an ADL viewer can include Stanford's NGDA data in its search area.

The bulk of Stanford's NGDA data comes from the David Rumsey collection.
He originally presented Stanford with a large stack of DVDs full of map
imagery. Stanford made the decision to purchase a bulk DVD reader, and
copied the data onto large Terabyte hard disks, copies of which were
sent back to David. Metadata for these images was stored separately.
David considered the metadata of record to be from the online catalog
hosted by Luna Imaging. This catalog is stored remotely on the Luna
servers. The Luna metadata lacked coordinate information for each image,
which required that we integrated it with the cataloging records for the
same images created in MARC and imported into the Stanford OPAC
(Socrates) from OCLC.

The Stanford NGDA team wrote software to consolidate and reconcile these
two metadata sources into Transfer Manifests, and began ingestion into
the archive. This created a complex set of problems unique to this
collection. For example, not all of the imagery had coordinate data due
to its inherent nature. We had to manually identify classes of imagery
that were "correct" without coordinate data such as title pages, views,
text pages, and ephemera. The Transfer Manifests were coded to look for
a bounding box of 4 coordinates. Some images were cataloged with only
two, which is allowed in AACR2R cataloging for individual points or
towns. The team at Stanford spent a significant amount of time manually
correcting and checking the metadata before ingesting it into the
archive. The vast majority of the Rumsey collection has been reconciled
and ingested into the repository.

The remaining metadata records are significantly more complex due to the
cataloging conventions chosen by the Rumsey team. For example, one OCLC
record is created for an atlas while in Luna each atlas page has its own
record. In Luna, individual atlas pages are occasionally shown with
their covers, while in OCLC there are no records created for the covers,
leaving imagery with no secondary metadata source. (While the cover
should not have any coordinates associated with it, the programming
notes this anomaly as an error, which then must be checked manually.)
Some images include more than one map (2 or more images on the same
page) which leads to an OCLC record with an equal number of coordinates
in repeating mathematical data fields. It is unclear if it is possible
or advisable to try and orient these images correctly in space in the
ADL viewer. Again, decisions must be made for this "class" of materials
by the Stanford team. We were surprised to discover that even with
robust metadata there were still so many types of exceptions that
required human intervention. Before working with the materials we had
been concerned with getting no metadata or poorly designed/completed
metadata. We did not expect to spend so much time reconciling records
and making decisions on a case by case basis for a significant portion
of the 15,000 items in the collection.

Work is well underway on the second large collection to be ingested into
the SDR, the Stanford Geological Survey. This collection includes over
5,000 images both of map and accompanying book related materials. The
Stanford team has been working for many months on cleaning up and
associating the right metadata records with the individual images. We
know in advance that coordinates will be a problem when accessing
through the ADL viewer. Due to budgeting constraints, the Branner
Library catalogers were only able to catalog the sheets at the county
level regardless of the granularity of the mapping. If the map fit
within a county, it was given the coordinates of the region. None of the
maps had coordinate data included (although we clearly knew the region
being mapped), which meant that each one would have had to be researched
against topographic maps of the same region.

Maps from each field season where gathered into one MARC record with the
primary author status being given to the first author listed on the
first map. The rest of the maps and their accompanying authors were
listed in a series of Contents Notes (repeating 505 fields).
Spreadsheets have been created linking each image to its correct
authors. This gives one the ability to pull up the correct map by author
name and lessens the chance for false hits when searching. The primary
display mechanism for this collection will be Stanford's Luna Insight
browser with the archival imagery being stored in the SDR for retrieval
when necessary. This collection will be ingested in the winter of
2007/8.

The SDR consists of four main pieces, only three of which are
architectural. It includes the Transfer Manifest to store metadata in a
standards-based, universally searchable and retrievable format
(mentioned elsewhere in the report), and three architectural components;
Ingest, Storage, and Access.

SDR Ingest is design to work with the Transfer Manifest to scan incoming
data of any sort for viruses, validate its metadata and file structure,
evaluate checksums to guarantee integrity, and then repackage digital
objects for archiving. It consists of a single server running the ingest
code, with a module designed to pull data from the non-secure staging
area into the secure ingest area after validation.

The Storage layer is a complex system run by a storage manager, which
stores all metadata in an online cache, and then writes the archival
packages (with metadata included) to tape and distributes them to
geographically remote tape storage locations. The storage layer also
includes a small server to accept outside requests from the access
layer, and to broadcast certain actions and changes. Some data
(including a full copy of all metadata) is stored in an online,
self-healing, replicated disk system. Archivas was originally chosen for
this, but due to support issues and some additional research, Stanford
has switched to Sun's Honeycomb system.

SDR Access is a service built onto the SDR, and is still under
construction. It pulls all the metadata into a third metadata cache
outside the repository, and uses Lucene (the XTF version) to make all
the metadata available through constrained or full-text search to a user
interface that allows users to search or browse the database, based on
their permission levels (a per-collection setting). Users can set
bookmarks, create lists, and search for individual objects within the
repository. This layer communicates directly with the access layer to
receive notifications of new digital objects.

This robust, redundant design ensures that data deposited into the SDR
will be preserved, accessible, and actively managed indefinitely. This
is a critical component of the NGDA project; not only can Stanford
download large quantities of data and hand them over to the Library of
Congress, but if allowed, SDR can store that data securely and
availably, making it accessible by whatever federation mechanism are
deemed appropriate. The ADL federation work is one example of the ease
of access to NGDA data in SDR -- Stanford believes that a solid, well
designed, long term repository is a core component of any project based
on data collection and preservation.

**Format Registry**

Technically, geospatial data is more complex than standard digital
formats. This must be accounted for when archiving the data. In order to
preserve a data format, information about that format must be known. The
archive has to have an automated way to understand the file it has
received and to verify that it is what it purports to be. This format
information is typically stored in a registry, which records detailed
metadata about the types of ﬁ les. For example, format information for a
GeoTIFF would include specifications for the correct TIFF standard and
explanations of any accompanying files, such as those containing
projection information. The format registry can be as complex as a
custom-made database or as simple as a Web page or text document.

The Library of Congress, along with many other organizations, has spent
a great deal of time describing formats and housing that information in
format registries. Caroline Arms and Carl Fleischhauer compiled the
format registry for the Library of Congress in order to help determine
the sustainability of any given format throughout its content life
cycle, to gain an understanding about which formats are more sustainable
than others, and to develop strategies for sustaining the content they
receive. The content categories studied include still image, sound,
text, and moving image. They did not populate a format registry for
geospatial data formats (Arms & Fleischhauer, 2005).

After searching unsuccessfully for other groups that had created a
format registry for geospatial data, the NGDA team decided to build its
own. Work is ongoing to describe the data elements necessary for
preservation on four formats: digital orthographic quarter quadrangles
(DOQQs), digital raster graphics (DRGs), Environmental Systems Research
Institute (ESRI) "shapefiles," and Landsat imagery. (Other document
formats will be analyzed as content selection for the archives
progresses.) These file formats have differing levels of complexity.

All of them contain multiple files that must travel together in order to
make the format usable now and in the future. For example, ESRI
shapeﬁles are in a proprietary data format and are used in proprietary
software. According to the specifications published by ESRI, only the
.shp, .dbf, and .shx ﬁles are mentioned as part of the shapeﬁle itself.
But, with each shapeﬁle there may also exist numerous other ﬁles, such
as .sbn, .ain., and .prj ﬁles. Public documentation to reference these
files and their role in the playability of the file itself is not
available. Correspondence with ESRI staff was necessary to ascertain
whether or not they considered the last three files necessary to
preserve in the archive along with the published specification files.
Building the format registry is labor intensive as it is necessary to
trace the dependencies of files (a GeoTIFF must also include the correct
TIFF specification, for example), and one must locally collect as much
documentation as possible about each format. However, the set of format
specifications should have to be created only once and then updated as
necessary.

Our task was initially defined as \"Analyze spatial data formats and
develop local format registry with programmatic interface to existing
format registry\". Originally, the first part of this task was headed up
by Meredith Williams of Stanford and the latter part by Catherine Masi
of UCSB. We have since determined that the first part is a research
task, rather than a technical task, and has thus been subsumed under the
Analysis and Research subgroup of the project, headed up by John
Banning. Nancy Hobelheinrich (SU) has completed John's report: [An
Investigation Into Archiving Geospatial Data
Formats](../../../../../../../../N:/reports/NGDA%20Banning.pdf).

We determined through research and in conformance with our technical
architecture that the high level technical requirements of NGDA Format
Registry were that it would:

-   Be independent and self-contained within the archive,

-   Contain sufficient semantic information to programmatically access
    format,

-   Contain definitions which would exist in simple documented file
    format in simple directory structure,

-   Have an access/search mechanism (GUI/database) built above it but
    this would not be necessary for access to basic definition files,

-   Eventually interface with collaborative authoritative format
    registries for updates and contributions.

We studied if the existing format registries contained geospatial
formats and found that they did not. Our task is to add geospatial
formats to an existing registry effort such as the Library of Congress
Digital Formats (LCDF). We then decided to focus preliminarily on the
LCDF since our cooperative agreement is to archive data for the Library
of Congress. We studied if LCDF supported access and contribution
mechanisms, how the formats were stored internally and whether a data
dictionary existed to define the fields. We determined that there were
no access and contribution mechanisms at this time. The formats were
stored in MS Word files and the data dictionary was located at
[http://www.digitalpreservation.gov/formats/fdd/fdd_explanation.shtml![](media/image3.png){width="7.291666666666667e-2in"
height="7.291666666666667e-2in"}](http://www.digitalpreservation.gov/formats/fdd/fdd_explanation.shtml).

Our next step was to coordinate our efforts with the LCDF, Global
Digital Format Registry (GDFR), Ockerbloom\'s Format Registry
Demonstrator (FRED), and TOM. Since Nancy Hoebelheinrich (SU) was
planning to attend the Digital Library Federation (DLF) meeting, she
initiated contact with Stephen Abrams of Harvard (GDFR), John Ockerbloom
of Penn (GDFR, FRED) and Steve Morris of NCSU and met with them at DLF.
We came up with the following questions regarding the technical aspects
of building a local format registry and possible interaction mechanisms
between LC, GDFR and our local format registry:

-   What vetting is in place to establish the authority of entries in
    the LCDF?

-   How are entries maintained? Who is responsible for submitting
    corrections, changes to versions, etc.?

-   What is the relationship of the LCDF to the GDFR and University of
    Pennsylvania efforts (TOM & FRED)?

-   Where and how are the actual format definitions stored? How might an
    interactive (service) be set up that would provide automated
    communication between local repositories and the LCDF?

-   Is there a data dictionary or other documentation for the fields in
    the LCDF?

There were still questions left after this discussion. Catherine Masi
subsequently contacted Stephen Abrams (Harvard - GDFR) and John Mark
Ockerbloom (Penn - FRED), to open up a discussion on the technical
aspects of developing a geospatial format registry. Abrams responded
that the GDFR is still only an idea rather than a reality and that a
technical discussion of how our GIS formats should be managed in a
GDFR-conformant way was a bit premature. Thus the task became to
\"Develop a local format registry focusing on technical specifications
and validation information with a view toward building a community of
experts to contribute formats and definitions in the future and with a
view toward interfacing with a collaborative format registry such as
GDFR when it is implemented.\"

The UCSB Technical Architecture team met continuously and discussed the
physical structure and content of the format registry. Catherine began
prototyping the physical structure using the California Spatial
Information Library (CASIL)
([http://gis.ca.gov/data.epl![](media/image3.png){width="7.291666666666667e-2in"
height="7.291666666666667e-2in"}](http://gis.ca.gov/data.epl)) formats.
During the course of building the registry we discussed, analyzed and
completed work on the following issues:

-   Physical structure of registry: Catherine created a hierarchical
    directory based registry and included the entire format
    specification locally, (e.g. as a website in the case of geotiff, as
    local pdf file in the case of shapefile). We subsequently decided to
    flatten the hierarchy of the directory structure because tfw, for
    example, is not a subtype of geotiff but can be attached to a tiff
    or another format.

-   Format record layout: We decided that the most significant
    information to be included in the registry is the format
    specification but that validation information could also be useful.
    We created the record layout with the following fields for our
    prototype with the idea of adding fields as necessary: format_name,
    description, related_formats, specification,
    supporting_documentation, validation_code, validation_documentation,
    file_extentions.

-   Completeness/independence: All links on the format record refer to
    local copies of format information. All documentation about the
    format is located locally in that format\'s directory. Catherine
    continues to work on making sure that the format specifications are
    complete and all information is located locally where possible.
    Several specifications have several layers of documentation making
    this task non trivial.

-   Format of format record: We discussed the format of the basic record
    and decided that xml with a simple XSL stylesheet would be most
    appropriate for sustainability.

-   Supporting formats: We decided to include supporting formats (such
    as html, gif, gzip, css) as well as geospatial formats in our format
    registry for sake of completeness.

-   Format of specification itself: We decided that we must make every
    effort to make the format spec itself readable in 100 years and that
    the best way to do that is to \"desiccate\" any complex format down
    to either plain text or a very simple raster image (e.g. gif). We
    decided that a format spec in a more complex format such as html or
    PDF was acceptable as long as that format was then defined in either
    text or gif format.

-   Validation: We are using JHOVE to validate formats where possible.
    We have implemented JHOVE locally and included the JHOVE validation
    code in the format registry as well as the validation module
    documentation. We discovered that JHOVE\'s audit tool may be useful
    for initial identification of format as well so we may include it as
    part of the ingest process at a later date.

Format records are currently available for bmp, css, geotiff, gif, gzip,
html, jpg, mrsid, pdf, png, shapefile, svg, tfw, tiff, xhtml, xml and
zip formats. Validation is also available for geotiff, jpg, pdf, tiff,
xml. We are concentrating on technical specs and validation only; we
will add further description and use notes later.

The format registry teams at UCSB and SU started working jointly in the
spring of 2007 on the format registry wiki developed at Santa Barbara.
The SU members provided feedback to UCSB on the structure of the wiki.
Mindy Syfert, Greg Janee and Julie Sweetkind-Singer met with the
relevant people at Safe Software, GIS format transformation firm, to
discuss possible collaboration. Mindy has been studying their FME format
and a report will soon be forthcoming on its usefulness to this project.

Stanford team members Mindy Syfert and the metadata specialist (to be
hired in October/November 2007) will build out the complete
specifications for the formats above and add new formats at an agreed
upon rate. Stanford team member, Rachel Gollub, will continue to be the
main point of contact with the GDFR as she serves on their board. Julie
Sweetkind-Singer will attend the NARA/GDFR governance meeting at NARA in
November 2007.

The format registry will be reviewed for technical completeness. It will
be made available in ARCHIVAS and via a graphical user interface. UCSB
will work on format validation for those formats not covered by JHOVE.
We will continue to look into better ways to update the format
specifications. Finally, the team will research models for community
building to create a robust Format Registry.

**Metadata**

Based on expectations for geospatial resources to be collected, the
initial steps were outlined to assess whether special metadata
requirements were needed. We set out to determine what metadata is
likely to accompany geospatial resources, if any, and what metadata
schemes were most prevalent. The next step was to assess the feasibility
of creating and / or adapting metadata to meet anticipated metadata
requirements for geospatial resources. Then we would establish
recommendations for descriptive, administrative, structural and
packaging metadata for geospatial resources destined for the NGDA. The
final result of this work, as we were anticipating, was a specification
of transfer manifest for geospatial resources to be distributed to data
providers interested in depositing such resources into the NGDA.

The initial steps we engaged in were first to identify four different,
classic geospatial formats to analyze: DOQQs, shapefiles, still images
in Tiff / GeoTiff format, and DRG's. Secondly we analyzed preservation
needs for these four formats in the areas of description,
administrative, structural, and packaging metadata. Then created draft,
more or less complete metadata records for sample resources of each
format type. Finally in the first phase we identified the utility, if
not necessity of a format registry.

Through these procedures we found resources to test of each format from
collections that were likely to be ingested into the NGDA. Based on the
existing metadata associated with the test resources, identified and
analyzed likely metadata schemes in the various metadata categories
including MARC21, MODS, FGDC, Z39.87, GER from CIESIN, PREMIS, and METS.
NGDA then developed preliminary specification for desired metadata
record for preservation of geospatial resources. These specifications
were tested by ingesting a collection of still image (maps).
Investigated and identified community-wide efforts to build digital
resource format registries, and began collaboration with format registry
developers to work on geospatial resources.

A few situations occurred that were not planned for ahead of time:
Changed metadata requirements for FGDC metadata when not already
present, and for coordinate and scale metadata for all geospatial
resources. Potential difficulties in scale of embedding full
descriptions of structures, and complete technical metadata for some
geospatial formats. To adjust to these experiences we adapted allowances
for more flexibility in the requirements for FGDC metadata, and for
coordinate and scale metadata. Additionally we identified other
mechanisms for describing structure of geospatial resource should the
current scheme prove impracticable. The primary outcome of this change
was that more resources were ingested into the SDR than would have been
possible with the original metadata requirements.

The next actions in the area of Metadata are as more collections of
different geospatial formats are ingested into the NGDA, we will
continue to test how feasible it is to require full preservation level
metadata, both from the point of view of the data provider, and from the
NGDA staff who will need to prepare the geospatial resources. Efforts
will be made to contribute relevant, accurate and reasonably complete
information about geospatial resource into appropriate, reliable format
registries.

**Collections**

The state of California as represented in the Digital Orthophoto
Quarter-Quadrangles includes approximately 13,200 scenes requiring
roughly 670 gigabytes of storage space. In order to ensure viability of
this dataset into the future, geographic redundancy is necessary in
addition to the information being stored on different types of storage
systems to lessen the chance of loss or corruption of data. This means
the large datasets cannot be stored in a single location, creating the
need for numerous, large, robust preservation environments.

In addition to the volume of data being produced, geospatial data are
often updated and changed, creating the need to save different versions
of the same information. How often the versions are collected will have
to be decided on a case-by-case basis. For example, the National
Elevation Dataset is updated on a bi-monthly basis by the United States
Geological Survey (USGS) as higher-resolution or higher-quality data
become available. Even a single data layer of a city GIS that is used by
many different departments may be updated as often as several times a
day. The different versions may be considered to be at risk because of
the possibility that each iteration may need to be preserved (for
example, for legal reasons, such as to prove when a change in a city's
infrastructure was made). A strong argument can be made that each
version need not be preserved in order to get a valid snapshot of the
data environment.

Government geospatial data may well be considered at risk given the
sensitive nature of some of the information, the decentralization of the
computing environment, the lack of distribution of digital content that
used to come to libraries as part of the Federal Depository Library
Program, and the ease with which content can be removed from a
government Web site. According to OMB Watch, the Bureau of
Transportation Statistics (BTS) removed all GIS data, maps, and
resources from their Web site after September 11, 2001. These data were
later restored after the decision was made that their release did not
pose a threat to national security (OMB Watch, 2005). Pipeline mapping
data was removed from the Department of Transportation's (DOT) Web site
around the same time and has not been released to the public again. The
DOT notes on their Web site that the data is now restricted "to pipeline
operators and Local, State, and Federal government officials ONLY"
(PHMSA, 2005).

Geospatial data is also potentially at risk for long-term preservation
when it is produced by a small group or a single person.

The ease with which content is now created and displayed has caused an
explosion of small producers of high-quality geospatial content. Digital
preservation requires a good deal of planning and expertise. It may also
be prohibitively expensive to undertake. Simply making a backup copy of
these data does not ensure that sufﬁcient metadata has been captured to
understand the environment in which the data was created, guard against
failure of the storage mechanism, allow for geographic distribution, or
solve the problem of ﬁle format migration over time. It is hoped that
through the work done by this group and others the ability for small
groups and individuals to archive safely their data for the long term
will increase.

The scope of NGDA's collecting is solely in the realm of geospatial
digital data. The term, \"digital geospatial data,\" is defined as
digital items, displayed as graphics that are georeferenced or are
geographically identified. These are primarily composed of: digital
maps; remotely sensed images (e.g., aerial photographs; data collected
by satellite sensors); datasets (e.g. shapefiles, layers, geodatabases,
etc.); atlases; globes (celestial and terrestrial); aerial views (e.g.,
panoramas); block diagrams; geologic sections; topographic profiles;
etc.

The purposes of the UCSB and Stanford nodes of the NGDA are to archive
collections of digital geospatial data of the United States of interest
to the primary users of each respective university (students, faculty
and staff), and make the metadata and data (the latter may be served out
by content providers rather than by the individual node) available to
both primary users at each institution and to primary and secondary NGDA
users. The primary users of the NGDA are citizens - present and future -
of the United States of America. The secondary users of NGDA are all
other people who use the Web, of which there are many; the following
brief list is a sampling:

-   From the university/academic world, undergraduate and graduate
    students and faculty, especially in those disciplines which deal
    with geographic areas, e.g., geography, anthropology, history, etc.;

-   From elementary and secondary schools, students and teachers looking
    for information about a specific country or city;

-   From the world of business and commerce, commercial vendors of
    imagery and mapping services; firms that need to know demographic
    information; realtors;

-   Non-profit, non-governmental organizations such as relief
    organizations; economic and social councils, etc.;

-   Persons and firms collecting information about the environment;

UCSB's emphasis is firstly on Santa Barbara County and California
coverages, of any theme, and then on the United States or portions of
the United States as appropriate to UCSB's Map and Imagery Lab's (MIL)
primary user group.

Stanford's Branner Earth Sciences Library and Map Collections (Branner)
focuses on government agencies at all levels and non-profit entities
such as professional organizations or environmentally focused
non-profits, with a secondary focus on commercial firms, and a tertiary
focus on products issued by people.

**\
**

**NGDA Collection Development Policy**

Collection development policies play a critical role in map libraries
and have been important for many decades. The University of
California/Stanford Map Libraries Group (UCSMLG) is still using the
Research Library Group (RLG) conspectus portion for maps and geospatial
data. The cooperative agreement is updated every five years and clearly
spells out policies related to collaborative purchasing, collecting
commitment levels for cartographic types of data and regions, and
interlibrary loan. This agreement and the list of collecting
responsibilities assigned to each university by call number have proved
to be useful to this day (UC/Stanford Map Libraries Group, 2006).

Collection development policies typically do not include directives for
long-term archiving of the collection itself. It has proven useful for
us to review the work being done in the archiving community. While
research libraries do, in general, keep their materials for a long
period of time, they also weed with impunity for reasons of cost, space,
and lack of use. An archive has made a commitment to keeping the
material with the idea of turning it over to another trusted archive
when they cannot or do not want to steward it any longer. In archives
provenance is an integral component of responsible stewardship.
Provenance details who or what group created and/or managed the records
and traces the history of ownership of the records. This is critical
information for a geospatial archive as well, and it must be included in
the metadata.

> A good primer on archival practices is available from the University
> of Albany's M. E. Grenader Department of Special Collections and
> Archives (Parker et al., 2005).

Another area where archival practices inﬂuence long-term preservation is
multiple ﬁle dependencies. Archival practices have codiﬁ ed the process
of accessioning items in a speciﬁc order. This is necessary to preserve
the contents as they were originally received and/or arranged. This is
important for a digital archive as well. Preserving such dependencies
becomes critical when one thinks about long-term preservation of
geospatial data incorporated on a Web site. The Arizona Model, being
developed by the Arizona State Library, Archives, and Public Records, is
using the framework of archival records management for the curation of
collections of Web documents. They note that archiving Web documents by
order translates into the correct management of the directories and
subdirectories, which are called series and subseries in archival
parlance. They argue that only through judicious use of archiving
practices can large amounts of data be captured with a relatively small
amount of human input. The system created must be able to scale and
cannot do so if curators must select items one by one (Pearce-Moses &
Kaczmarek, 2005).

The collection development policy for the geospatial archives being
built by the NGDA will be a hybrid between a library policy and an
archival one. It will include standard sections of a collection
development statement that outline the user community; the geographic
scope; the methods, scales, and frequency by which the materials are
collected and updated; and the types of materials included. In addition,
the policy statement will include descriptions of the type and quality
of metadata that need to be included for ingestion into the
repositories. Widely used ﬁle formats and types will be explained on a
general level with the expectation that these will need to be updated
over time. The Cornell University Geospatial Information Repository
(CUGIR) has posted its collection development policy on its Web site,
and it is a good example of this hybrid format (CUGIR WorkGroup of Mann
Library, 2006).

The NGDA librarians will also produce specific collection development
guidelines for their respective institutions. We hope that over time
there will be many partner repositories in the federated network with
broad collecting responsibilities. The individual nodes will focus on
the needs of their primary audience, revising the policy to reflect
individual institutional priorities. It is expected that areas of
collecting interest will fall roughly along the same lines that were
used when accessioning print materials. For example, the UCSB Map and
Imagery Laboratory has a long history of collecting aerial and satellite
imagery, while the Stanford Map Collections has focused heavily on
geologic mapping and data. It is imperative that multiple collecting
bodies be engaged in the process of selection and retention. There is
just too much geospatial information being produced for a few libraries
or institutions to preserve it all.

In 2004, the collections librarians began to sketch out the requirements
for identifying, selecting, and acquiring data for the National
Geospatial Digital Archive. Map librarians typically acquire content
based upon a collection development policy (CDP) built around the needs
of their primary patrons. Our plan was to create a general policy
specifically for geospatial materials that would be broad enough to
cover most collections. We would then create separate policies for UCSB
([http://www.ngda.org/research/Collections/UCSB_CDP_05_07.pdf](http://www.ngda.org/research/Collections/UCSB_CDP_05_07.pdf ))
and Stanford
([http://www.ngda.org/research/Collections/SU_CDP_5_07.pdf](http://www.ngda.org/research/Collections/SU_CDP_5_07.pdf ))
that would be specific to their patrons and collections.

We began by researching other geospatial collection policies that had
been written specifically for digital media. We received information
from a few other librarians and on geospatial clearinghouse web sites
that gave us useful information. However, we realized we would have to
create our own policy from scratch. We modeled the policy on the
existing map CDP framework which included sections for: identification,
selection, and acquisition. The selection section included headings for
geographic areas, languages, chronologic limits, dates of issuance,
types of publisher, and type of materials.

As we wrote the policy, we realized that collecting digital media
required that we change the format of the CDP simply due to the nature
of the materials. Collecting materials in this arena requires knowledge
of different formats and (possibly) producers of content. We found it
wise to include much longer sections on the scope of coverage to explain
how one would choose the type and scale of data to be collected.
Suggestions were included about where to go to get the information once
the type of data was chosen. In the date/chronology section, the need
for versioning was discussed as the decisions about when to collect
content have been pushed onto the librarian rather than being made by
publishers of paper products.

A large section was added entitled metadata recommendations. Paper maps
come with their metadata printed on the material itself. In order to
collect and preserve digital files, the need to have good metadata is
paramount and the likelihood of getting it varies by the data provider.

Finally, we added a glossary in the appendix figuring that many
librarians would not be familiar with the terminology included in the
CDP. Also included was an appendix providing further information about
collection policies and relevant agencies collecting digital geospatial
information.

When the initial policy was complete, Stanford and UCSB collections
librarians created individual policies reflecting the needs of their
individual institutions. The individual policies do not include the
glossary or the full explanations about data providers and potential
sources of content.

Stanford's primary users are the students, faculty and staff, of
Stanford University. The secondary users of the content are the citizens
of the United States. The primary collecting emphasis is on the
geographic area firstly of Santa Clara and San Mateo Counties, and of
California as a whole, and secondarily of the United States. The main
areas of collecting focus are topography, geology, and environmental
aspects. All formats of geospatial materials will be collected. As with
the paper-based collections, Stanford will actively collaborate with
other libraries to collect data as widely and deeply as possible.

UCSB's primary users are the students, faculty and staff of the
University of California, Santa Barbara. The secondary users of the
content are the citizens of the United States. The primary collecting
emphasis is on the geographic area firstly of Santa Barbara County, and
of California as a whole, and secondarily of the United States. The main
areas of collecting focus are topography, biogeography, and historical
geography. All formats of geospatial materials will be collected. As
with the paper-based collections, Stanford will actively collaborate
with other libraries to collect data as widely and deeply as possible.

All three policies have been posted on the NGDA Web site at:
<http://www.ngda.org/research/Collections/CDP_Nov_06.pdf>. The Policies
were presented at the ESRI Education User Conference by Tracey Erwin in
June 2007. The collections librarians (Mary Larsgaard, Julie
Sweetkind-Singer, and Tracey Erwin) will write an article for a relevant
journal on the policies, encouraging other geospatial librarians to do
the same. Finally, the librarians will gather content for the NGDA in
line with the policies set forth in the collections statements.

**Rights Management and Contracts**

Information regarding the rights governing the ownership, use, and
copyright status of the data is associated with each ﬁle included in the
repositories. A great deal of domestic geospatial data is produced by
the U.S. government, which allows for wide use of its output due to the
fact that most of it is in the public domain. But even government data
may have copyright stipulations attached to it if it has been
distributed through a third-party vendor contracted through a
Cooperative Research and Development Agreement (CRADA) or has had value
added to it by a commercial ﬁrm.

Important datasets, such as California's SPOT Image coverage and the
base data on ESRI's Data & Maps CD-ROMS, are governed by strict
licensing and use agreements. These datasets provide high-quality base
map layers for GIS work and, especially with the yearly release of ESRI
Data & Maps, provide longitudinal data that allow for the study of
change over time. As the NGDA project moves forward, these agreements
may make it impossible to federate data if the other potential
repositories in the NGDA federation do not also have the legal right to
hold the data. During the second year of the grant, the NGDA staff will
begin a dialog with commercial data and imagery producers to assess
their preservation strategies, awareness, and willingness to work with
preservation archives.

In order to codify the rights and responsibilities of the repositories,
each depositor will sign a contract licensing their content for
preservation in the NGDA repositories. The goal is to create a single
contract that can be used, and modiﬁed if necessary, by both Stanford
and UCSB. The contract (in draft form as of this writing) governs the
use, display, delivery, and preservation of materials in the NGDA. It
clearly states who owns the copyright to the materials and ensures that
those depositing materials in the archive have a right to do so.

It further clariﬁes that the copyright stays with the original depositor
and that the archives are not responsible in cases of copyright
infringement. It explains what may be distributed from the
archives---the metadata, the data, or both---and to whom. It details how
the repositories' rights and responsibilities will be carried out,
including the need to use best practices and standards for preservation.
The archives agree to take measures to prevent unauthorized access to
the data, to permit only authorized users to access the content, to
credit the copyright holders, and to use the utmost care in the
preservation of the content. The contract explicitly allows the archives
to manage the data to maximize its chances of survival over the long
term.

In addition to legal protection for both parties afforded by the
contract, a well-thought-out contract explicating the roles of each
party builds an important element of trust that will encourage content
creators to deposit their content in our repositories. The contract
embodies one of the aspects of the trust-building activities recommended
in the Research Libraries Group/Online Computer Library Center report,
"Trusted Digital Repositories: Attributes and Responsibilities"
(Research Libraries Group, 2002).

In order to further investigate how copyright law affects archiving of
digital data, the Library of Congress has convened the Section 108 Study
Group. Section 108 of the Copyright Act, created in 1976 and amended in
1998, governs the use of copyrighted materials held in libraries and
archives. It is believed that even with the 108 revisions, the law is
designed to meet the needs of the analog world, not the complex issues
and needs of the digital one. This group has been charged with reviewing
existing copyright laws as they pertain to libraries and archives, and
speciﬁcally as they apply to digital media. The group will advise the
Librarian of Congress in May 2006 on their ﬁndings and make
recommendations based upon the needs of the content producers as well as
those wishing to archive and access their output (Library of Congress,
n.d.).

**Content Provider Agreement**

One of the achievements of the first three years of NGDA was our
development of a Content Provider Agreement, also known as a depositor
agreement. The purpose of the agreement is to enable Stanford University
and the University of California, Santa Barbara, as custodians of the
NGDA, to accept content from organizations and individuals seeking to
archive their geospatial data collections.

Original plan for research in the area, with any anticipated outcomes
included:

-   In order to legally acquire content for the NGDA we knew that
    certain collections, especially those not in the public domain,
    would require a legal agreement between the depositor and the
    archive.

-   Our original plan was to create a draft agreement and get feedback
    from a depositor of data who had pledged to make his collection
    available to us. Using that feedback, our goal was to create a
    depositor agreement that would cover all types of geospatial data
    that we could foresee collecting.

The collections librarians at SU and UCSB formed a task group that
included several administrators with experience creating and negotiating
contracts for electronic data. This group met regularly to explore types
of agreements and issues that need to be covered by such a document.

We reached a point where we realized that we each needed to seek legal
counsel to address questions that were beyond our expertise and to
ensure that our agreement followed current legal practice at our
respective universities.

Some of the legal considerations brought up by both institutions
stimulated further discussion with our colleagues. Eventually we agreed
upon a three part structure for the agreement. Each institution would
write its own "front section" to the agreement. This front section would
follow the legal and administrative requirements unique to each
institution. We agreed that the material in the "front section" would
match as closely as possible across the institutions striving for
standardization of the agreements to the greatest degree possible.
Exhibit A in the contract includes information about the specifics of
each collection as it is brought into the archives. Exhibit B in the
contract governs the authorized uses of licensed materials as well as
the management of licensed material by custodians. Exhibit B, then, is
the same for each custodial node of the NGDA. The front section is also
substantially the same for each node. This uniformity ensures that the
same standards of digital preservation will be followed by current and
future nodes of the NGDA.

The agreement
([http://www.ngda.org/research/Rights/SUL_NGDA_3.0COPY.pdf](http://www.ngda.org/research/Rights/SUL_NGDA_3.0COPY.pdf )
) was completed by the task group in August 2007. It was sent to
Stanford's University Librarian and received final approved. The UCSB
version, tailored to their needs, is pending final approval by the UCSB
lawyers with acceptance expected shortly. The process took longer than
we had anticipated due to the complexity of the agreement, our
inexperience drafting contracts, and our need to seek legal counsel
repeatedly along the way. We learned the value of having in-house
counsel help us understand the complexities inherent and the subtleties
needed in drafting an agreement.

The Content Provider Agreements will now be used when accepting content
from those individuals and organizations depositing content with the
NGDA. Although changes are not anticipated, we do expect to receive
comments over time from those signing it and will consider them in
future iterations of the agreement.

Our present goal is to complete an agreement between the nodes. The
agreement will address the rights and responsibilities of custodial
nodes to one another. As future nodes join the NDGA this operational
agreement (and an accompanying procedure manual) will maintain
professional standards and provide a framework for efficient operations
and clear communication.

**Node-to-Node and Procedural Documents**

The basis for a "Node-to-Node Agreement" is to obtain acknowledgement
of, and commitment to, the legal and financial indemnification of each
node against the actions of any other node(s). It is designed as an
inflexible contract requiring the nodes to agree to the basic tenets of
archiving and distribution of geospatial materials as put forth by the
NGDA. It is a binding contract signed by an authorized party with the
expectation that, shy of significant legal or practical academic
changes, it is a static document.

In contrast, the "Procedural" document is an operational guide that
provides the rules of the NGDA and practical expectations of
participating nodes. This is highly malleable reference source that may
change as additional nodes join the archive or as general changes in
technology occur.

The development of these two documents grew from the discovery that the
"Content Provider Agreement" would be too cumbersome if it were overly
inclusive. It would restrict nodes to developing a universal, cumbersome
Content Provider Agreement, in a format that may be inconsistent with
those previously established at their institutions. Almost immediately
it was determined that this would not be possible as the two partners,
UCSB and Stanford could not reach accord on format or content.

Core elements were determined to be inflexible. The list of these became
"Exhibit B." Pursuant to filtering these concrete, legally binding
elements into "Exhibit B," UCSB and Stanford began development of a
"Node Agreement." The intent was to create a document that, irrespective
of any node's contractual format with their content providers, would
require agreement for participation. The document would address several
matters:

-   Content Acquisition

-   Content Removal Policies

-   Governance

-   Intra-Nodal Communication

-   Nodal Participation

Discussions of subsets within each of these categories led to
realization that the issues themselves are permanent, though the means
of manipulating them would likely change over time. Thus, it was deemed
that practices that might change due to changes in technology,
participants, and library standards should be separated into a third
document describing current procedures for maintaining these policies.
The categories listed above became the "NGDA Node Agreement", the
operational subsets, the "NGDA Procedural Manual".

The final draft of the "Node Agreement" is currently under submission to
the respective university attorneys for final legal authorization. This
approval will, for all intents and purposes, allow for the inception of
a legally binding coalition.

**NGDA Procedural Manual**

The "NGDA Procedural Manual" is in the drafting stages. Completion of
this document will be the final link in the chain of a set of practices
that is expected to allow for fluid building of a federation:

-   Collection Development Policy

-   Content Provider Agreement

-   Node-to-Node Agreement

-   Procedural Manual

It is expected that the "NGDA Procedural Manual" will change markedly in
the first few years of integrating new nodes. As there are currently
only two nodes, partners UCSB and Stanford, the governance structure
will likely be redesigned early on; communications procedures will
expand beyond those currently in place; and the means by which nodes may
participate/resign will change.

These ongoing modifications will be overseen by the NGDA Rights and
Responsibilities group.

**Conclusion**

The National Geospatial Digital Archive team has completed the ﬁrst
three years of their contract with the Library of Congress. Phase two
goals include investigating to what degree, if any, commercial
geospatial data producers are concerned with archiving and whether there
is an interest in partnering with academic institutions; gaining a
better understanding of existing mandates for archiving
government-produced geospatial data; continuing to grapple with complex
legal issues surrounding archiving; and ongoing technical development of
the repositories themselves.

The decision to create a common data record format and translate all
sourced geospatial record descriptors into it is a very sensible
future-proofing concept, and the ingestion of all sourced data into a
single archive is similarly sensible for both future-proofing and
usability. The system design is in itself user-oriented and seeks to be
optimized for access through the interfaces being built.

Having two user interfaces provides alternative access to the Archive
and thus maximizes its use potential. This is particularly true of the
access through ADL. User access to the Archive via ADL adds both the
advantages of using existing data processing and information retrieval
tools, while also extending the user's potential for seamless access to
the entire ADL collection in association with the geospatial data in the
Archive.

Finally, the work produced by the first three years of the NGDA
highlight a methodologically contemporary, elegant and scalable system
design, with a seemingly robust and efficient architecture, together
with a data record format that should provide longevity for the
geospatial items in the repository, which in itself is effectively
administered by the archiving software being utilized.

NGDA plans to move forward in its work emphasizing interoperability in
the next two years of the project. NGDA will include Vanderbilt
University and the University of Tennessee at Knoxville in their work to
continue to federate the archive. In order for different archives to
*federate* their resources and collections, they must interoperate at
some level. Interoperability is also necessary to enable the
multilayered series of transitions, or "handoffs," that long term
archives are bound to experience as, over time, their content is moved
to new storage media, new management systems, new policy regimes, and so
on. In the second phase of the project, NGDA will demonstrate a
***Federated Archive Cyberinfrastructure Testbed*** (***FACIT***) that
builds on a new architecture for digital preservation designed to
significantly enhance the ability of archives to interoperate. Using
FACIT, NGDA expects to be able to show how this architectural approach
can facilitate federated resource sharing, redundancy and improved
access. FACIT will also demonstrate the retention of data semantics (the
descriptive information about the technical characteristics of the data
itself and how it relates to other things), persistent association of
data with those semantics, migration of files to new formats, and
archive-to-archive content migration.

FACIT will comprise a minimum of two, independent archives holding
separate content, and a format registry holding format specifications,
metadata, and other contextual information that supports long-term reuse
and reconstruction of the content in archives A and B.  The format
registry will itself be implemented as an archive, and in particular,
format specifications will themselves be represented and stored as
archival objects.  UCSB will be the primary location for the format
registry will. It will be capable of being replicated at other sites.

Success criteria for this project will include:

-   Setting some performance specs (e.g. sustained MB per second).

-   Setting fault tolerance goals - seamlessly loose and rebuilt sites
    without loss of service.

-   Content migration/replication amounts to be determined based on
    final goals for demonstration.

FACIT will build on logistical networking technology, which uses the
*Internet Backplane Protocol* (*IBP*) to provide a highly generic
interface for managing network storage resources, called *depots*.

We expect that, when fully implemented, FACIT will be able to
demonstrate a number of features we believe to be desirable in
preservation architecture.  At the storage level, it will demonstrate
the use of protocol-based, logistical networking storage as an archive
substrate.  We explicitly note that a third archive in this testbed may
be hosted at the Library of Congress.  LC will have the
ability/opportunity to become a full node.

**References**

Arms, C., & Fleischhauer, C. (2005). Sustainability of digital formats
planning for the Library of Congress collections. Retrieved January 24,
2006, from http://www.digitalpreservation .gov/formats/.

CUGIR WorkGroup of Mann Library. (2006). About CUGIR: CUGIR collection
development policy. Retrieved January 31, 2006, from
http://cugir.mannlib.cornell.edu/about.jsp. Library of Congress.

(n.d.). The Section 108 working group. Retrieved January 24, 2006, from
http://www.loc.gov/section108/.

Library of Congress. (2003). Program announcement to support building a
network of partners: Collaborative collection development. Retrieved
January 24, 2006 from http://www
.digitalpreservation.gov/partners/pa_081203.pdf.

Library of Congress. (2004). Library of Congress announces awards of
\$13.9 million to begin building a network of partners for digital
preservation. Retrieved January 24, 2006, from http://
www.loc.gov/today/pr/2004/04-171.html.

Library of Congress. (2006a). Digital preservation: National Digital
Information Infrastructure and Preservation Program. Retrieved January
31, 2006, from
[http://www.digitalpreservation.gov](http://www.digitalpreservation.gov/).

Library of Congress. (2006b). Funding legislation for NDIIPP. Retrieved
January 24, 2006, from
<http://www.digitalpreservation.gov/about/fund.html>.

OMB Watch. (2005). Access to government information post September 11th.
Retrieved January 31, 2006, from
http://www.ombwatch.org/article/articleview/213/1/DOT.

Parker, J., Williams, G., Keough, B., & Schnidler, A. (2005). M. E.
Grenander Department of Special Collections and Archives: Accessioning
and processing manual. Retrieved January 31, 2006, from
http://library.albany.edu/speccoll/processing.htm.

Pearce-Moses, R., & Kaczmarek, J. (2005). An Arizona Model for
preservation and access of Web documents. Retrieved January 31, 2006,
from http://www.lib.az.us/DigGovt/azmodel /AzModel.pdf.

PHMSA. (2005). National Pipeline Mapping System: Data dissemination.
Retrieved January 31, 2006, from
http://www.npms.rspa.dot.gov/data/data_dissem.htm.

Research Libraries Group, Inc. (2002). Trusted digital repositories:
Attributes and responsibilities, an

RLG-OCLC report. Retrieved July 11, 2006, from http://www.rlg.org/legacy
/longterm/repositories.pdf.

UC/Stanford Map Libraries Group. (2006). Cooperative agreements.
Retrieved January 31, 2006, from
http://library.ucsc.edu/maps/ucsmg/agreemts.html.

**Appendix**

**Technical Architecture**

Additional information about NGDA's technical architecture research can
be found at: <http://www.ngda.org/research.php#TA>

**Format Registry**

The NGDA Format Registry Report can be located at:
<http://www.ngda.org/research.php#FR> Further information about the NGDA
format registry can also be found at:
<http://ngda.library.ucsb.edu/format/index.php/Main_Page>

**Collections**

Collections referenced documents can be found at:
<http://www.ngda.org/research.php>

**Operational Specs**

NGDA operational specs specify the need to accept data in any format and
transform its meta-data into a form that is both consistent with ADL and
the Format Registry Interface currently being developed.

Providing access through the ADL browser immediately provides a user
market that will assist in the promulgation of the NGDA Archive, give
access to a wider collection of materials, and enable/encourage use of
it by a wider population of researchers and other interested parties.
This in itself should assist in the library education goal cited in the
Goals and Objectives for August 31, 2006 paper referred to earlier in
this report.

A decision has been made to store all geospatial data items in the
Archive, rather than merely referring users to other locations.
Apparently this is both to ensure a complete and secure repository of
items and thus achieve the primary focus of the project from the
sponsor's perspective, and to have all items conform to a new single
data record specification format, which ensures collection integrity and
ease of system maintenance into the future. This approach is both
conceptually sensible and strategically robust from a system
architecture perspective.

The usability and usefulness of being able to retrieve any data directly
from the Archive, either through ADL or the Simple Archive Interface
(both depicted in the diagram at Figure 2) cannot be over-stated as a
forward-looking and extremely efficient method for user acquisition of
its content. In addition to the 'one-stopshop' approach, which adds to
retrieval effectiveness, this approach provides a very appealing
potential user interface, cutting out mouse clicks, navigation, format
and style changes, etc. It also ensures an enduring accessibility
because as format and meta-data definitions change over time, the
collection will be neutral.

The overall system architecture is to be commended on its thoughtful
conceptualization. Its implementation appears to be well managed and
on-track in terms of the timeline expectations recorded in the planning
documents. Multiple locations of personnel for collaborative work can
add a level of complexity to project management, document control and
system administration. These elements appear to be well directed and
effective for this project.

![](media/image4.jpeg){width="6.94375in" height="5.331944444444445in"}

*Figure 2. Operational specification for the NGDA project*

**Stanford Digital Repository (SDR) Technical Architecture Model**

<http://library.stanford.edu/depts/dlss/collections/sdr.htm>

![](media/image5.wmf){width="7.001388888888889in" height="4.2in"}

*Figure 3. Technical Architecture in 2006/7*

The high-level architecture in 2006/7 is shown above in Figure 3. A more
detailed Component diagram is available below. The salient points of
this architecture are:

The Ingest and Pre-Ingest (Conversion) process is more structured.
\[Separate machines made the system more fault-tolerant.\] A dedicated
machine is used for conversion from external formats to the SDR Transfer
Manifest. A dedicated machine is used to hold and verify objects in the
Staging area prior to Ingest. A dedicated machine is used to run the
Ingest process.

The Ingest process is more automated. \[Human intervention to invoke
Ingest/Store is minimized.\] A Directory Watcher continually polls the
Staging area, and feeds objects automatically to the Ingest process. SDR
is now in Production with \~ 3 TB of content. SDR is capable of
ingesting \~ 200 GB / day. The Storage system is now Honeycomb. \[The
vendor Archivas' commitment to the project tapered off.\]

Storage pre-processing now decides (based on a policy file) which
objects are stored only on Tape and which are stored online as well.
\[This allows us to use scarce online storage for high priority
objects.\] All metadata is always stored online. Content is also backed
up to offsite storage (Iron Mountain).

All SDR components now log informational and error messages to a logging
subsystem. \[Erstwhile silent failures and errors can be detected,
system problems can be traced to the faulty module. Rudimentary Storage
performance statistics are also stored in the logs.\] Simple versioning
(of digital objects) functionality is introduced.

The earlier Reconstructor component has now evolved into an Access layer
with greater functionality. The first version of the Access layer was
available in mid-2006.

![](media/image6.wmf){width="6.977777777777778in"
height="5.079861111111111in"}

*Figure 4. The components of the SDR. .*

SDR Architectural Enhancements for 2007-2008 include:

-   A Consistency Checker component was introduced in 2007 which
    verifies the integrity of ingested digital objects in the online
    storage system (Honeycomb). Future versions will check across
    multiple storage systems.

-   A Log Analyzer component was added to facilitate Reporting.

-   A new version of the Access layer will be released in mid-2008.

-   Advanced versioning of digital objects (all versions of a digital
    object are accessible) will be introduced by end-2007.