The Evolving Process to Add Preservation Support for New Formats at Harvard Library

In 2000 Harvard Library began populating the Digital Repository Service (DRS), its digital preservation repository, with digitized text, images and audio. Over the years the Library continued to add preservation support to the DRS for new formats including born digital websites, PDF documents and email. Library collections however continued to grow and diversify to include a wide range of formats not supported by the DRS, including video and a variety of born-digital formats. In 2013 the Library began to bridge that preservation support gap by refining the process of how new formats are supported in the DRS. This paper describes the new process, which is a more consistent workflow and includes external expertise; as well as analysis tools that could be used by other institutions to broaden the range of digital formats that they are able to preserve. Introduction Over the last decade the collections at Harvard Library have diversified to include content in many different analog and digital formats. While preservation solutions exist for the more “traditional” analog and digital formats at Harvard, much of the media-based collections and born-digital material is at risk. The media collections, on VHS, mini-DV, U-matic video tapes, and many other carrier formats are at risk primarily because of media degradation and because much of the playback technology is now obsolete. The born-digital material is at similar risk of permanent loss for many reasons, but primarily because of format obsolescence, lost or corrupt bits, and lack of metadata and documentation needed to interpret the bits. Curators and collection managers at Harvard have requested that many of these media and born-digital formats be supported by the Library’s digital preservation repository, the Digital Repository Service (DRS). Support for new formats has been added to the DRS on an almost continuous basis since it was launched in 2000, but not fast enough to keep up with all of the demands (see Figure 1). To prevent permanent loss, the Library needed to speed up the process for adding support for new formats to the DRS. This problem is not unique to Harvard. This is a reflection of the Information Age we live in where content now routinely comes in many different formats, from a variety of creators. If collecting institutions everywhere do not already have this problem, they will soon need to determine how to go beyond preserving wellunderstood digitized content to include the more complex media and born-digital formats. Figure 1. This graph shows the percentage of requests for DRS support for various formats made by Harvard curators and collection managers since 2004 for which there still was not preservation support in 2014. Almost a quarter of the requests have been for video formats, while there is a long tail of requests to support many other formats. What is a Supported Format in the DRS? All content preserved in the DRS conforms to one of twenty content models, for example Still Image, Audio, Document and Email Message. Each content model prescribes preferred and accepted formats, valid relationships between the object’s files and to other objects, associated metadata schemas, delivery applications and associated preservation plans. Two content models allow files in any format Opaque and Opaque Container. The main difference between the two is that the files of Opaque Container objects are aggregated into compressed zip files while the files of Opaque objects are left as separate files. Although these two content models allow for any file format to be deposited to the DRS, it is not considered that the DRS supports all formats because these files receive only bit-level preservation. Their disposition as opaque is considered to be temporary, until sometime in the future when the formats can be fully supported directly or as a result of a migration. A format is “supported” by the DRS when preservation staff are reasonably confident that material in this format can be used now and that it can be made usable on an ongoing basis through preservation interventions. It requires policy and strategy decisions, the provision of guidelines and often tool development and enhancements. Supported formats can be deposited to the DRS, can be managed within a DRS administrative interface and can be delivered for use by researchers through Library-maintained APIs and delivery tools. 12 © 2015 Society for Imaging Science and Technology Specifically it requires the provision of specifications to collection managers and DRS depositors on the formats that are preferred and acceptable; content creation guidelines, for example, do not use encryption; and instructions on metadata or documentation that should accompany the content. All of this necessarily needs to be backed up by research and analysis. It also requires that the DRS tools accurately understand the new format and support any technical metadata schemas or elements that are adopted for the format. The DRS deposit and management tools must accept and understand the format. Most importantly, discovery and delivery systems must exist for the format; otherwise the content cannot be used by researchers. Before this project, it often took a great deal of time to add new formats to the DRS for several reasons. All of the analysis and development work was done by existing in-house preservation and IT staff who were also working on other projects and ongoing operations concurrently. Because the format support work was intermittent as time allowed, some time was lost to refamiliarization with the work, and the overall completion time was longer than if it had been uninterrupted. If preservation staff were unfamiliar with the formats or associated metadata, time would need to be spent on research as a first step before any other progress could be made. In addition, the overall process for adding new formats was somewhat ad-hoc and largely designed from scratch each time new formats were added. New Approach To make significant headway on adding preservation support for the most requested formats, a three-year project to fast-track support was launched with the support of the Arcadia Foundation. The formats were scoped to include video, office formats (word processing, spreadsheets and presentations), 2D vector formats including CAD, Adobe DNG raw camera images and 3D formats. A few formats that had not been formally requested were included for strategic reasons: disk images and image stacks. To support all these formats within three years the Library’s format support process needed to be transformed into a routine that was much faster and could easily be repeated. At a high-level the approach was to separate the preliminary analysis tasks from the software development work into two different sub-projects because they required different knowledge and skills. For any format, the analysis could be done as soon as expertise was available so that the specifications were already waiting when developer resources became available. In this way the analysis and implementation could be staggered in a way that was more efficient than trying to line up the staff resources needed for all of the analysis and development at one time. Another significant change was that external consultants with specialized format expertise were sought to help preservation staff with the analysis. Besides temporarily increasing the pool of preservation expertise, it had several other benefits. If a consultant could be identified who already had the right format expertise, there was no lag time needed for the up-front format research. Also, having to specify to a consultant the analysis tasks and desired deliverables required preservation staff to clearly define the overall analysis framework in a way that could be repeated for other formats. And most importantly the analysis work for different formats could be done in parallel (provided that multiple different consultants were used), so that the elapsed time was shorter. As a result of these changes the overall methodology for adding support for new formats was made more consistent and efficient. Results An immediate result of this project is that the Library is able to make substantial progress in addressing the backlog of format support requests. By adding support for the video and born-digital formats described earlier, sixty-four percent of the preservation support requests will be addressed by the end of this three-year project. Many of these formats, such as CAD and 3D object models, are very complex and few if any institutions are confident preserving content in these formats [1]. The Library will share its format, metadata and tool decisions on its public website so that this information will be available to other institutions. In addition, because the workflow and analysis tools that were developed might also be of interest to other institutions, those are described here. Workflow At an operational level the Library now has a clear and repeatable workflow for adding support for new formats to the DRS. Table 1 lists the workflow’s analysis and implementation tasks. In general the tasks are sequential but they are discrete enough that they can be distributed among internal staff and external consultants. The first task is to decide which tasks will be done internally or by consultants or by a combination, depending on in-house expertise and the ability to identify external experts. Table 2 shows how the analysis tasks were distributed among internal staff and consultants for each format group. Table 1: Sequential steps of the new process to add support for formats to the DRS Analysis (by preservation, format & metadata experts) 1. Divide up analysis responsibilities 2. Format criteria (Determine key criteria to compare formats) 3. Format analysis (Compare formats using criteria, identifying key pros and cons, recommend which to prefer & accept) 4. Format profiles (Describe the subset that wi