Information Management: Challenges in Implementing an Electronic Records Archive
Released: 2008-05-14T00:00:00

Letter

Background

The ability to find, organize, use, share, appropriately dispose of, and save records—the essence of records management—is vital for the effective functioning of the federal government. In the wake of the transition from paper-based to electronic processes, records are increasingly electronic, and the volumes of electronic records produced by federal agencies are vast and rapidly growing, providing challenges to NARA as the nation’s recordkeeper and archivist.

Besides sheer volume, other factors contributing to the challenge of electronic records include their complexity and their dependence on software and hardware. Electronic records come in many forms: text documents, e-mails, Web pages, digital images, videotapes, maps, spreadsheets, presentations, audio files, charts, drawings, databases, satellite imagery, geographic information systems, and more. They may be complex digital objects that contain embedded images (still and moving), drawings, sounds, hyperlinks, or spreadsheets with computational formulas. Some portions of electronic records, such as the content of dynamic Web pages, are created on the fly from databases and exist only during the viewing session. Others, such as e-mail, may contain multiple attachments, and they may be threaded (that is, related e-mail messages are linked into send–reply chains).

In addition, the computer operating systems and the hardware and software that are used to create electronic documents can become obsolete. If they do, they may leave behind records that cannot be read without the original hardware and software. Further, the storage media for these records are affected by both obsolescence and decay. Media may be fragile, have limited shelf life, and become obsolete in a few years. For example, few computers today have disk drives that can read information stored on 8- or 5¼-inch diskettes, even if the diskettes themselves remain readable.

In response to these widely recognized challenges, the Archives began a research and development program to develop a modern archive for electronic records. In 2001, NARA hired a contractor to develop policies and plans to guide the overall acquisition of an electronic records system. In December 2003, the agency released a request for proposals for the design of ERA. In August 2004, NARA awarded two firm fixed-price contracts for the design phase totaling about $20 million—one to Harris Corporation and the other to Lockheed Martin Corporation. On September 8, 2005, NARA announced the selection of Lockheed Martin Corporation to build the ERA system. The contract with Lockheed is a cost plus award fee contract with a total value through 2012 of about $317 million. As of March 2008, the life cycle cost of the system through 2011 was estimated at $453 million; total life cycle cost includes not only the development contract costs, but also program management, research and development, and program office support, among other things.

The ERA system is planned as a comprehensive and systematic means for preserving electronic records, free from dependence on any specific hardware or software. NARA plans for the system to manage the entire lifecycle of electronic records, from their ingestion into the system through preservation and dissemination to customers. According to the Archives, the most fundamental characteristic of ERA is that it must be able to evolve over time to allow new types of electronic records to be brought into the system and preserved.

As planned, the ERA system will consist of six major components, each of which supports a specific business area: “Ingest” enables transfer of electronic records from federal agencies. “Managed Storage” enables stored records to be managed in a way that guarantees their integrity and availability. “Dissemination” enables users to search descriptions and business data about all types of records, and to search the content of electronic records and retrieve them. “Records Management” supports scheduling, appraisal, description, and requests to transfer custody (from agencies to the Archives) of all types of records, as well as ingesting and managing electronic records, including the capture of selected records data (such as origination date, format, and disposition). “Preservation” enables secure and reliable storage of files in formats in which they were received, as well as creating backup copies for offsite storage. “Local Services and Control” regulates how the ERA components communicate with each other, manages internal security, and enables telecommunications and system network management.

Acquiring a major information technology system like ERA is a significant challenge for a relatively small organization such as NARA, which had no previous experience in acquiring major information systems. In 2002 and 2003, we pointed out this and other challenges facing the ERA project: for example, the solution to the electronic records archiving challenge had not been developed, and no electronic archive system existed that was comparable in complexity or scale to NARA’s vision. Further, some key technologies associated with electronic document archiving and storage were not available commercially. As a result, ERA was technically and managerially complex and challenging, requiring the development of an advanced architecture for the conversion and preservation of electronic records.

In view of this complexity, the ERA program planned to use an acquisition and implementation approach in which the system would be deployed in five separate increments. The increments were to provide the following capabilities: Increment 1: provide the hardware, software, and communications needed to deploy the system; capabilities for unclassified and sensitive records (enabling online transfer of electronic records to NARA, annual transfers of records, and secure storage of electronic records in original formats); and support for certain records management functions (such as scheduling and appraisal).

Increment 2: provide additional capabilities, such as content searching, responding to Freedom of Information Act requests, and preservation planning. It is also to handle classified data.

Increment 3: provide public access.

Increments 4 and 5: provide additional preservation capability and storage.

Under the ERA program plan, developed in September 2006, Increment 1 was to take 2 years to develop, and the remaining increments were each to take 1 year.

Among the requirements that ERA is to fulfill are storing and providing access to electronic records from past presidential administrations. The Presidential Records Act gives the Archivist of the United States responsibility for the custody, control, and preservation of presidential records upon the conclusion of a President’s term of office. The act states that the Archivist has an affirmative duty to make such records available to the public as rapidly and completely as possible consistent with the provisions of the act. At the time of the last presidential transition, NARA met these requirements by recreating the Clinton Administration’s computer systems (software and hardware) that originally held the records and developing simple search interfaces so that NARA personnel could search for requested information. NARA reported that it took about 400 days to process the 2 terabytes of data from the Clinton administration. According to NARA, it planned to use ERA to receive the Bush Administration’s presidential records when the Archivist takes custody of them in January 2009. The ERA schedule had planned for the required capabilities (part of Increment 2) to be available in mid-2008.

We have issued several reports on ERA and its development. Most recently, in July 2007, we reported that NARA had taken action to implement our prior recommendations. The agency (1) established a baseline and target architecture, a plan to transition from the baseline to the target, and an architecture review board; (2) revised four of five key acquisition policies and planned to comply with appropriate standards; and (3) developed a fiscal year 2007 expenditure plan that contained an appropriate level and scope of information needed for the Congress to understand its plans and commitments relative to system capabilities, benefits, schedules, and costs.

However, we also reported that the ERA project had experienced schedule delays and cost increases. Factors contributing to the delays included low productivity of contractor software programmers, difficulties in securing an acceptable contract to prepare the site that was to house the system, and problems with software integration. Although the contractor took actions to address the factors causing these delays and overruns, including replacing several programmers with a more experienced software development staff, NARA concluded that the project schedule should be revised so that reduced functionality would be available by the initial operational capability date. Subsequently, the schedule as of July 2007 delayed the date of the initial operational capability (IOC) from September 7, 2007, to March 31, 2008. NARA and the contractor projected that the estimated cost at completion of the IOC would be between $8 million and $12 million over budget. In May 2007, NARA and the contractor were in the process of negotiating additional mitigating actions needed and changes to the project’s cost and schedule.

NARA Is Working to Overcome ERA Schedule Delays through Parallel Development Projects, but Uncertainties Remain

In response to the delays that occurred in 2007 and their effect on the Archives’ readiness to receive the presidential records of the Bush Administration in January 2009, NARA developed a two-pronged strategy for continuing ERA development. First, it has developed plans to achieve IOC by June 2008 with somewhat reduced capabilities from those that had been planned; it refers to the system to be delivered at IOC as the “base” ERA system. Second, it is planning to pursue a parallel development of another part of the ERA system that is to be dedicated initially to presidential records from the Bush Administration; this part of ERA, which is being developed using a different architecture from that of the base system, is referred to as the “EOP (Executive Office of the President) system.” When these developments are complete, NARA plans to merge the two architectures into one ERA system.

This strategy was developed through a series of negotiations that followed the contractor’s acknowledgement that the IOC deadline would not be met. In July 2007, NARA issued a cure notice to the developer, asking for a recovery plan that would include revised delivery dates, complete cost and schedule estimates, and a cost overrun sharing plan. In response to NARA’s cure notice, the contractor provided a plan in August to deliver software in three iterations (or “drops”) that would lead to IOC in May 2008; as part of this plan, the software to be delivered would not include all the capabilities that had been planned: for example, support for certain records management functions was postponed. After review of the contactor’s proposed plan, NARA determined that more time would be needed to complete and test the software and achieve security certification and accreditation of the system; accordingly, it adjusted the schedule to put IOC at the end of June 2008.

In addition, in September 2007, the contractor demonstrated to NARA a prototype for the EOP system that would address basic requirements for processing presidential electronic records, such as rapid ingest of records and ability to search content. To ensure that a system to receive presidential records would be available by the next change of Administration, NARA decided to pursue development of the contractor’s prototype system.

Although NARA’s cure notice asked for a cost overrun sharing plan, the contractor did not offer such a plan. According to NARA, the contractor developed the prototype EOP system at its own expense, in partial mitigation of the cost overrun incurred in 2007. Officials also stated that the contractor was independently funding research on issues related to preservation, whose results would be applied to future work. In addition, the Archives applied the development contractor’s allocated award fee to the cost overrun. According to NARA, it would continue to work with the development contractor to devise ways to mitigate the cost overrun without loss of functionality, but officials acknowledged that under a cost plus contract, achieving such mitigation was uncertain.

In December 2007, NARA estimated a cost overrun for the development of ERA of about $15 million. NARA now estimates this cost overrun at $16.3 million.

ERA Base System Is Generally on Schedule to Achieve IOC, but Testing Delays Are a Risk

Under the most recent plan, the base system is to achieve IOC at the end of June 2008; this is a 9-month delay from the earlier plan (of September 2006) for an IOC in September 2007. In carrying out this plan, the contractor met all deadlines for delivering the software drops, as shown in table 1.

In preparation for achieving IOC, NARA is currently conducting a series of tests aimed at verifying that ERA requirements are met. As part of this process, according to NARA officials, a NARA “Tiger Team” meets weekly to discuss testing and other activities leading to IOC and may adjust the schedule in response to test progress and results. Table 2 shows the status of the testing compared to the current schedule and the high-level schedule provided by the contractor.

As the table shows, the dates for product acceptance testing and operational acceptance testing have been extended, and later test dates have been adjusted in response. According to NARA, it decided to extend the time to ensure adequate time for government testing review of the delivered systems and related documentation. Further, NARA stated that when the Tiger Team analyzed the contractor’s high-level schedule, it determined that NARA’s systems engineering test team required 6 weeks for product acceptance testing, and the end users required only 1 week for operational acceptance testing. Accordingly, product acceptance testing was extended to May 23, and operational acceptance testing was scheduled for the end of that period. NARA officials remain confident that their testing will be completed and necessary fixes made in time for IOC at the end of June.

However, the ERA program faces near-term challenges in achieving IOC of the base system. Unforeseen problems uncovered through testing could lead to further delays. According to NARA, it is mitigating the risk of delays by paying close and continuing attention to the testing process, through such actions as the weekly meetings of NARA’s test team. Delays in achieving IOC could lead to further delays in the development of ERA and the success of the Archives’ long-term plan for electronic records management and preservation.

EOP System Is Being Developed, but Completing the Development in Time for the Presidential Transition Is Uncertain

In September 2007, the contractor demonstrated to NARA a prototype for the EOP system, designed to show the ability to address basic requirements for processing presidential electronic records, such as rapid ingest of records and ability to search content. To ensure that the EOP system would be available by the change of Administration, NARA decided to modify its architectural approach to the EOP system by continuing development of the system along the lines provided by the contractor’s prototype. However, details of the system requirements and capabilities, as well as the specific development plans, are still being defined and final contract terms negotiated while development continues.

Under the new approach, the EOP system would be separately designed, use a dedicated system team (separate from the team working on the ERA base), and be developed concurrently with the work to complete the base system. The EOP system is to use an architecture based on a commercial off-the-shelf (COTS) product that supplies basic EOP requirements, including rapid ingest of records and immediate and flexible search of content. (According to NARA, the EOP system is also to use key elements of the base system infrastructure.) Developing the EOP system in this way decouples it from dependence on the completion of IOC for the base system.

From October 2007 to March 2008, NARA spent $13 million on EOP system planning and development, of which $7 million was used to buy hardware.

At this point, the contractor has delivered three initial software drops for the EOP system; it demonstrated Drop 3 software to NARA on April 28 and 29. (As of March 2008, two additional drops were planned, but the ultimate number of software drops is still being negotiated.)

Table 3 shows the dates that the contractor is to provide the planned drops as well as selected functions associated with each. As the table shows, the first three drops were provided later than originally planned.

Although the development is continuing, challenges remain. In particular, because NARA and the contractor are still negotiating the detailed development plans and specific system requirements, the IOC date for the EOP system and specific details regarding functionality remain uncertain.

The scope of work and requirements for the EOP are not settled. In response to NARA’s request for a proposal describing detailed plans for developing the system, the contractor delivered a proposal in February 2008. After review, NARA asked the contractor to respond to over 200 comments and provide a revised proposal. The revision, delivered April 25, 2008, is currently being reviewed. NARA expects to complete its review on or before May 16, and it expects to reach agreement with its contractor no later than May 30, with a signed contract modification by mid-June.

Currently, NARA and its contractor plan to develop physically separate EOP system segments (or “instances”) for classified and unclassified presidential records, with IOC for the unclassified instance preceding that for the classified instance. NARA is planning for both instances to achieve IOC by the end of 2008, but the dates are not settled.

Among the difficulties of finalizing the negotiations are uncertainties regarding the exact nature of the presidential records that will be transferred to NARA in 2009. Although NARA and the Administration have held meetings on this topic, according to NARA, the Administration has not yet provided NARA with specific information required for it to plan for reliable ingest, indexing, and accessing of the electronic records involved. Both the volume and many of the data types are uncertain: NARA has estimated that the volume of data from the Bush Administration will likely be 100 terabytes—about 50 times greater than that received from the Clinton Administration, and it has based its capacity requirements on that estimate.

The COTS product on which the EOP system is based supports about 370 common data formats, so it can already accommodate some portion of the Bush Administration records. NARA has also obtained limited information regarding the e-mail and image records that will be transferred. However, it has not received details on the data types and formats for about 32 systems for which the Administration has provided brief descriptions, nor has it received sample data on which it could base its planning. According to NARA officials, if further information is received too late to be included in initial system development, they would plan, where possible, to convert the data types received to a generic format that would allow the EOP system to ingest and search the information.

These uncertainties mean that achieving a working EOP system in time for the presidential transition in January 2009 will be a challenging goal. According to NARA, if it cannot ingest the electronic classified and unclassified records from the Bush Administration in a way that supports the search, processing, and retrieval of records immediately after the presidential transition, it will not be able to meet the requirements of the Congress, the former and incumbent Presidents, and the courts for information in these records in a timely fashion.

In summary, NARA has developed a strategy to overcome the earlier ERA schedule delays: it has developed the software and begun testing for the base system, and it has begun development of the EOP system. However, in the near term it faces the challenge of completing the testing of the base system and the larger challenge of defining the requirements and scope of the EOP system and completing its development. In the long term, NARA also plans to merge the two architectures of the ERA base and EOP systems into a coordinated whole. Continuing careful oversight by NARA and the Congress will be important in achieving the ultimate aims for the ERA system: to automate NARA’s records management and archiving life cycle and preserve and provide access to all types and formats of electronic records.

Mr. Chairman, this concludes my testimony today. I would be happy to answer any questions you or other members of the subcommittee may have.

Contacts and Acknowledgements

If you have any questions concerning this testimony, please contact Linda Koontz, Director, Information Management Issues, at (202) 512-6240, or koontzl@gao.gov. Other individuals who made key contributions to this testimony were Barbara Collier, Pamlutricia Greenleaf, Charles C. Hinnant, Tarunkant N. Mithani, and James R. Sweetman, Jr.

This is a work of the U.S. government and is not subject to copyright protection in the United States. This published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.