National Archives: Progress and Risks in Implementing its Electronic Records Archive Initiative
Released: 2009-11-05T00:00:00

Letter

Background

The ability to find, organize, use, share, appropriately dispose of, and save records—the essence of records management—is vital for the effective functioning of the federal government. In the wake of the transition from paper-based to electronic processes, records are increasingly electronic, and the volumes of electronic records produced by federal agencies are vast and rapidly growing, providing challenges to NARA as the nation’s recordkeeper and archivist.

Besides sheer volume, other factors contributing to the challenge of electronic records include their complexity and their dependence on software and hardware. Electronic records come in many forms: text documents, e-mails, Web pages, digital images, videotapes, maps, spreadsheets, presentations, audio files, charts, drawings, databases, satellite imagery, geographic information systems, and more. They may be complex digital objects that contain embedded images (still and moving), drawings, sounds, hyperlinks, or spreadsheets with computational formulas. Some portions of electronic records, such as the content of dynamic Web pages, are created on the fly from databases and exist only during the viewing session. Others, such as e-mail, may contain multiple attachments, and they may be threaded (that is, related e-mail messages are linked into send–reply chains).

In addition, the computer operating systems and the hardware and software that are used to create electronic documents can become obsolete. If they do, they may leave behind records that cannot be read without the original hardware and software. Further, the storage media for these records are affected by both obsolescence and decay. Media may be fragile, have limited shelf life, and become obsolete in a few years. For example, few computers today have disk drives that can read information stored on 8- or 5¼-inch diskettes, even if the diskettes themselves remain readable.

Another challenge is the growth in electronic presidential records. The Presidential Records Act gives the Archivist of the United States responsibility for the custody, control, and preservation of presidential records upon the conclusion of a President’s term of office. The act states that the Archivist has an affirmative duty to make such records available to the public as rapidly and completely as possible consistent with the provisions of the act.

In response to these widely recognized challenges, the Archives began a research and development program to develop a modern archive for electronic records. In 2001, NARA hired a contractor to develop policies and plans to guide the overall acquisition of an electronic records system. In December 2003, the agency released a request for proposals for the design of ERA. In August 2004, NARA awarded two firm-fixed-price contracts for the design phase totaling about $20 million—one to Harris Corporation and the other to Lockheed Martin Corporation. On September 8, 2005, NARA announced the selection of Lockheed Martin Corporation to build the ERA system. The contract with Lockheed is a cost-plus-award- fee contract with a total value through 2012 of about $317 million. As of April 2009, the life-cycle cost for ERA through March 2012 was estimated at $551.4 million; the total life-cycle cost includes not only the development contract costs, but also program management, research and development, and program office support, among other things. Through fiscal year 2008, NARA had spent about $237 million on ERA, including about $112 million in payments to Lockheed Martin.

The purpose of ERA is to ensure that the records of the federal government are preserved for as long as needed, independent of the original hardware or software that created them. ERA is to provide the technology to ensure that NARA’s electronic records holdings can be widely accessed with the technology currently in use.

The system is to enable the general public, federal agencies, and NARA staff to search and access information about all types of federal records, whether in NARA custody or not, as well as to search for and access electronic records stored in the system. Using various search engines, the system is to provide the ability to create and execute searches, view search results, and select assets for output or presentation.

NARA currently plans to deliver ERA in five separate increments: ● Increment 1, also known as the ERA base, included functions focused on the transfer of electronic records into the system. ● Increment 2 includes the Executive Office of the President (EOP) system, which was designed to handle electronic records from the White House at the end of the previous administration. The EOP system uses an architecture based on a commercial off-the-shelf product that supplies basic requirements, including rapid ingest of records and immediate and flexible search of content. Increment 2 also includes basic case management for special access requests. ● According to NARA’s 2010 ERA expenditure plan, Increment 3 is to include new Congressional and Public Access systems. It is also to augment the base system with commercial off-the-shelf technology to increase flexibility and scalability. NARA plans to complete this increment by June 2010. ● Increments 4 and 5 are to provide additional ERA functionality, such as backup and restore functions and wider search capabilities, and provide full system functionality by 2012.

NARA Has Completed Two of Five ERA Increments, but Also Experienced Schedule Delays and Cost Overruns While Deferring Functionality

NARA’s progress in developing ERA includes achieving initial operating capability for the first two of its five planned increments. However this progress came after NARA had experienced significant project delays and increased costs. NARA also deferred functions planned for Increment 1 to later increments.

As we reported in 2007, the initial operating capability for Increment 1 was originally scheduled to be achieved by September 2007. However, the project experienced delays due to factors such as low productivity of contractor software programmers, difficulties in securing an acceptable contract to prepare the site that was to house the system, and problems with software integration. These delays put NARA’s initial plan to use ERA to receive the electronic presidential records of the Bush Administration in January 2009 at risk.

In response, NARA and Lockheed Martin agreed to a revised schedule and strategy that called for the concurrent development of two separate systems, which could later be reintegrated into a single system: ● First, they agreed to continue development of the original system but focused the first increment on the transfer of electronic records into the system. Other initially planned capabilities were deferred to later increments, including deleting records from storage, searching item descriptions, and ingesting records redacted outside of the system. NARA now refers to this as the “base” ERA system. Initial operating capability for this increment was delayed to June 2008. ● Second, NARA conducted parallel development of a separate increment dedicated initially to receiving electronic records from the outgoing Bush Administration in January 2009. This system, referred to as the Executive Office of the President (EOP) system, uses a different architecture from that of the ERA base: it was built on a commercial product that was to provide the basic requirements for processing presidential electronic records, such as rapid ingestion of records and the ability to search content. NARA believed that if it could not ingest the Bush records in a way that supported search and retrieval immediately after the transition, it risked not being able to effectively respond to requests from Congress, the new administration, and the courts for these records—a critical agency mission.

As we reported earlier this year, NARA certified that it achieved initial operating capability for Increment 1 in June 2008, following its revised plan. According to NARA’s 2010 expenditure plan, this increment cost $80.45 million to deliver, compared to a planned cost of $60.62 million.

NARA also reported that it completed Increment 2 on time in December 2008 at a cost of $10.4 million (compared to a planned cost of $11.1 million). However, it was not functioning as intended because of delays in ingesting records into the system. Specifically, before the transition, NARA had estimated that the Bush electronic records would be fully ingested into EOP, where they would be available for search and retrieval, by May 2009. However, as of April 27, only 2.3 terabytes of data were fully ingested into the EOP system. This constituted about 3 percent of all Bush Administration unclassified electronic records. NARA later estimated that ingest of all 78.4 terabytes of unclassified records would not be complete until October 2009. In its recently released 2010 expenditure plan, NARA reported that the Bush records were fully ingested into EOP by September 2009.

NARA officials attributed EOP ingest delays, in part, to unexpected difficulties. For example, according to NARA officials, once they started using the EOP system, they discovered that records from certain White House systems were not being extracted in the expected format. As a result, the agency had to develop additional software tools to facilitate the full extraction of data from White House systems prior to ingest into EOP. In addition, in April 2009, NARA discovered that 31 terabytes of priority data that had been partially ingested between December 2008 and January 2009 were neither complete nor accurate because they were taken from an incomplete copy of the source system.

Because the records had not been ingested into the EOP system, NARA had to use other systems to respond to requests for presidential records early in 2009. As of April 24, 2009, NARA had received 43 special access requests for information on the Bush Administration. Only one of these requests used EOP for search, and no responsive records were found. To respond to 24 of these requests, NARA used replicated systems based on the software and related hardware used by the White House for records and image management. NARA’s current expenditure plan reports that after completing ingest of the Bush electronic records in September 2009, it retired the replicated systems.

In fiscal 2010, NARA plans to complete Increment 3 and begin work on Increment 4. According to its 2010 expenditure plan, Increment 3 will cost $42.2 million and be completed in the fourth quarter of fiscal year 2010. It is to provide new systems for congressional records and public access, as well as improvements to the existing base system and the incorporation of several deferred functions, such as the ability to delete records and search and view their descriptions. Fiscal year 2010 work on Increment 4 is to consist primarily of early planning, analysis, and design.

NARA Faces Several Significant Risks to the Successful Completion of ERA

Despite the recent completion of the first two ERA increments, NARA faces several risks that could limit its ability to successfully complete the remaining three increments by 2012. These risks include the lack of specific plans describing the functions to be delivered in future increments, inconsistent application of earned value management (a key management technique), and the lack of a tested contingency plan for the ERA system.

First, NARA’s plans for ERA have lacked sufficient detail. For several years, NARA’s appropriations statute has required it to submit an expenditure plan to congressional appropriations committees before obligating multi-year funds for the ERA program, and to, among other conditions, have the plan reviewed by GAO. These plans are to include a sufficient level and scope of information for Congress to understand what system capabilities and benefits are to be delivered, by when and at what costs, and what progress is being made against the commitments that were made in prior expenditure plans. However, several of our reviews have found that NARA’s plans lacked sufficient detail. Most recently, we reported in July that NARA’s 2009 plan did not clearly show what functions had been delivered to date or what functions were to be included in future increments and at what cost.

For example, the fiscal year 2009 plan did not specifically identify the functions provided in the two completed increments. In addition, while the plan discussed the functions deferred to later increments, it did not specify the cost of adding those functions at a later time. Additionally, NARA’s 2009 plan lacked specifics about the scope of improvements planned for Increment 3. For example, it described one of the improvements as extend storage capacity but did not specify the amount of extended storage to be provided. Also, NARA’s plan did not specify when these functions will be completed or how much they would cost. NARA officials attributed the plan’s lack of specificity to ongoing negotiations with Lockheed Martin.

Another risk is NARA’s inconsistent use of earned value management (EVM). NARA’s 2009 expenditure plan stated that, in managing ERA, the agency used EVM tools and required the same of its contractors. EVM, if implemented appropriately, can provide objective reports of project status, produce early warning signs of impending schedule delays and cost overruns, and provide unbiased estimates of a program’s total costs. We recently published a set of best practices on cost estimation that addresses the use of EVM. Comparing NARA’s EVM data to those practices, we determined that NARA fully addressed only 5 of the 13 practices. For example, we found weaknesses within the EVM performance reports, including contractor reports of funds spent without work scheduled or completed, and work completed and funds spent where no work was planned. In addition, the program had not recently performed an integrated cost-schedule risk analysis. This type of analysis provides an estimate of the how much the program will cost upon completion and can be compared to the estimate derived from EVM data to determine if it is likely to be sound. NARA officials attributed these weaknesses, in part, to documentation that did not accurately reflect the program’s current status.

Another significant risk is the lack of a contingency plan for ERA. Contingency planning is a critical component of information protection. If normal operations are interrupted, network managers must be able to detect, mitigate, and recover from service disruptions while preserving access to vital information. Therefore, a contingency plan details emergency response, backup operations, and disaster recovery for information systems. Federal guidance recommends 10 security control activities related to contingency planning, including developing a formal contingency plan, training employees on their contingency roles and responsibilities, and identifying a geographically separate alternative processing site to support critical business functions in the event of a system failure or disruption.

An internal NARA review found weaknesses in all 10 of the required contingency planning control activities for ERA. As of April 2009, NARA had plans to address each weakness, but had not yet addressed 10 of the 11 weaknesses. In addition, NARA reported that the backup and restore functions for the commercial off-the-shelf archiving product used at the ERA facility in West Virginia tested successfully, but there were concerns about the amount of time required to execute the process. In lab tests, the restore process took about 56 hours for 11 million files. This is significant because, while the backup is being performed, the replication of data must be stopped; otherwise it could bring the system to a halt. Subsequently, NARA officials stated that they have conducted two successful backups, but the restore process had not been fully tested to ensure that the combined backup and restore capability can be successfully implemented.

Implementation of GAO’s Recommendations Could Reduce Risks

To help mitigate the risks facing the ERA program, we previously recommended that NARA, among other things: include more details in future ERA expenditure plans on the functions and costs of completed and planned increments; ● strengthen its earned value management process following best ● develop and implement a system contingency plan for ERA.

In its 2010 expenditure plan, NARA reported that it had taken action to address our recommendations. For example, NARA reported that a test of the ERA contingency plan was completed on August 5, 2009, and the plan itself finalized on September 16, 2009. We have not yet fully reviewed this plan or the results of the reported test. However, if NARA fully implements our recommendations, we believe the risks can be significantly reduced.

In summary, despite earlier delays, NARA has made progress in developing the ERA system, including the transfer of Bush administration electronic records. However, future progress could be at risk without more specific plans describing the functions to be delivered and the cost of developing those functions, which is critical for the effective monitoring of the cost, schedule, and performance of the ERA system. Similarly, inconsistent use of key project management disciplines like earned value management would limit NARA’s ability to effectively manage this project and accurately report on its progress.

Mr. Chairman, this concludes my testimony today. I would be happy to answer any questions you or other members of the subcommittee may have.

Contact and Staff Acknowledgments

If you or your staff have any questions about matters discussed in this testimony, please contact David A. Powner at (202) 512-9286 or pownerd@gao.gov. The other key contributor to this testimony was James R. Sweetman, Jr., Assistant Director.

This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.