Electronic Records Archive: The National Archives and Records Administration's Fiscal Year 2009 Expenditure Plan
Released: 2009-07-24T00:00:00

Letter

Conclusions

While NARA has continued to make progress on the ERA system, that progress cannot be fully quantified because NARA’s expenditure plan does not clearly identify what functions have already been delivered, how much was spent to provide each function, or how much is required to maintain the delivered increments. NARA’s current plan similarly lacks details on the functions to be provided in future increments and the costs associated with them, including development efforts scheduled to take place in the remainder of this year. In addition, although NARA has been using Earned Value Management to track program cost, schedule, and performance, weaknesses in its EVM data limit NARA’s ability to accurately report on the project’s progress. Without more specific and accurate information on the immediate and long-term goals of the program and the outcomes expected from its resulting efforts, NARA will be hindered in effectively monitoring and reporting on the cost, schedule, and performance of the ERA system, and congressional appropriators will lack information necessary to evaluate the agency’s requests for funds.

Although NARA certified that the EOP portion of the ERA system had achieved initial operating capability as planned in December 2008, the system has been of limited use because of delays in ingesting electronic records into the system. Further, even though it has obligated nearly $40 million on EOP, NARA has instead answered requests for Bush electronic records by using existing systems or replicating White House systems that cost less than $600,000. Under its current schedule, the Bush Administration records will not be fully ingested into the system until October 2009. However, even when the data are fully ingested, the system’s lack of a complete, fully tested contingency plan increases the risk that a system failure or disruption will result in the system being unavailable for several days or more.

Recommendations for Executive Action

We recommend that the Archivist of the United States take the following actions: Report to Congress on the specific outcomes to be achieved by ERA program funding for the remainder of fiscal year 2009.

Provide detailed information in future expenditure plans on what was spent and delivered for deployed increments of the ERA system and cost and functional delivery plans for future increments.

Strengthen the earned value process so that it follows the practices described in GAO’s guide and more reliable cost, schedule, and performance information can be included in future expenditure plans and monthly reports.

Include in NARA’s next expenditure plan an analysis of the costs and benefits of using the EOP system to respond to presidential records requests compared to other existing systems currently being used to respond to such requests.

Develop and implement a system contingency plan for ERA that follows contingency guidance for federal systems.

Agency Comments and Our Evaluation

In written comments on a draft of this report, which are reprinted in appendix II, the Acting Archivist of the United States stated that she appreciated the insight into the expenditure plan observations addressed in the report. She stated that she was pleased to note our recognition that the Fiscal Year 2009 ERA Expenditure Plan met the legislative conditions. In addition, the Acting Archivist summarized actions taken or planned in response to four of our five recommendations. Specifically, she stated that NARA provided a briefing to Congress on the specific outcomes to be achieved by the ERA program on April 27, 2009, and that further detail will be added to the next expenditure plan to address costs and functions delivered to date and what is planned for future increments. She further stated that NARA is in the process of upgrading its EVM system and will strive to comply with all 13 best practices. Finally, the Acting Archivist stated that an ERA Contingency Plan has been developed and is in final review.

The Acting Archivist disagreed with our observation that the EOP system is not currently fulfilling its intended purpose. She stated that this observation failed to differentiate between what the system is capable of doing and the work currently being done on the system. She added that the system does provide required functionality—including searching any records ingested in the system—and that all the remaining data from the Bush Administration will be ingested by the end of the fiscal year. The Acting Archivist also stated that the problem was with the copies of the records that NARA received from the White House, not with the EOP system, and that the data issues have been resolved. We disagree with the Acting Archivist’s statements because the EOP system still lacks the capability to search and retrieve all Bush Administration records in NARA’s possession. Specifically, as we reported in our briefing, NARA used the EOP system to satisfy one request for presidential records during the first 3 months of the transition, but used the less expensive replicated systems to answer another 24 requests. In addition, before the transition, NARA estimated that it would not complete ingesting data until May 2009, limiting the capability of the system in the transition’s early months regardless of the replication errors that ultimately occurred. Finally, the Acting Archivist’s estimate that the Bush data will not be fully ingested until the end of the fiscal year further highlights the fact that NARA still possesses Bush Administration electronic records that cannot be searched using EOP. Until all the Bush Administration records are ingested, the EOP system will not be performing its stated function.

Regarding our recommendation that NARA include in its next expenditure plan an analysis of the costs and benefits of using the EOP system to respond to presidential records requests compared to the systems currently being used, the Acting Archivist stated that it did not seem cost- effective to conduct a retrospective analysis as to whether there might have been technology solutions for systems that have already been retired. However, NARA has not yet provided evidence that the mitigation plan systems have been retired. In addition, we are not recommending a study of past alternatives, but a study of the ongoing costs and benefits of using the EOP system compared with the technology used under NARA’s risk mitigation plan. This would provide the agency with useful information in planning future spending, given that NARA was able to respond to requests for Bush Administration records using systems costing significantly less than what has been spent to date on the EOP system. Such an analysis could also inform the contingency plan the Acting Archivist said is being finalized in response to our recommendation.

We are sending copies of this report to the Archivist of the United States. The report will also be available at no charge on the GAO Web site at http://www.gao.gov.

If you or your staffs have any questions concerning this report, please contact me at (202) 512-9286 or by e-mail at pownerd@gao.gov. Contact points for our Office of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix III.

Appendix I: Briefing to Staff of Congressional Committees on NARA’s Fiscal Year 2009 Expenditure Plan

Review of NARA’s Fiscal Year 2009 Electronic Records Archive Expenditure Plan

Attachments

Since 2001, the National Archives and Records Administration (NARA or the Archives) has been working to develop a modern Electronic Records Archive (ERA). This major information system is estimated to cost more than $550 million and is intended to preserve and provide access to massive volumes of all types and formats of electronic records, including presidential records, independent of their original hardware or software. NARA plans for the system to manage the entire life cycle of electronic records, from their ingestion through preservation and dissemination to customers. It is to consist of infrastructure elements, including hardware and operating systems; business applications that will support the transfer, preservation, dissemination, and management of all types of records; and a means for public access via the Internet.

Because of the system’s complexity, NARA awarded a contract to Lockheed Martin to develop ERA in phases, or increments, the first of which was originally scheduled to achieve initial operating capability (IOC) in September 2007. However, the contractor did not meet the original cost and schedule milestones due, in part, to productivity issues with its initial development team.

In response, NARA and Lockheed Martin agreed to a revised schedule and strategy, consisting of a two-pronged development approach.

First, they agreed to continue development of the original system but delayed capabilities to later increments. According to NARA, IOC for this system, now referred to as the “base” ERA system, was achieved in June 2008 as planned under the revised schedule.

Second, NARA conducted parallel development of a separate system dedicated initially to receiving electronic records from the outgoing Bush Administration in January 2009. This system, referred to as the Executive Office of the President (EOP) system, uses a different architecture from that of the ERA base: it was built on a commercial product that was to provide the basic requirements for processing presidential electronic records, such as rapid ingestion of records and the ability to search content. NARA believed that if it could not ingest the Bush records in a way that supported search and retrieval immediately after the transition, it risked not being able to effectively respond to requests from Congress, the new administration, and the courts for these records—a critical agency mission. NARA certified the EOP system for IOC in December 2008.

As mandated by the Omnibus Appropriations Act, NARA is required to submit an expenditure plan before obligating multiyear funds for the ERA program. As in the previous year, the plan must satisfy the following legislative conditions: meet the capital planning and investment control review requirements established by the Office of Management and Budget (OMB), including Circular A-11; comply with the agency’s enterprise architecture; conform to the agency’s enterprise life-cycle methodology; comply with the acquisition rules, requirements, guidelines, and system acquisition management practices of the federal government; be approved by the agency and OMB; and be reviewed by GAO. tion Act, 2009, Pub. L. No. 111-8, div. D, title V, 123 St. 524, 667 (Mr. 11, 2009).

On March 6, 2009, the agency finalized the 2009 expenditure plan that was submitted to the House and Senate appropriations committees to support its request for $67 million in ERA funding for fiscal year 2009, which is comprised of $45.8 million in multi-year funds and $21.2 million in single-year funds. fil yer 2009 budget authority totled $67.6 million which inclde $0.6 million crried over from mlti-yer fnd pproprited in previous ye.

Objectives, Scope, and Methodology determine whether NARA’s fiscal year 2009 expenditure plan satisfies the provide an update on NARA’s progress in implementing our prior expenditure plan provide any other observations about the expenditure plan and the ERA acquisition.

To assess compliance with the legislative conditions, we reviewed NARA’s fiscal year 2009 exhibit 300 submissionto OMB to determine the extent to which the agency has complied with OMB’s capital planning and investment control requirements; obtained and reviewed data on NARA’s enterprise architecture to determine the status of the agency’s enterprise architecture efforts; reviewed NARA’s enterprise systems development life cycle methodology that includes processes for managing system investments, configuration management, and risks, and reviewed related documentation concerning how these processes were implemented for the ERA project, such as minutes of oversight boards and the risk management plan; reviewed internal assessments of ERA; obtained and reviewed OMB’s approval of the expenditure plan; and reviewed and analyzed the fiscal year 2009 expenditure plan submitted by the agency in March 2009. develop n exhiit 300, o known as the Cpitl Asset Plnd Business CasSummry, to justify ech reqt for jor informtion technology invetment. OMB et forth reqirement for the exhiit 300 in Circr A-11, Prt 7, Plnning, Bdgeting, Acqition, nd Mgement of Cpitl Asset.

Objectives, Scope, and Methodology To determine the status of our two prior recommendations, we obtained and reviewed monthly congressional reports and the EOP mitigation plan. We reviewed these documents to determine whether NARA (1) developed a risk mitigation plan to ensure indexing and searching of records from the Bush Administration in the event that the EOP system was not complete in time for the January 2009 presidential transition and (2) included earned value data in its monthly reports to Congress.

Objectives, Scope, and Methodology To develop observations on the ERA expenditure plan and acquisition, we analyzed the cost and schedule information contained in the expenditure plan, reviewed agency and contractor documents such as EOP test results—including acceptance tests and security and risk assessments—and cost and schedule reports, and performed analysis of Earned Value Management (EVM) data and key processes used in NARA’s EVM system. We also reviewed federal requirements for contingency planning and NARA plans for corrective action on issues identified. In addition, we interviewed NARA officials. We did not evaluate the controls over the procedures used to transfer records from the White House to NARA or controls over processes used to determine which records were presidential records. project mgement tool tht integrte the techniccope of work with chedle nd cot element for invetment plnning nd control. It compre the ve of work ccomplihed in given period with the ve of the work expected in tht period. Difference in expecttion re measured in oth cond chedle vrince. OMB reqire gencie to use EVM in their performnce-based mgement tem for the prt of n invetment in which development effort i reqired or tem improvement re nder wy.

Objectives, Scope, and Methodology To assess the reliability of the cost and schedule information contained in the expenditure plan, we interviewed NARA officials in order to gain an understanding of the data and discuss our use of the data in this briefing. In addition, we compared schedule information in the fiscal year 2009 plan with information in the fiscal year 2008 plan and the ERA integrated schedule. We did not, however, assess the accuracy and reliability of the information reported in these documents.

We conducted this performance audit from March 2009 to May 2009 at NARA’s College Park, Maryland, location in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

NARA’s fiscal year 2009 plan satisfies the six legislative conditions contained in the 2009 Omnibus Appropriations Act.

NARA implemented one of the recommendations we made last year, and partially implemented the other: We recommended that NARA develop a risk mitigation plan to ensure indexing and searching of records from the Bush Administration in the event that the EOP system was not complete in time for the January 2009 presidential transition. NARA finalized its plan in November 2008. It stated that, in the event that records could not be ingested into EOP in a timely manner, NARA intended to acquire hardware and software to replicate the systems containing the data and use them to retrieve requested records.

We recommended that, to improve the utility of information provided to Congress, NARA include summary measures of project performance against ERA cost and schedule estimates in its monthly reports. In July 2008, NARA began including an Earned Value Summary as an appendix to its monthly reports to Congress. However, after reviewing NARA’s earned value data, we found methodological weaknesses that could limit NARA’s ability to accurately report on program cost, schedule, and performance (these weaknesses are discussed further below).

We have four observations related to the ERA program and fiscal year 2009 expenditure plan: Cost, schedule, and performance data in the expenditure plan do not provide a clear picture of ERA system progress. The plan does not specifically identify whether completed increments included all previously planned functionality or what functionality will be provided in future increments. For example, the plan does not specify what outcomes NARA expects to achieve with the remainder of its fiscal year 2009 funding. NARA officials attributed the plan’s lack of specificity to ongoing negotiations with its contractor. Until NARA fully describes the outcomes expected for the remainder of the year, congressional appropriators will lack information important for evaluating the agency’s request for ERA funds.

NARA’s expenditure plan states that it relies on Earned Value Management (EVM), an important tool for project management and control that is intended to provide, among other things, objective reports of program status. However, NARA is fully addressing only 5 of the 13 practices required to effectively implement EVM, which limits the reliability of its progress reports. NARA officials attributed these weaknesses, in part, to documentation that did not accurately reflect the program’s current status. Without consistently following these best practices, NARA will be hindered in accurately monitoring and reporting on the cost, schedule, and performance of the ERA system.

Although NARA certified initial operating capability for the EOP system in December 2008, the system is not currently fulfilling its intended purpose. Less than 3 percent of the Bush Administration electronic records NARA received have been successfully ingested into the system. NARA officials estimate that the records will not be fully ingested until October 2009. Agency officials attributed delays in part to unexpected difficulties, such as data not being extracted in expected formats and incomplete replication of one type of data. In the interim, NARA is primarily supporting search, processing, and retrieval of presidential records using the replicated systems described in the risk mitigation plan we recommended the agency develop last year, which cost less than $600,000 to put in place, compared to nearly $40 million it has obligated for EOP. Until NARA completely and accurately ingests the Bush Administration presidential records into EOP, it will be unable to use the system for its intended purpose and will incur additional costs maintaining the systems it is now using to support requests for these records.

NARA lacks a contingency plan for the ERA system in the event of a system failure or disruption. While NARA identified 11 security weaknesses related to contingency planning during acceptance testing and listed actions planned to address them, NARA has completed only one of the 11 planned actions. Further, NARA does not have a fully functional backup and restore process for the ERA system, a key component of planning for system availability in the event of a failure or disruption. According to NARA officials, a full system contingency plan is under development. However, until such a plan is tested and implemented, NARA risks prolonged unavailability of the ERA system in the event of a failure or disruption.

We are recommending that the Archivist of the United States take the following actions: Report to Congress on the specific outcomes to be achieved by ERA program funding for the remainder of fiscal year 2009.

Provide detailed information in future expenditure plans on what was spent and delivered for deployed increments of the ERA system and cost and functional delivery plans for future increments.

Strengthen the earned value process so that it follows the practices described in GAO’s guide and more reliable cost, schedule, and performance information can be included in future expenditure plans and monthly reports.

Include in NARA’s next expenditure plan an analysis of the costs and benefits of using the EOP system to respond to presidential records requests compared to other existing systems currently being used to respond to such requests.

Develop and implement a system contingency plan for ERA that follows contingency guidance for federal systems.

In written comments on a draft of this briefing, the Acting Archivist of the United States agreed with 4 of our 5 recommendations. Regarding our recommendation that NARA report on the costs and benefits of using the EOP system to answer requests for presidential records compared to other systems, the Acting archivist stated that such a comparison would not be valid because of the differing capabilities of the systems and that the EOP system has accomplished enormous amounts of work. We disagree that a comparison of the costs and benefits of EOP and other systems would be invalid, because a valid analysis should account for the differences in capabilities. In addition, we noted that the replicated systems currently used to answer requests include nearly half of the records transferred to NARA and have been used to answer most of the requests received to date.

The ability to find, organize, use, share, appropriately dispose of, and save records—the essence of records management—is vital for the effective functioning of the federal government. In the wake of the transition from paper-based to electronic processes, records are increasingly electronic, and the volumes of electronic records produced by federal agencies are vast and rapidly growing, providing challenges to NARA as the nation’s record keeper and archivist.

Besides sheer volume, other factors contributing to the challenge of electronic records include their complexity and their dependence on software and hardware. Specifically, the computer operating systems and the hardware and software that are used to create electronic documents can become obsolete. If they do, they may leave behind records that cannot be read without the original hardware and software. Further, the storage media for these records are affected by both obsolescence and decay. Media may be fragile, have limited shelf life, and become obsolete in a few years. For example, few computers today have disk drives that can read information stored on 8- or 5¼-inch diskettes, even if the diskettes themselves remain readable.

Another challenge is the growth in electronic presidential records. The Presidential Records Act gives the Archivist of the United States responsibility for the custody, control, and preservation of presidential records upon the conclusion of a President’s term of office. The act states that the Archivist has an affirmative duty to make such records available to the public as rapidly and completely as possible consistent with the provisions of the act.

In response to widely recognized challenges, the Archives began a research and development program to develop a modern archive for electronic records. In 2001, NARA hired a contractor to develop policies and plans to guide the overall acquisition of an electronic records system. In December 2003, the agency released a request for proposals for the design of ERA. In August 2004, NARA awarded two firm-fixed-price contracts for the design phase, totaling about $20 million—one to Harris Corporation and the other to Lockheed Martin Corporation. On September 8, 2005, NARA announced the selection of Lockheed Martin Corporation to build the ERA system. l Acqition Regtion, firm-fixed-price contrct provide for price tht i not subject to ny djustment on the bas of the contrctor’ cot experience in performing the contrct.

The total value of the cost plus award fee contract with Lockheed through 2012 is about $317 million. As of fiscal year 2008, NARA has paid Lockheed $111.9 million for system development. t plus rd fee contrct i cot reimbuement contrct tht provide for fee conting of basmont fixed t inception of the contrct plus rd mont tht me given based pon dgmentl evuation y the government of contrct performnce.

The purpose of ERA is to ensure that the records of the federal government are preserved for as long as needed, independent of the original hardware or software that created them. ERA is to provide the technology to ensure that anyone, anywhere, anytime can access NARA’s electronic records holdings with the current technology that will be in use.

The system is to enable the general public, federal agencies, and NARA staff to search and access information about all types of federal records, whether in NARA custody or not, as well as to search for and access electronic records stored in the system. Using various search engines, the system is to provide the ability to create and execute searches, view search results, and select assets for output or presentation.

Figure 1 provides a simplified depiction of the system’s business concept.

As currently planned, the ERA system is to consist of six major components.

Ingest will enable the transfer of electronic records from federal agencies. Archival Storage will enable stored records to be managed in a way that guarantees their integrity and availability.

Dissemination will enable users to search descriptions and business data about all types of records, and to search the content of electronic records and retrieve them.

Records Management will support scheduling, appraisal, description, and requests to transfer custody of all types of records, as well as ingesting and managing electronic records, including the capture of selected records data (such as origination date, format, and disposition).

Preservation will enable secure and reliable storage of files in formats in which they were received, as well as creating backup copies for off-site storage.

Local Services & Control will regulate how the ERA components communicate with each other, manage internal security, and enable telecommunications and system network management. chedle i docment tht decri gency record, eablihe period for their retention y the gency, nd providendtory intrction for wht to do with them when they re no longer needed for crrent government business. pprsal i the process of determining the vnd the finl dipoition of record, mking them either temporry or permnent.

NARA currently plans to deliver these components in five separate increments: Increment 1 was deployed in two releases. Release 1 established the ERA base system—the hardware, software, and communications needed to deploy the system. Release 2 enabled functional archives with the ability to preserve electronic data in their original format, enable disposition agreements and scheduling, and receive unclassified and sensitive data from four federal agencies; according to NARA officials, this increment was completed in June 2008.

Increment 2 includes the EOP system, which was designed to handle records from the Executive Office of the President. The EOP system uses an architecture based on a commercial off-the-shelf product that supplies basic requirements, including rapid ingest of records and immediate and flexible search of content. Increment 2 includes basic case management for special access requests. This release was certified for initial operating capability (IOC) in December 2008.

The second release of Increment 2 and Increments 3 through 5 are to provide additional ERA functionality, such as public access. originl EOP pl inclded NtionSecrity Stem. NARA subseqently deferred the cability to inget classified ntionecrity d, ting tht the volme to e trferred from the Bush Adminitrtion did not support the eablihment of ll le classified EOP tem as plnned. Inted, NARA migrted the classified d from the Bush Adminitrtion to n exiting classified NARA preidentil liry tem. re req NARA receive from the crrent nd former dminitrtion, Congress, nd the cort for ccess to preidentil record.

Figure 2 shows the current incremental timetable for deploying ERA and the functionality planned for each increment.

Since 2002, we have issued several reports on ERA and its development. In May 2008, we testified that the development of the ERA base was proceeding according to the revised schedule, although it faced challenges in meeting several testing deadlines. We also testified that the timely completion of the EOP system was uncertain, in part, due to ongoing negotiations between NARA and Lockheed Martin on system capabilities, tim frames, and limited information about the nature of the records to be delivered. ne 17, 2002). Records Management: Planning for the Electronic Records Archives Has Improved, GAO-04-927 (Washington, D.C.: Sept. 23, 2004); Information Management: Acquisition of the Electronic Records Archives Is Progressing, GAO-05-802 (Washington, D.C.; Jly 15, 2005); Electronic Records Archives: The National Archives and Records Administration’s Fiscal Year 2006 Expenditure Plan, GAO-06-906 (Washington, D.C.: Ag. 18, 2006); nd Information Management: The National Archives and Records Administration’s Fiscal Year 2007 Expenditure Plan, GAO-07-987 (Washington, D.C.: Jly 27, 2007). y 14, 2008).

In September 2008 we reported that NARA did not fully comply with its enterprise life- cycle methodology because it had not yet developed a mitigation plan to process the outgoing administration’s records into the ERA system at the time of the January 2009 presidential transition. NARA intended to develop a mitigation plan at the end of 2008, when it expected to know more about the types and volume of the presidential records that it would receive. We reported that this proposed schedule would leave NARA little time to prepare for and implement the plan.

As of April 2009, the life-cycle cost for ERA through March 2012 was estimated at $551.4 million; the total life-cycle cost includes not only the development contract costs, but also program management, research and development, and program office support, among other things.

Table 1 shows the amount spent for ERA in fiscal year 2008. Table 2 shows the reported spending from the program’s inception to the end of fiscal year 2008.

In March 2009, NARA submitted a fiscal year 2009 expenditure plan as required to obtain the release of multiyear funds for ERA. The Omnibus Appropriations Act states that the agency could not obligate these funds until the appropriations committees reviewed and approved the expenditure plan. As of March 31, 2009, the appropriations committees have released $28.5 million of fiscal year 2009 multi-year funds with $17.3 million remaining to be released. Single-year funds appropriated for fiscal year 2009 totaled $21.2 million.

NARA’s estimated ERA obligations for fiscal year 2009, including both single-year and multi-year funds, are $67.6 million. NARA plans to spend $11.1 million of this amount on EOP, and the remainder on the ERA base system and ERA program. As of March 31, 2009, NARA had obligated $23.9 million. Table 3 shows how NARA planned to distribute funds across the ERA program in fiscal year 2009.

Objective 1: NARA’s expenditure plan satisfies the fiscal year 2009 legislative conditions.

Objective 2: NARA has partially implemented our previous recommendations In July 2008, we made two recommendations to NARA—developing a mitigation plan and enhancing ERA project oversight. NARA implemented one of our recommendations and partially implemented the other.

Objective 3: Observations about NARA’s ERA Acquisition and Expenditure Plan Observation 1: Cost, schedule, and performance data in the expenditure plan do not provide a clear picture of system progress.

NARA’s expenditure plan should include a sufficient level and scope of information for Congress to understand what system capabilities and benefits are to be delivered, by when, and at what costs, and what progress is being made against the commitments that were made in prior expenditure plans. However, NARA’s plan does not clearly show what functions have been delivered to date or what functions will be included in future increments and at what cost.

As of fiscal year 2008, NARA has spent $237.4 million for ERA, or about 43 percent of the program’s estimated life-cycle costs through 2012. NARA’s expenditure plan describes the results achieved with ERA funding at a high level. For example, as described in the plan, Increment 1, also known as the ERA base, includes the system hardware and software as well as some capability to preserve electronic data in their original format. Also, according to the plan, Increment 2, the EOP system, includes the capability to perform content searching and basic case management for presidential records. However, the plan does not provide a complete picture of what capabilities were delivered in these increments. In addition, even though NARA reported that planned functionality in both of these increments had been deferred to future increments, the plan does not provide cost estimates for the deferred functionality. As a result, it is not possible to fully identify what functionality has been delivered to date and at what cost.

Additionally, NARA states it will spend $21.3 million to continue development of Increment 3, including implementing a common architecture, extending storage capabilities, and providing public access and preservation capabilities. However, NARA’s fiscal year 2009 plan lacks specifics about the scope of these improvements. For example, it does not quantify by how much it will extend storage capacity. Also, NARA’s plan does not specify when these functions will be completed or how much will be spent on each one.

Finally, while NARA’s plan reports that fiscal year 2009 funds will be used to support operations and maintenance of the completed base and EOP increments, it does not specify how much will be needed to address issues deferred when the systems were accepted. For example, the plan does not specify the funding required during the fiscal year to address issues in the EOP system that were identified but deferred when the system was certified for initial operating capability in December 2008. NARA officials subsequently estimated that such efforts will cost $1.1 million.

NARA officials attributed the plan’s lack of specificity to ongoing negotiations with Lockheed Martin. NARA provided broad descriptions of tasks for future development to Lockheed and is currently working with the contractor to refine them, and expects Lockheed to provide a proposal for accomplishing these tasks in May 2009. It is also in negotiations over the costs of addressing issues that were deferred at IOC.

Without more specific information on what functionality has been and will be delivered and the costs associated with those functions, NARA and other interested parties will have limited abilities to measure overall program progress. In addition, until NARA specifies how it will use funds requested for the remainder of the year, congressional appropriators will lack information important for evaluating NARA’s request.

Observation 2: NARA is not consistently following best practices in Earned Value Management NARA’s expenditure plan states that, in managing ERA, the agency uses Earned Value Management (EVM) tools and requires the same of its contractors. EVM, if implemented appropriately, can provide objective reports of project status, produce early warning signs of impending schedule delays and cost overruns, and provide unbiased estimates of a program’s total costs. We recently published a set of best practices on cost estimation that addresses the use of EVM. Comparing NARA’s EVM data to those practices, we determined that NARA fully addressed only 5 of the 13 practices. For example, we found weaknesses within the EVM performance reports, including contractor reports of funds spent without work scheduled or completed, and work completed and funds spent where no work was planned. rch, 2009).

In addition, the program has not recently performed an integrated cost-schedule risk analysis. This type of analysis provides an estimate of the how much the program will cost upon completion and can be compared to the estimate derived from EVM data to determine if it is likely to be sound. NARA officials attributed these weaknesses, in part, to documentation that did not accurately reflect the program’s current status. By not consistently adhering to identified best practices, NARA’s program management data may be unreliable, likely hindering NARA’s ability to accurately monitor and report on program costs, schedule, and performance. Further detail on these best practices and our assessment can be found in attachment 1.

Observation 3: Although NARA certified initial operating capability for EOP, the system is not currently fulfilling its intended purpose.

NARA designed and developed the EOP system to support the transfer of electronic records at the end of the George W. Bush Administration as required by the Presidential Records Act. In fiscal year 2008, NARA obligated $27.9 million for the planning and development of EOP in order for the system to be ready for the presidential transition. The system was to ingest the electronic records and to support search, processing, and retrieval of records—particularly for “special access” requests for the ongoing business of Congress and the next administration—immediately after the presidential transition on January 20, 2009. Responding to special access requests from the current administration, Congress, and the courts is a critical part of NARA’s mission and one of the major capabilities intended for the EOP system. re req NARA receive from the crrent nd former dminitrtion, Congress, nd the cort for ccess to preidentil record. The prioritie re determined y NARA’ Office of Preidentil Lirie based on experience with the record of previous dminitrtion.

In December 2008, NARA certified IOC for the EOP system, based on successful completion of acceptance and security testing. In January 2009, NARA took custody of approximately 78.4 terabytes of unclassified Bush Administration electronic records, which agency officials characterized as all such records. Before the transition, NARA had estimated that it was going to receive almost 124 terabytes of presidential records. According to NARA officials, the difference between the estimated volume of records and the actual records received is due to inaccurate information provided by the previous administration.

NARA copied the records data to large storage arrays, which were initially stored in the Washington, D.C. area, then moved to the ERA facility in West Virginia, where ingestion of records into EOP takes place. As of April 27, only 2.3 terabytes of data have been fully ingested from the first storage array into the EOP system, where they are available for search and retrieval. This constitutes about 3 percent of all Bush Administration unclassified electronic records. NARA currently estimates that ingest of all 78.4 terabytes of unclassified records will not be complete until October 2009. Before the transition, NARA has estimated that ingest would be completed by May 2009. abt one trillion yte or abt 1,000 gigabyte.

NARA officials attributed delays, in part, to unexpected difficulties. For example, according to NARA officials, once they started using the EOP system, they discovered that records from certain White House systems were not being extracted in the expected format. As a result, it had to develop additional software tools to facilitate the full extraction of data from White House systems prior to ingest into EOP. In addition, in April 2009, NARA discovered that 31 terabytes of priority data that had been partially ingested between December 2008 and January 2009 was neither complete nor accurate because it was taken from an incomplete copy of the source system. NARA has started to re-copy data from what it believes to be a complete version of the source data.

Because the records had not been ingested into the EOP system, NARA had to use other systems to respond to requests for presidential records. As of April 24, 2009, NARA had received 43 special access requests for information on the Bush Administration. Only one of these requests used EOP for search, and no responsive records were found. To respond to 24 of these requests, NARA used the replicated White House systems described in its risk mitigation plan, which called for acquiring the software and related hardware used by the White House to manage a records management system and an image database. According to NARA officials, these replicated systems cost $570,000 to put into service. The sources used to respond to the 42 requests that did not use EOP are listed in table 6 below.

Observation 4: NARA lacks a contingency plan for ensuring continuity of the ERA system.

Contingency planning is a critical component of information protection. If normal operations are interrupted, network managers must be able to detect, mitigate, and recover from service disruptions while preserving access to vital information. Therefore, a contingency plan details emergency response, backup operations, and disaster recovery for information systems. It is important that these plans be clearly documented, communicated to potentially affected staff, updated to reflect current operations, and regularly tested.

The Federal Information Security Management Act (FISMA) requires each agency to develop, document, and implement an information security program that includes plans and procedures to ensure continuity of operations for information systems that support the agency’s operations and assets. The National Institute of Standards and Technology (NIST) requires that agencies’ systems have contingency plans and that the plans address, at a minimum, identification and notification of key personnel, plan activation, system recovery, and system reconstitution. Specifically, NIST identifies 10 security control activities related to contingency planning, including developing a formal contingency plan, training employees on their contingency roles and responsibilities, and identifying a geographically separate alternative processing site to support critical business functions in the event of a system failure or disruption. tionl Intitte of Sndrd nd Technology, Recommended Security Controls for Federal Information Systems, Specil Publiction 800-53 Reviion 1 (Githersburg, MD: Decemer 2006).

NARA has not developed a contingency plan for the ERA system to ensure availability in the event of a system failure or disruption. In December 2008, NARA issued an assessment that detailed the results of an evaluation of the security features associated with the ERA system. This assessment identified weaknesses related to all 10 of the contingency planning control activities, which were attributed to the system not having the required contingency plans. OMB guidancerequires that when an agency identifies security weaknesses, it must also develop a plan of action to address the identified weaknesses, including the resources required to fix the weaknesses, the scheduled completion date of each fix, and the current status of the mitigation. Accordingly, NARA developed a plan of action and milestone document which was to detail all the primary risk elements present in the system and the corresponding plan of actions to mitigate or correct identified deficiencies before the ERA system reached IOC. However, while the initial plan of action and milestones discussed fixes for eight contingency planning control activities, it did not address two others—contingency planning policies and procedures and telecommunications services. For those eight control activities mentioned in the document, no timetable was provided for completion to address the identified issues. A subsequent planning document was developed that provided further detail on the identified risks. gement nd Bdget, “Gidnce for Prepring nd Submitting Secrity Pl of Action nd Miletone,” Memorndm for the He of Exective Deprtment nd Agencie, Mitchell E. Dniel, Jr., Director, M-02-01, Octoer 17, 2001.

The revised plan of action and milestones document provided the scheduled dates of completion as well as the current status of the mitigation activities. However, this version did not mention the various risk categories (i.e., “low,” “medium,” “high”) and used a variety of terms to describe the status of each weakness (e.g., “pending,” “planned,” “ongoing”) without providing clear definitions of their meaning. Further, 10 of 11 items in the plan of action to address contingency planning weaknesses remain open as of April 2009.

In addition, NIST standards require that agencies back up information contained within a system and employ a mechanism that would restore the system after a disruption or failure. Following product acceptance tests for EOP, NARA reported that the backup and restore functions for the commercial off-the-shelf archiving product used at the ERA facility in West Virginia tested successfully, but there were concerns about the amount of time required to execute the process. In lab tests, the restore process took about 56 hours for 11 million files. This is significant because, while the backup is being performed, the replication of data must be stopped; otherwise it could bring the system to a halt. Subsequently, NARA officials stated that they have conducted two successful backups, but the restore process had not been fully tested to ensure that the combined backup and restore capability can be successfully implemented. The most recent backup, conducted at Rocket Center on March 20, 2009, took 95 hours and 38 minutes for 11.25 terabytes of data, but NARA could not provide any documentation to substantiate the success of the backups or information on the length of time needed to perform the restore function. timte tht it has received more thn 300 million file from the Bush Adminitrtion.

NARA considered the initial backup and restore test a success because it met the requirements set for the contractor. According to NARA’s requirements documentation, among other things, the Backup and Recovery service was supposed to “provide the capability to recover EOP ERA records as required to re-establish the system,” and “provide the capability to recover application files to re-establish the EOP ERA system.” However, as in the examples cited, none of the backup/restore requirements specified how long the backup and restore process should take. According to NARA officials, a full system contingency plan is currently under development.

Without a complete and successfully tested contingency plan, there is an increased risk that, in the event of a major system failure or disruption, NARA would not be able to effectively restore its information and ensure system availability. Such a significant risk severely limits the reliability of the system.

While NARA has continued to make progress on the ERA system, that progress cannot be fully quantified because NARA’s expenditure plan does not clearly identify what functions have already been delivered, how much was spent to provide each function, or how much is required to maintain the delivered increments. NARA’s current plan similarly lacks details on the functions to be provided in future increments and the costs associated with them, including development efforts scheduled to take place in the remainder of this year. In addition, although NARA has been using Earned Value Management to track program cost, schedule, and performance, weaknesses in its EVM data limit NARA’s ability to accurately report on the project’s progress. Without more specific and accurate information on the immediate and long-term goals of the program and the outcomes expected from its resulting efforts, NARA will be hindered in effectively monitoring and reporting on the cost, schedule, and performance of the ERA system and congressional appropriators will lack information necessary to evaluate the agency’s requests for funds.

Although NARA certified that the EOP portion of the ERA system had achieved initial operating capability as planned in December 2008, the system has been of limited use because of delays in ingesting electronic records into the system. Further, even though it has obligated nearly $40 million on EOP, NARA has instead answered requests for Bush electronic records using existing systems or replicated White House systems that cost less than $600,000. Under its current schedule, the Bush Administration records will not be fully ingested into the system until October 2009. However, even when the data are fully ingested, the system’s lack of a complete, fully tested contingency plan increases the risk that a system failure or disruption will result in the system being unavailable for several days or more.

We recommend that the Archivist of the United States take the following actions: Report to Congress on the specific outcomes to be achieved by ERA program funding for the remainder of fiscal year 2009.

Provide detailed information in future expenditure plans on what was spent and delivered for deployed increments of the ERA system and cost and functional delivery plans for future increments.

Strengthen the earned value process so that it follows the practices described in GAO’s guide and more reliable cost, schedule, and performance information can be included in future expenditure plans and monthly reports.

Include in NARA’s next expenditure plan an analysis of the costs and benefits of using the EOP system to respond to presidential records requests compared to other existing systems currently being used to respond to such requests.

Develop and implement a system contingency plan for ERA that follows contingency guidance for federal systems.

Agency Comments and Our Evaluation In written comments on a draft of this briefing, the Acting Archivist of the United States agreed with 4 of our 5 recommendations. Specifically, in response to our recommendation that NARA report to Congress on the specific outcomes to be achieved during the remainder of the year, she indicated that a briefing on that subject was provided on April 27, 2009. In addition, she agreed to: ensure that future expenditure plans include more details on the costs and functions of future increments, strive to comply with EVM best practices, and complete the ERA system contingency plan.

Agency Comments and Our Evaluation Regarding our recommendation that NARA report on the costs and benefits of using the EOP system to answer requests for presidential records compared to other systems, the Acting Archivist raised several points. First she wrote that the EOP system is accomplishing an enormous amount of work at exceptional speed, and that the system has the capability to transfer and ingest records as they became available. However, as indicated in our briefing, the EOP system has been of limited use to answering the requests received to date. Further, the ingest of presidential records into the system required the use of additional software tools and is now scheduled to take 5 months longer than NARA estimated before the transition, even though NARA received a significantly lower volume of records than anticipated. The Acting Archivist also wrote that EOP supports functions other than the search of presidential records, such as the lifecycle management of presidential records, and that comparisons between the replicated systems and EOP are not valid. While we acknowledge that EOP was designed to perform functions other than search, we disagree that a comparison between EOP and other systems would not be valid. Instead, we believe that an appropriate analysis of the costs and benefits of using the various systems should account for the need to satisfy non-search requirements, such as lifecycle management of records, using systems other than EOP.

Agency Comments and Our Evaluation Finally, the Acting Archivist wrote that the replicated systems were being used for a small subset of records. However, these systems include nearly half of the records NARA received, by volume. In addition, these same records have been the subject of most of the requests received to date. We also addressed the Acting Archivist’s technical comments, as appropriate.

NARA’s EVM processes only partially adhere to the thirteen practices that GAO has identified as necessary in developing a reliable EVM process. Specifically, NARA fully adhered to 5 of the 13 best practices in implementing EVM, partially adhered to 6, and did not adhere to 2. While NARA’s EVM data indicate that the project is meeting cost and some schedule targets, not adhering to best practices does not allow NARA to effectively measure progress or forecast cost overruns. See table 7 for details.

Appendix III: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the individual named above, key contributions to this report were made by James R. Sweetman, Jr., Assistant Director; Monica Perez Anatalio; Nabajyoti Barkakati; Carol Cha; Barbara Collier; Tisha Derricotte; Jennifer Echard; Pamlutricia Greenleaf; Kush Malhotra; Lee McCracken; and Tarunkant Mithani.