Information Management: Challenges in Implementing an Electronic Records Archive
Published: 2008-05-14T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Since 2001, the National Archives and Records Administration (NARA) has been working to develop a modern Electronic Records Archives (ERA) system, a major information system that is intended to preserve and provide access to massive volumes of all types and formats of electronic records. The system is being developed incrementally over several years; the first system increment is to provide an initial set of functions, with additional capabilities to be added in future increments. However, in 2007, NARA's contractor acknowledged that it would not be able to meet the planned date for the initial operational capability of the first ERA increment. GAO was asked to provide information on the steps that NARA has taken to respond to the delays encountered in the development. To prepare this testimony, GAO reviewed its previous work in this area, as well as the preliminary results of an ongoing performance audit. For this ongoing audit, GAO analyzed NARA reports, contract documents, and other material related to the ERA development project, and interviewed agency and contractor officials.
section_title: What GAO Found
paragraphs: NARA is working to overcome the ERA schedule delays that occurred in 2007 by changing to a two-pronged development strategy, but uncertainties remain. First, NARA developed plans to achieve an initial operational capability for the ERA system in June 2008 with somewhat reduced capabilities from those that had been planned. For this initial system, known as the "base" system, software development deadlines have been met, and testing began on schedule. However, NARA has extended some test periods beyond what was originally planned, leaving less time at the end of the schedule for completing final activities. Although officials remain confident that the schedule changes will not affect the date of the initial operational capability, problems uncovered through testing could lead to its delay. Archives officials said they are mitigating the risk of delays by closely monitoring the testing process. Second, the development delays of 2007 put at risk NARA's plan to use ERA to receive the presidential records of the Bush Administration in January 2009. In response to this risk, NARA and its contractor are pursuing a parallel development of a separate part of the system that is to be dedicated initially to the Bush records; this part of ERA--referred to as the "EOP (Executive Office of the President) system"--uses a different architecture from that of the base system: it is being built on a commercial product that can provide basic requirements for processing presidential electronic records, such as rapid ingest of records and ability to search content. Pursuing this as a separate development decouples the EOP system from dependence on the base system. However, completing the EOP system in time for the presidential transition remains uncertain, primarily because NARA and its contractor are still negotiating the precise scope of work and system requirements. These negotiations are challenging because, among other things, NARA does not know the exact nature of the presidential records that it is to receive in 2009. Although NARA and Bush Administration officials have held meetings on this topic, according to NARA officials, the Administration has not yet provided specific information on the volume and types of data to be transferred. System development is nonetheless proceeding based on NARA's volume estimates and the information available so far. According to NARA, developing the EOP system in time for the presidential transition is critical so that it can respond in a timely fashion to the information requirements of the Congress, the former and incumbent Presidents, and the courts. Challenges remain for the ERA program in both the near and long term. In the near term, NARA has to finish testing the base system, define the scope and requirements of the EOP system, and complete its development. In the long term, it plans to merge the two architectures of these systems into an integrated whole. Meeting these challenges will be important to achieving NARA's ultimate aims for the ERA system: preserving and providing access to all types and formats of electronic records.

section_title: Letter
paragraphs: I appreciate the opportunity to participate in today’s hearing on challenges  and progress in overseeing the preservation of our nation’s historical  documents. As you know, since 2001, the National Archives and Records  Administration (NARA or the Archives) has been working to develop a  modern Electronic Records Archives (ERA) system. This major  information system is intended to preserve and provide access to massive  volumes of all types and formats of electronic records. ERA is to automate  NARA’s records management and archiving life cycle; the system is to  consist of  infrastructure elements, such as hardware and operating systems;   business applications that will support the transfer, preservation,  dissemination, and management of all types of records and the  preservation of and online access to electronic records; and   a means for public access via the Internet.
In view of its complexity, the system is being developed incrementally  over several years; the first piece (or increment) of the ERA system is to  provide an initial set of functions, with additional capabilities to be added  in future increments. However, in 2007, NARA’s contractor acknowledged  that it would not be able to meet the planned date for the initial  operational capability of the first ERA increment.
As agreed, in my testimony today, I will provide information on the steps  that NARA has taken to respond to the delays encountered in the  development. My comments today are based on our previous work in this  area, as well as the preliminary results of an ongoing performance audit,  which began in February 2008 in response to a legislative mandate. For  this ongoing performance audit, we analyzed NARA reports, contract  documents, and other material related to the ERA development project,  interviewed agency and contractor officials, and observed an ERA  software demonstration. All work on which my statement is based was  conducted in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: The ability to find, organize, use, share, appropriately dispose of, and save  records—the essence of records management—is vital for the effective  functioning of the federal government. In the wake of the transition from  paper-based to electronic processes, records are increasingly electronic,  and the volumes of electronic records produced by federal agencies are  vast and rapidly growing, providing challenges to NARA as the nation’s  recordkeeper and archivist.
Besides sheer volume, other factors contributing to the challenge of  electronic records include their complexity and their dependence on  software and hardware. Electronic records come in many forms: text  documents, e-mails, Web pages, digital images, videotapes, maps,  spreadsheets, presentations, audio files, charts, drawings, databases,  satellite imagery, geographic information systems, and more. They may be  complex digital objects that contain embedded images (still and moving),  drawings, sounds, hyperlinks, or spreadsheets with computational  formulas. Some portions of electronic records, such as the content of  dynamic Web pages, are created on the fly from databases and exist only  during the viewing session. Others, such as e-mail, may contain multiple  attachments, and they may be threaded (that is, related e-mail messages  are linked into send–reply chains).
In addition, the computer operating systems and the hardware and  software that are used to create electronic documents can become  obsolete. If they do, they may leave behind records that cannot be read  without the original hardware and software. Further, the storage media for  these records are affected by both obsolescence and decay. Media may be  fragile, have limited shelf life, and become obsolete in a few years. For  example, few computers today have disk drives that can read information  stored on 8- or 5¼-inch diskettes, even if the diskettes themselves remain  readable.
In response to these widely recognized challenges, the Archives began a  research and development program to develop a modern archive for  electronic records. In 2001, NARA hired a contractor to develop policies  and plans to guide the overall acquisition of an electronic records system.  In December 2003, the agency released a request for proposals for the  design of ERA. In August 2004, NARA awarded two firm fixed-price  contracts for the design phase totaling about $20 million—one to Harris  Corporation and the other to Lockheed Martin Corporation. On September  8, 2005, NARA announced the selection of Lockheed Martin Corporation to  build the ERA system. The contract with Lockheed is a cost plus award fee  contract with a total value through 2012 of about $317 million. As of  March 2008, the life cycle cost of the system through 2011 was estimated  at $453 million; total life cycle cost includes not only the development  contract costs, but also program management, research and development,  and program office support, among other things.
The ERA system is planned as a comprehensive and systematic means for  preserving electronic records, free from dependence on any specific  hardware or software. NARA plans for the system to manage the entire  lifecycle of electronic records, from their ingestion into the system  through preservation and dissemination to customers. According to the  Archives, the most fundamental characteristic of ERA is that it must be  able to evolve over time to allow new types of electronic records to be  brought into the system and preserved.
As planned, the ERA system will consist of six major components, each of  which supports a specific business area:  “Ingest” enables transfer of electronic records from federal agencies. “Managed Storage” enables stored records to be managed in a way that  guarantees their integrity and availability. “Dissemination” enables users to search descriptions and business data  about all types of records, and to search the content of electronic records  and retrieve them. “Records Management” supports scheduling, appraisal, description, and  requests to transfer custody (from agencies to the Archives) of all types of  records, as well as ingesting and managing electronic records, including  the capture of selected records data (such as origination date, format, and  disposition). “Preservation” enables secure and reliable storage of files in formats in  which they were received, as well as creating backup copies for offsite  storage.  “Local Services and Control” regulates how the ERA components  communicate with each other, manages internal security, and enables  telecommunications and system network management.
Acquiring a major information technology system like ERA is a significant  challenge for a relatively small organization such as NARA, which had no  previous experience in acquiring major information systems. In 2002 and  2003, we pointed out this and other challenges facing the ERA project: for  example, the solution to the electronic records archiving challenge had not  been developed, and no electronic archive system existed that was  comparable in complexity or scale to NARA’s vision. Further, some key  technologies associated with electronic document archiving and storage  were not available commercially. As a result, ERA was technically and  managerially complex and challenging, requiring the development of an  advanced architecture for the conversion and preservation of electronic  records.
In view of this complexity, the ERA program planned to use an acquisition  and implementation approach in which the system would be deployed in  five separate increments. The increments were to provide the following  capabilities:  Increment 1: provide the hardware, software, and communications needed  to deploy the system; capabilities for unclassified and sensitive records  (enabling online transfer of electronic records to NARA, annual transfers  of records, and secure storage of electronic records in original formats);  and support for certain records management functions (such as  scheduling and appraisal).
Increment 2: provide additional capabilities, such as content searching,  responding to Freedom of Information Act requests, and preservation  planning. It is also to handle classified data.
Increment 3: provide public access.
Increments 4 and 5: provide additional preservation capability and storage.
Under the ERA program plan, developed in September 2006, Increment 1  was to take 2 years to develop, and the remaining increments were each to  take 1 year.
Among the requirements that ERA is to fulfill are storing and providing  access to electronic records from past presidential administrations. The  Presidential Records Act gives the Archivist of the United States  responsibility for the custody, control, and preservation of presidential  records upon the conclusion of a President’s term of office. The act states  that the Archivist has an affirmative duty to make such records available  to the public as rapidly and completely as possible consistent with the  provisions of the act. At the time of the last presidential transition, NARA  met these requirements by recreating the Clinton Administration’s  computer systems (software and hardware) that originally held the  records and developing simple search interfaces so that NARA personnel  could search for requested information. NARA reported that it took about  400 days to process the 2 terabytes of data from the Clinton  administration. According to NARA, it planned to use ERA to receive the  Bush Administration’s presidential records when the Archivist takes  custody of them in January 2009. The ERA schedule had planned for the  required capabilities (part of Increment 2) to be available in mid-2008.
We have issued several reports on ERA and its development. Most  recently, in July 2007, we reported that NARA had taken action to  implement our prior recommendations. The agency (1) established a  baseline and target architecture, a plan to transition from the baseline to  the target, and an architecture review board; (2) revised four of five key  acquisition policies and planned to comply with appropriate standards;  and (3) developed a fiscal year 2007 expenditure plan that contained an  appropriate level and scope of information needed for the Congress to  understand its plans and commitments relative to system capabilities,  benefits, schedules, and costs.
However, we also reported that the ERA project had experienced schedule  delays and cost increases. Factors contributing to the delays included low  productivity of contractor software programmers, difficulties in securing  an acceptable contract to prepare the site that was to house the system,  and problems with software integration. Although the contractor took  actions to address the factors causing these delays and overruns, including  replacing several programmers with a more experienced software  development staff, NARA concluded that the project schedule should be  revised so that reduced functionality would be available by the initial  operational capability date. Subsequently, the schedule as of July 2007  delayed the date of the initial operational capability (IOC) from September  7, 2007, to March 31, 2008. NARA and the contractor projected that the  estimated cost at completion of the IOC would be between $8 million and  $12 million over budget. In May 2007, NARA and the contractor were in the  process of negotiating additional mitigating actions needed and changes to  the project’s cost and schedule.
section_title: NARA Is Working to Overcome ERA Schedule Delays through Parallel Development Projects, but Uncertainties Remain
paragraphs: In response to the delays that occurred in 2007 and their effect on the  Archives’ readiness to receive the presidential records of the Bush  Administration in January 2009, NARA developed a two-pronged strategy  for continuing ERA development. First, it has developed plans to achieve  IOC by June 2008 with somewhat reduced capabilities from those that had  been planned; it refers to the system to be delivered at IOC as the “base”  ERA system. Second, it is planning to pursue a parallel development of  another part of the ERA system that is to be dedicated initially to  presidential records from the Bush Administration; this part of ERA,  which is being developed using a different architecture from that of the  base system, is referred to as the “EOP (Executive Office of the President)  system.” When these developments are complete, NARA plans to merge  the two architectures into one ERA system.
This strategy was developed through a series of negotiations that followed  the contractor’s acknowledgement that the IOC deadline would not be  met. In July 2007, NARA issued a cure notice to the developer, asking for  a recovery plan that would include revised delivery dates, complete cost  and schedule estimates, and a cost overrun sharing plan. In response to  NARA’s cure notice, the contractor provided a plan in August to deliver  software in three iterations (or “drops”) that would lead to IOC in May  2008; as part of this plan, the software to be delivered would not include  all the capabilities that had been planned: for example, support for certain  records management functions was postponed. After review of the  contactor’s proposed plan, NARA determined that more time would be  needed to complete and test the software and achieve security  certification and accreditation of the system; accordingly, it adjusted the  schedule to put IOC at the end of June 2008.
In addition, in September 2007, the contractor demonstrated to NARA a  prototype for the EOP system that would address basic requirements for  processing presidential electronic records, such as rapid ingest of records  and ability to search content. To ensure that a system to receive  presidential records would be available by the next change of  Administration, NARA decided to pursue development of the contractor’s  prototype system.
Although NARA’s cure notice asked for a cost overrun sharing plan, the  contractor did not offer such a plan. According to NARA, the contractor  developed the prototype EOP system at its own expense, in partial  mitigation of the cost overrun incurred in 2007. Officials also stated that  the contractor was independently funding research on issues related to  preservation, whose results would be applied to future work. In addition,  the Archives applied the development contractor’s allocated award fee to  the cost overrun. According to NARA, it would continue to work with the  development contractor to devise ways to mitigate the cost overrun  without loss of functionality, but officials acknowledged that under a cost  plus contract, achieving such mitigation was uncertain.
In December 2007, NARA estimated a cost overrun for the development of  ERA of about $15 million. NARA now estimates this cost overrun at $16.3  million.
subsections: section_title: ERA Base System Is Generally on Schedule to Achieve IOC, but Testing Delays Are a Risk
paragraphs: Under the most recent plan, the base system is to achieve IOC at the end  of June 2008; this is a 9-month delay from the earlier plan (of September  2006) for an IOC in September 2007. In carrying out this plan, the  contractor met all deadlines for delivering the software drops, as shown in  table 1.
In preparation for achieving IOC, NARA is currently conducting a series of  tests aimed at verifying that ERA requirements are met. As part of this  process, according to NARA officials, a NARA “Tiger Team” meets weekly  to discuss testing and other activities leading to IOC and may adjust the  schedule in response to test progress and results. Table 2 shows the status  of the testing compared to the current schedule and the high-level  schedule provided by the contractor.
As the table shows, the dates for product acceptance testing and  operational acceptance testing have been extended, and later test dates  have been adjusted in response. According to NARA, it decided to extend  the time to ensure adequate time for government testing review of the  delivered systems and related documentation. Further, NARA stated that  when the Tiger Team analyzed the contractor’s high-level schedule, it  determined that NARA’s systems engineering test team required 6 weeks  for product acceptance testing, and the end users required only 1 week for  operational acceptance testing. Accordingly, product acceptance testing  was extended to May 23, and operational acceptance testing was  scheduled for the end of that period. NARA officials remain confident that  their testing will be completed and necessary fixes made in time for IOC at  the end of June.
However, the ERA program faces near-term challenges in achieving IOC of  the base system. Unforeseen problems uncovered through testing could  lead to further delays. According to NARA, it is mitigating the risk of  delays by paying close and continuing attention to the testing process,  through such actions as the weekly meetings of NARA’s test team. Delays  in achieving IOC could lead to further delays in the development of ERA  and the success of the Archives’ long-term plan for electronic records  management and preservation.
section_title: EOP System Is Being Developed, but Completing the Development in Time for the Presidential Transition Is Uncertain
paragraphs: In September 2007, the contractor demonstrated to NARA a prototype for  the EOP system, designed to show the ability to address basic  requirements for processing presidential electronic records, such as rapid  ingest of records and ability to search content. To ensure that the EOP  system would be available by the change of Administration, NARA decided  to modify its architectural approach to the EOP system by continuing  development of the system along the lines provided by the contractor’s  prototype. However, details of the system requirements and capabilities,  as well as the specific development plans, are still being defined and final  contract terms negotiated while development continues.
Under the new approach, the EOP system would be separately designed,  use a dedicated system team (separate from the team working on the ERA  base), and be developed concurrently with the work to complete the base  system. The EOP system is to use an architecture based on a commercial  off-the-shelf (COTS) product that supplies basic EOP requirements,  including rapid ingest of records and immediate and flexible search of  content. (According to NARA, the EOP system is also to use key elements  of the base system infrastructure.) Developing the EOP system in this way  decouples it from dependence on the completion of IOC for the base  system.
From October 2007 to March 2008, NARA spent $13 million on EOP system  planning and development, of which $7 million was used to buy hardware.
At this point, the contractor has delivered three initial software drops for  the EOP system; it demonstrated Drop 3 software to NARA on April 28 and  29. (As of March 2008, two additional drops were planned, but the ultimate  number of software drops is still being negotiated.)
Table 3 shows the dates that the contractor is to provide the planned  drops as well as selected functions associated with each. As the table  shows, the first three drops were provided later than originally planned.
Although the development is continuing, challenges remain. In particular,  because NARA and the contractor are still negotiating the detailed  development plans and specific system requirements, the IOC date for the  EOP system and specific details regarding functionality remain uncertain.
The scope of work and requirements for the EOP are not settled. In  response to NARA’s request for a proposal describing detailed plans for  developing the system, the contractor delivered a proposal in February  2008. After review, NARA asked the contractor to respond to over 200  comments and provide a revised proposal. The revision, delivered April 25,  2008, is currently being reviewed. NARA expects to complete its review on  or before May 16, and it expects to reach agreement with its contractor no  later than May 30, with a signed contract modification by mid-June.
Currently, NARA and its contractor plan to develop physically separate  EOP system segments (or “instances”) for classified and unclassified  presidential records, with IOC for the unclassified instance preceding that  for the classified instance. NARA is planning for both instances to achieve  IOC by the end of 2008, but the dates are not settled.
Among the difficulties of finalizing the negotiations are uncertainties  regarding the exact nature of the presidential records that will be  transferred to NARA in 2009. Although NARA and the Administration have  held meetings on this topic, according to NARA, the Administration has  not yet provided NARA with specific information required for it to plan for  reliable ingest, indexing, and accessing of the electronic records involved.  Both the volume and many of the data types are uncertain:    NARA has estimated that the volume of data from the Bush Administration  will likely be 100 terabytes—about 50 times greater than that received  from the Clinton Administration, and it has based its capacity  requirements on that estimate.
The COTS product on which the EOP system is based supports about 370  common data formats, so it can already accommodate some portion of the  Bush Administration records. NARA has also obtained limited information  regarding the e-mail and image records that will be transferred. However,  it has not received details on the data types and formats for about 32  systems for which the Administration has provided brief descriptions, nor  has it received sample data on which it could base its planning. According  to NARA officials, if further information is received too late to be included  in initial system development, they would plan, where possible, to convert  the data types received to a generic format that would allow the EOP  system to ingest and search the information.
These uncertainties mean that achieving a working EOP system in time for  the presidential transition in January 2009 will be a challenging goal.  According to NARA, if it cannot ingest the electronic classified and  unclassified records from the Bush Administration in a way that supports  the search, processing, and retrieval of records immediately after the  presidential transition, it will not be able to meet the requirements of the  Congress, the former and incumbent Presidents, and the courts for  information in these records in a timely fashion.
In summary, NARA has developed a strategy to overcome the earlier ERA  schedule delays: it has developed the software and begun testing for the  base system, and it has begun development of the EOP system. However,  in the near term it faces the challenge of completing the testing of the base  system and the larger challenge of defining the requirements and scope of  the EOP system and completing its development. In the long term, NARA  also plans to merge the two architectures of the ERA base and EOP  systems into a coordinated whole. Continuing careful oversight by NARA  and the Congress will be important in achieving the ultimate aims for the  ERA system: to automate NARA’s records management and archiving life  cycle and preserve and provide access to all types and formats of  electronic records.
Mr. Chairman, this concludes my testimony today. I would be happy to  answer any questions you or other members of the subcommittee may  have.
section_title: Contacts and Acknowledgements
paragraphs: If you have any questions concerning this testimony, please contact Linda  Koontz, Director, Information Management Issues, at (202) 512-6240, or  koontzl@gao.gov. Other individuals who made key contributions to this  testimony were Barbara Collier, Pamlutricia Greenleaf, Charles C.  Hinnant, Tarunkant N. Mithani, and James R. Sweetman, Jr.
This is a work of the U.S. government and is not subject to copyright protection in the  United States. This published product may be reproduced and distributed in its entirety  without further permission from GAO. However, because this work may contain  copyrighted images or other material, permission from the copyright holder may be  necessary if you wish to reproduce this material separately.