Department of Energy: Major Construction Projects Need a Consistent Approach for Assessing Technology Readiness to Help Avoid Cost Increases and Delays
Published: 2007-03-27T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Department of Energy (DOE) spends billions of dollars on major construction projects that help maintain the nuclear weapons stockpile, conduct research and development, and process nuclear waste so that it can be disposed of. Because of DOE's long-standing project management problems, GAO determined the extent to which (1) DOE's major construction projects are having cost increases and schedule delays and the major factors contributing to these problems and (2) DOE ensures that project designs are sufficiently complete before construction begins to help avoid cost increases and delays. We examined 12 DOE major projects with total costs of about $27 billion, spoke with federal and contractor officials, and reviewed project management documents.
section_title: What GAO Found
paragraphs: Of the 12 DOE major projects GAO reviewed, 9 exceeded their original cost or schedule estimates, principally because of ineffective DOE project oversight and poor contractor management. Specifically, 8 of the 12 projects experienced cost increases ranging from $79.0 million to $7.9 billion, and 9 of the 12 projects were behind schedule by 9 months to more than 11 years. Project oversight problems included, among other things, inadequate systems for measuring contractor performance, approval of construction activities before final designs were sufficiently complete, ineffective project reviews, and insufficient DOE staffing. Furthermore, contractors poorly managed the development and integration of the technology used in the projects by, among other things, not accurately anticipating the cost and time that would be required to carry out the highly complex tasks involved. Even though DOE requires final project designs to be sufficiently complete before beginning construction, it has not systematically ensured that the critical technologies reflected in these designs have been demonstrated to work as intended (technology readiness) before committing to construction expenses. Specifically, only one of the five DOE project directors with projects that have recently begun or are nearing construction had systematically assessed technology readiness. The other four directors also told us that they have or will have completed prior to construction, 85 to 100 percent of their projects' final design, but they had not systematically assessed technology readiness. Proceeding into construction without also demonstrating a technology's readiness can lead to cost increases and delays. For example, one technology to be used in DOE's Waste Treatment and Immobilization Plant was not sufficiently demonstrated--that is, shown to be technologically ready for its intended application--before construction began. Consequently, the technology did not perform as expected, which resulted in about $225 million in redesign costs and schedule delays of more than 1 year. To help avoid these problems, the National Aeronautics and Space Administration (NASA) pioneered and the Department of Defense (DOD) has adopted for its projects a method for measuring and communicating technology readiness levels (TRL). Using a scale from one (basic principles observed) through nine (total system used successfully in project operations), TRLs show the extent to which technologies have been demonstrated to work as intended in the project. DOE project directors agreed that such an approach would help make technology assessments more transparent and improve stakeholder communication prior to making critical project decisions, such as authorizing construction.

section_title: Letter
paragraphs: The Department of Energy (DOE) spends billions of dollars on major  construction projects that, among other things, are used to help maintain  the nuclear weapons stockpile, conduct research and development in the  areas of high-energy physics and nuclear physics, and process nuclear  waste into forms suitable for longer-term storage or permanent disposal.  DOE oversees the construction of facilities primarily at government- owned, contractor-operated sites throughout the nation. In July 2006, DOE  revised its dollar threshold defining a construction project as “major”; it is  now at $750 million, up from $400 million when we began our review in  December 2005. The following 12 projects included in our review had  estimated project costs exceeding the original threshold. The total cost of  these projects is currently estimated at about $27 billion. These 12  projects and their locations are as follows:    Chemistry and Metallurgy Research Facility Replacement—Los Alamos  National Laboratory, Los Alamos, New Mexico.
Depleted Uranium Hexafluoride 6 Conversion Facility—Portsmouth, Ohio,  and Paducah, Kentucky.
Highly Enriched Uranium Materials Facility—Y-12 National Security  Complex, Oak Ridge, Tennessee.
Linac Coherent Light Source—Stanford Linear Accelerator Center, Menlo  Park, California.
Microsystems and Engineering Sciences Applications—Sandia National  Laboratories, Albuquerque, New Mexico.
Mixed Oxide Fuel Fabrication Facility—Savannah River Site, Aiken, South  Carolina.
National Ignition Facility—Lawrence Livermore National Laboratory,  Livermore, California.
Pit Disassembly and Conversion Facility—Savannah River Site, Aiken,  South Carolina.
Salt Waste Processing Facility—Savannah River Site, Aiken, South  Carolina.
Spallation Neutron Source—Oak Ridge National Laboratory, Oak Ridge,  Tennessee.
Tritium Extraction Facility—Savannah River Site, Aiken, South Carolina.
Waste Treatment and Immobilization Plant—Hanford Site, Richland,  Washington.
These major projects require the construction of large building complexes  and the development of innovative cleanup and other technologies. Many  of these technologies are developed for the project or are applied in a new  way. DOE project directors are responsible for managing these major  projects and overseeing the contractors that design and construct the  facilities. In doing so, project directors follow specific departmental  directives, policies, and guidance for project management. Among these  are DOE Order 413.3A and Manual 413.3-1, which establish protocols for  planning and executing a project. The protocols require DOE projects to  go through a series of five critical decisions as they enter each new phase  of work. Two of the decisions made before construction begins are key:  (1) formally approving the project’s definitive cost and schedule estimates  as accurate and complete—an approval that is to be based on a review of  the project’s completed preliminary design, and (2) reaching agreement  that the project’s final design is sufficiently complete and that resources  can be committed toward procurement and construction. To oversee  projects and approve these critical decisions, DOE conducts its own  reviews, often with the help of independent technical experts. In addition,  projects are regularly subject to reviews by DOE’s Office of Engineering  and Construction Management and it’s Office of Inspector General, the  Department of Defense’s (DOD) U.S. Army Corps of Engineers, the  Defense Nuclear Facilities Safety Board, and the National Research  Council, among others.
We and others have reported over the past decade that project  management weaknesses have impaired these projects. For example, for  the Waste Treatment and Immobilization Plant, we reported that DOE’s  use of a “fast-track, design-build” approach to construction—in which  design and construction activities overlap—has been problematic for  highly complex, first-of-a-kind facilities. We found that the designs for  these facilities were not sufficiently complete for construction to begin,  which has resulted in significant cost increases and schedule delays.
In this context, we determined the extent to which (1) DOE’s active major  construction projects are experiencing cost increases and schedule delays  and the key factors contributing to these problems and (2) DOE ensures  that project designs are sufficiently complete before construction begins  to help avoid cost increases and schedule delays.
To determine the extent to which DOE projects are experiencing cost  increases or schedule delays and factors contributing to these problems,  we sent a survey to the 12 DOE directors of major projects and reviewed  the project management documents for these projects—6 projects that  were above the $750 million threshold, 2 estimated to cost between the  previous $400 million threshold and $750 million, and 4 estimated to cost  between $300 million and $400 million. (App. II describes these projects.)  These 12 projects are managed by DOE’s Office of Science, Office of  Environmental Management, or National Nuclear Security Administration  (NNSA). We conducted site visits and analyzed independent project  studies for the 9 projects that experienced cost increases or schedule  delays. During the course of our review, we identified a method used by  the DOE project director for the Pit Disassembly and Conversion Facility  to systematically assess the extent to which a technology is sufficiently  developed for its intended purpose. The project director based this  method on a system developed by the National Aeronautics and Space  Administration (NASA). We had previously reported on the use of a similar  assessment system—technology readiness levels (TRL), which DOD has  adopted for its major projects. We obtained and reviewed documents  regarding these two assessment systems.
To determine the extent to which DOE ensures that project designs are  sufficiently complete before construction, we reviewed in detail 5 of the   12 projects that were approaching or had recently begun construction.  Specifically, we obtained information on the extent to which project  designs were, or are expected to be, complete before beginning  construction, and the actions DOE has taken to ensure that technologies  used in these designs are sufficiently ready to begin construction.
Because we and others have previously expressed concern about the data  reliability of a key DOE project management tracking database—the  Project Assessment and Reporting System—we did not develop  conclusions or findings on the basis of information generated through that  system. Instead, we collected information directly from project site  offices. In addition, we spoke with officials from DOE program offices and  DOE’s Office of Engineering and Construction Management in  Washington, D.C. We provided interim briefings to the Subcommittee on  the status of our work in May and September, 2006. We performed our  work between December 2005 and January 2007, in accordance with  generally accepted government auditing standards. Appendix I contains a  detailed description of our scope and methodology.
subsections: section_title: Background
paragraphs: To meet its diverse missions, DOE pays its contractors billions of dollars  each year to implement hundreds of projects, ranging from hazardous  waste cleanups at sites in the weapons complex to the construction of  scientific facilities. Many of these complex, unique projects are designed  to meet defense, energy research, environmental, and fissile materials  disposition goals. They often rely on technologies that are unproven in  operational conditions. In recent years, DOE’s budget has been dominated  by the monumental task of environmental restoration and waste  management to repair damage caused by the past production of nuclear  weapons.
DOE has long had a poor track record for developing designs and cost  estimates and managing projects. We reported in 1997 that from 1980 to  1996, 31 of DOE’s 80 major projects were terminated prior to completion,  after expenditures of over $10 billion; 15 of the projects were completed,  but most of them were finished behind schedule and with cost overruns;  and the remaining 34 ongoing projects also were experiencing schedule  slippage or cost overruns. In addition, for over a decade, DOE’s Office of  Inspector General, the National Academy of Sciences, and others have  identified problems with DOE’s management of major construction  projects. Projects were late or never finished; project costs increased by  millions and sometimes billions of dollars; and environmental conditions  at the sites did not significantly improve. According to the National  Research Council, DOE’s construction and environmental remediation  projects take much longer and cost about 50 percent more than  comparable projects by other federal agencies or projects in the private  sector. A 2004 assessment of departmental project management  completed by the Civil Engineering Research Foundation recommended,  among other things, that DOE develop a core group of highly qualified  project directors and require peer reviews for first-of-a-kind and  technically complex projects when the projects’ preliminary baselines are  approved.
To address project management issues, DOE began a series of reforms in  the 1990s that included efforts to strengthen project management  practices. To guide these reforms, the department formed the Office of  Engineering and Construction Management in 1999. The reforms instituted  to date have included planning, organizing, and tracking project activities,  costs, and schedules; training to ensure that federal project managers had  the required expertise to manage projects; increasing emphasis on  independent reviews; and strengthening project reporting and oversight.
section_title: Most Major Projects Have Exceeded Original Costs and Are Years Late, Principally Because of Ineffective DOE Project Oversight and Contractor Management
paragraphs: The estimated costs of many of the DOE major construction projects we  reviewed have significantly exceeded original estimates and schedules  have slipped. On the basis of our analysis of independent project studies  and interviews with project directors, cost growth and schedule slippage  occurred principally because of ineffective DOE project oversight and  poor contractor project management. Furthermore, unreliable initial cost  and schedule estimates resulting from a now-defunct policy may have  been a contributing factor, according to DOE project officials. Although  external factors, such as additional security and safety requirements,  contributed to cost growth and delays, the management of these  requirements was complicated by ineffective and untimely DOE  communication and decision making.
Eight of the 12 DOE projects we reviewed had increases in estimates of  total project cost (TPC) ranging from $79.0 million to $7.9 billion. As table  1 shows, the percentage of cost increase for these 8 projects ranged from   2 percent to over 200 percent.
In addition, as shown in table 2, 9 of the 12 projects experienced schedule  delays ranging from 9 months to more than 11 years. Of the 9 projects, 7  had schedule delays of at least 2 years or more.
As table 3 shows, ineffective DOE project oversight and poor contractor  management were frequently cited reasons for cost increases and  schedule delays for the projects we reviewed, according to our review of  independent studies of the 9 projects experiencing cost growth and  schedule delays and our follow-up interviews with DOE project directors.  Project officials, in commenting on our draft report, were concerned that  table 3 might misrepresent the overall successful execution and  completion of some projects, such as the Spallation Neutron Source, and  that some problems may have already been addressed. Nevertheless, to  clarify our main purpose for table 3, our intent is to show broad categories  of major reasons for cost increases and schedule delays, regardless of  when they occurred or whether they have been adequately addressed.
The DOE project oversight issues mentioned in table 3 include the  following:  inadequate systems for measuring contractor performance;   approval of construction activities before final designs were sufficiently  insufficient DOE staffing and experience; inadequate use of project management controls;  lack of headquarters assistance and oversight support of field project  directors;  failure to detect contractor performance problems, including inadequate  federal inspection activities; and   poor government cost estimates, including inadequate funding for  contingencies.
DOE’s lack of adequate systems to measure contractor performance was  cited in a December 2005 DOE Inspector General review of the Mixed  Oxide Fuel Fabrication Facility. The Inspector General criticized DOE’s  NNSA for failing to approve a baseline against which to measure  contractor performance and relying on outdated cost plans. According to  the report, NNSA relied on confusing and misleading information detailed  in the monthly project reports to monitor progress and track costs— reports that the contractor acknowledged as being “useless for evaluating  performance or managing the project.” Furthermore, although the  contractor reported unfavorable cost and schedule variances for months,  these variances were inaccurate and meaningless because performance  was being compared against a 2-year-old plan. NNSA, in commenting on  our draft report, stated that project oversight and contractor management  problems identified in previous GAO, Inspector General, and other  independent assessments have led to extensive improvements to the  project, and that major findings identified during a recent independent  review have been successfully addressed.
Similarly, DOE’s approval of construction activities before final designs  were sufficiently complete has contributed significantly to project cost  growth and schedule delays. As we have previously reported, the  accelerated fast-track, design-build approach used for the Waste  Treatment and Immobilization Plant, a highly complex first-of-a-kind  nuclear facility, resulted in significant cost increases and schedule delays.  DOE also allowed the contractor on another project, the Tritium  Extraction Facility, to begin construction before the final design was  completed to meet schedule commitments. According to a 2002 DOE  Inspector General report on the project, this revised acquisition strategy  of simultaneous design and construction directly resulted in at least   $12 million in project overruns.
The contractor management issues mentioned in table 3 include the poor  management of technological challenges, among other contractor  performance issues, according to DOE project directors. Cost increases  and schedule delays for 6 of the 9 projects were due in part to contractors’  poor management of the development and integration of technologies  used in project designs by, among other things, not accurately anticipating  the cost and time that would be required to carry out the highly complex  tasks involved. For example:    The National Ignition Facility had over $1 billion in cost overruns and  years of schedule delays, in large part because of technology integration  problems. The requirements for the National Ignition Facility—the use of  192 high-power laser beams focused on a single target in a “clean room”  environment—had not been attempted before on such a large scale.  According to the DOE project director, early incorrect assumptions about  the original facility design and the amount of work necessary to integrate  the technologies and assemble the technical components contributed to  about half of the project’s cost increases and schedule delays.
The design of the Mixed Oxide Fuel Fabrication Facility has presented  technical challenges in adapting the design of a similar plant in France to  the design needs of this project. Although the technological challenge  related to adopting the process designs from the French designs was not  the primary contributor to the project’s cost increases and schedule  delays, according to NNSA officials, it has affected the project’s  complexity. The basic technology—combining plutonium oxide with  depleted uranium to form fuel assemblies for use in commercial power  reactors—has been previously demonstrated in France. However, the DOE  project director told us that the DOE facility design must, among other  things, account for processing surplus weapon-grade plutonium, a  different type of material than processed in the French facility, and must  be adapted to satisfy U.S. regulatory and other local requirements. In  addition, the DOE facility faced the technological challenge of reducing  the scale of components used in the French facility. Although definitive  cost estimates are not yet available, expected costs for completing this  project have grown by about $3.3 billion since 2002, and the schedule has  been extended by more than 11 years, in part because the contractor did  not initially understand the project’s complexity and underestimated the  level of effort needed to complete the work. NNSA explained that the  capability of the reference plants currently in operation in France, and by  extension, the Mixed Oxide Fuel Fabrication Facility process design, is  currently being demonstrated by several prototype fuel assemblies  manufactured with weapon-grade plutonium oxide, which are currently  being successfully used in a reactor in South Carolina.
For the Waste Treatment and Immobilization Plant, a technology  application used on the project had not been tested before construction.  Filters, widely used in the water treatment industry, were being designed  for the project to concentrate and remove radioactive particles in liquid  waste, a new application for the filters. Although tests are currently under  way to demonstrate the effectiveness of this application, project officials  conceded that these filters may still not be appropriate for the project.
Other contractor performance problems are illustrated by two examples.  First, DOE cited the contractor working on the Highly Enriched Uranium  Materials Facility for inadequate quality assurance that resulted in  concrete work that did not meet design specifications. The subsequent  suspension of construction activities and rework added to the project’s  estimated cost and schedule. Second, the DOE project director of the  Depleted Uranium Hexafluoride 6 Conversion Facility told us that the  project was delayed 2 years because the contractor (1) did not have  experience in government contracts, (2) underestimated the design effort  needed, and (3) failed to properly integrate the operations of three  separate organizations it managed.
As table 3 shows, external factors were cited as also contributing to cost  growth and schedule delays, such as additional work to implement  requirements for higher levels of safety and security in project operations,  among other things. For example, design rework for 4 of the projects  occurred in response to external safety oversight recommendations by the  Defense Nuclear Facilities Safety Board that large DOE construction  projects meet a certain level of personnel safety, and that their designs be  robust enough to withstand certain seismic events. In addition, owing to  new security requirements implemented after September 11, 2001, project  officials on the Highly Enriched Uranium Materials Facility had to  redesign some aspects of the project to ensure that heightened security  measures were addressed.
While DOE faced additional requirements for safety and security, it did not  always reach timely decisions on how to implement these requirements,  which contributed significantly to cost increases and schedule delays for  the Salt Waste Processing Facility. The DOE project director for this  project told us the Defense Nuclear Facilities Safety Board had expressed  concerns in June 2004, 5 months after the preliminary design was started,  that the facility design might not ensure nuclear wastes would be  adequately contained in the event of earthquakes. However, DOE did not  decide how to address this concern until 17 months later, as the project  continued to move forward with the existing project design. According to  the project director, better and more timely discussions between site  officials and headquarters to decide on the actions needed to adequately  address these safety and security requirements might have hastened  resolution of the problem, and up to 1 year of design rework might have  been avoided. The delay, the director told us, added $180 million to the  total project cost and extended the schedule by 26 months. In commenting  on our draft report, EM officials noted that it is now requiring a more  rigorous safety analysis earlier in the decision-making process.
Other external factors also contributed significantly to cost increases and  delays for 2 interrelated projects we reviewed—the Mixed Oxide Fuel  Fabrication Facility and the Pit Disassembly and Conversion Facility.  Project officials for these projects told us that 25 to 50 percent of the cost  increases and over 70 percent of the schedule delays they experienced  were the direct result of Office of Management and Budget funding  constraints and restrictions resulting from international agreements with  Russia. That is, work that is delayed to a subsequent year because of  funding constraints and other work restrictions can delay project  completion, which likely increases total project costs. Similarly, Office of  Science officials, commenting on our draft report, stated that external  factors caused the largest percentage cost increase and schedule delay for  the Spallation Neutron Source, including a reduced level of funding  appropriated at a time when project activities and costs were increasing  considerably. However, congressional funding was reduced in fiscal year  2000 because of concerns about poor project oversight and management  in the early stages of this project.
DOE officials also explained that a now-defunct policy may have  contributed to increased costs and delays for several projects we  examined. Until 2000, DOE required contractors to prepare cost and  schedule estimates early in the project, before preliminary designs were  completed. These estimates were used to establish a baseline for  measuring contractor performance and tracking any cost increases or  schedule delays. However, these estimates often were based on early  conceptual designs and, thus, were subject to significant change as more  detailed designs were developed. To improve the reliability of these  estimates, DOE issued a new order in October 2000 that required the  preparation of a cost estimate range at the start of preliminary design, and  delayed the requirement for a definitive cost and schedule baseline  estimate until after the preliminary design was completed. Consequently,  DOE officials explained, the new policy should result in improved  estimates and a more accurate measure of cost and schedule performance.
We also sent a survey to DOE project directors for all 12 projects asking  them to identify key events that led to the greatest cost increases or  schedule delays, and the major factors contributing to these key events.  However, no individual factors were identified as being major contributors  to the cost increases or schedule delays. In responding to our survey, DOE  project directors cited several factors that affected changes in cost and  schedule. However, when asked to rate the relative significance of these  factors for their impact on cost and schedule changes, the project  directors generally did not judge them to be significant contributors to the  changes. The most frequently cited factors were    an absence of open communication, mutual trust, and close coordination;    changes in “political will” during project execution (e.g., project changes  resulting from political decisions, both internal and external to the  project); interruptions in project funding; and   project managers’ lack of adequate professional experience.  (For detailed survey results covering these four factors, see app. IV.)
In contrast to the cost increases and schedule delays incurred on most of  the projects we reviewed, 3 projects had not yet experienced cost  increases or schedule delays—Microsystems and Engineering Sciences  Applications, the Linac Coherent Light Source, and the Chemistry and  Metallurgy Research Facility Replacement. DOE project officials identified  key conditions that they believed helped avoid those cost increases and  delays. These conditions included    active oversight—that is, the DOE project directors were never  “blindsided” by contractor issues;   a lack of technological complexity;    an effective system to measure contractor performance;    effective communication with and integration of all stakeholders; and    sustained leadership.
However, we observed that the Linac Coherent Light Source and the  Chemistry and Metallurgy Research Facility Replacement facilities are still  in a relatively early stage in the project development process, and thus it  may be too early to gauge the overall success of either project.  Additionally, because none of these 3 projects are highly technologically  complex, they may be less susceptible to the types of problems associated  with other projects we reviewed that experienced cost increases and  delays.
section_title: DOE Does Not Consistently Measure Technology Readiness to Ensure That Critical Technologies Will Work as Intended before Construction Begins
paragraphs: Although DOE requires its final designs to be sufficiently complete before  beginning construction, it has not systematically ensured that the critical  technologies reflected in project designs are technologically ready.  Recognizing that a lack of technology readiness can result in cost overruns  and schedule delays, other federal agencies, such as NASA and DOD, have  issued guidance for measuring and communicating technology readiness.
subsections: section_title: DOE Does Not Consistently Assess Technology Readiness
paragraphs: Only 1 of the 5 projects we reviewed to determine how DOE ensures that  project designs are sufficiently complete before construction—projects  that were approaching or had recently begun construction—had a  systematic assessment of technology readiness to determine whether the  project components would work individually or collectively as expected in  the intended design. Specifically, the DOE project director for the Pit  Disassembly and Conversion Facility systematically measured and  assessed readiness levels for each critical component of the overall  project. The assessment was based on a method developed by NASA, that  is, rating each technology from 0 to 10 in terms of relative maturity.  Because the project has not yet begun construction, we could not  determine whether the technology readiness assessment has helped  project managers to avoid cost increases or schedule delays during  construction. However, according to DOE and contractor officials  responsible for the project, the assessment helped focus management  attention during project design on critical technologies that may require  additional resources to ensure that they are sufficiently ready before  construction begins. In reviewing the assessment, however, we noted that  project officials had not updated the assessment tool for this project for  over 3 years. DOE’s project director acknowledged the delay in updating  the assessment and responded that he plans to begin updating the  assessment annually.
The other 4 projects did not have systematic assessments of technological  readiness. Therefore, the risk associated with the technology may not be  clearly and consistently understood across all levels of management.  Formally approving the project’s cost and schedule estimates as accurate  and complete, or proceeding into construction, without having clearly  assessed evidence of technology readiness can result in cost overruns and  schedule delays.
DOE’s experience with the Waste Treatment and Immobilization Plant is a  case in point. Specifically, technology known as “pulse jet mixers” was  used in the design of a subsystem intended to prepare radioactive material  for processing. However, this technology had not been used previously in  this application, and it did not work in tests as expected, even after  construction had already begun. Consequently, DOE incurred about   $225 million in redesign costs and over 1 year in schedule delays,  according to the DOE project director.
Over the past several years, we and others have stressed the importance of  assessing technology readiness to complete projects successfully, while  avoiding cost increases and schedule delays. Specifically, by 1999, we  reported that organizations using best practices recognize that delaying  the resolution of technology problems until production or construction  can result in at least a 10-fold cost increase. Furthermore, we reported  that delaying the resolution until after the start of production could  increase costs by 100-fold. Reporting on similar concerns, the National  Research Council has identified factors common to large construction  projects—in the areas of cost, schedule, and scope—that help to ensure  projects are completed successfully. Among key technical conditions for  defining project scope, the council stated, is a project plan that is based on  employing the best available, state-of-the-art technology, but not  experimental or unproven technology. As such, employing a consistent,  systematic method for measuring the extent to which technology is still  experimental or unproven is of critical importance.
An assessment of technology readiness is even more crucial at certain  points in the life of a project—particularly as DOE decides to accept a  project’s (1) preliminary design and formally approve the project’s cost  and schedule estimates as accurate and complete and (2) final design as  sufficiently complete so that resources can be committed toward  procurement and construction. Proceeding through these critical decision  points without a credible and complete technology readiness assessment  can lead to problems later in the project. Specifically, if DOE proceeds  with the project when technologies are not yet ready, there is less  certainty that the technologies specified in the preliminary or final designs  will work as intended. Project managers may then need to modify or  replace these technologies to make them work properly, which can result  in costly and time-consuming redesign work.
Moreover, modifying the design of a facility after construction has already  begun can be expensive and time consuming. First, changes to an already  designed work plan are not necessarily subject to competition because the  new work can occur through “change orders”—that is, modifications to  existing contracts. These change orders can be expensive, according to  DOE project directors. Second, worker productivity can be lost if, for  example, extra downtime results from delays to interrelated construction  work. Finally, tearing down and rebuilding items already constructed,  such as concrete floors, walls, and doors, might be necessary to  accommodate a design change.
DOE’s experience in the predecessor project to the Salt Waste Processing  Facility—the In-Tank Precipitation (ITP) project process—at the  Savannah River Site illustrates the potential consequences of proceeding  with technology that is not sufficiently ready. As we reported in 2000, the  ITP project was selected in 1983 as the preferred method for separating  highly radioactive material from 34 million gallons of liquid stored at the  Savannah River site—a step considered necessary to effectively handle  this large quantity of waste. A 1983 test using the ITP technology on a  tank containing 500,000 gallons of waste resulted in a significant buildup  of benzene—a highly explosive and hazardous compound. The buildup of  benzene was more than the tank instruments could register. Nevertheless,  project managers decided to proceed with the project. In 1985, DOE  estimated that it would take about 3 years and $32 million to construct the  ITP facility. After a number of delays, the ITP facility was constructed and  began start-up operations in 1995, which were halted because of safety  concerns about the amount of benzene that the facility generated. In 1998,  after about a decade of delays and costs of almost $500 million, DOE  suspended the project because it did not work as safely and efficiently as  designed. This suspension put an effective remedy for treating high-level  waste at the Savannah River Site years behind schedule. DOE then  directed its contractor to begin a process to identify and select an  alternative technology, which has developed into the current project  intended to treat this waste—the Salt Waste Processing Facility project.
In response to our concerns about the 4 projects without systematic  assessments of technology readiness, DOE project directors explained  that they have alternative methods for assessing readiness. They are  required to submit a project execution plan, which includes an assessment  of risks, including technological risks, and a plan for mitigating risks. They  also rely upon independent reviews, including extensive design reviews,  before making critical decisions to accept designs, and cost and schedule  estimates, or to proceed with construction. For example, DOE’s Office of  Engineering and Construction Management formally reviews major  projects in an effort to ensure that the designs are sufficiently complete to  begin construction. Specifically, an external independent readiness review  is performed, often using the services of various independent technical  experts, that, at a minimum, is intended to verify the readiness of the  project to proceed into construction or to identify remedial action. Finally,  several DOE project directors stated that they intentionally have avoided  using fast-track, design-build approaches because of the many problems it  posed for the Waste Treatment and Immobilization Plant project. The DOE  project directors of the 5 DOE projects that are nearing, or have recently  begun construction, told us they have completed, or expect to complete  prior to construction, 85 to 100 percent of their projects’ final design.
In addition to following the more standard approaches for managing  projects, such as preparing risk assessment plans, some DOE offices have  developed their own tools for assessing the readiness of projects. For  example, DOE’s Office of Environmental Management (EM) uses a  Product Definition Rating Index (PDRI) as a tool to assess how well a  project is planned, and whether it is ready to proceed to the next project  phase. Project elements rated include cost, schedule, scope/technical,  management planning and control, and external factors. Among the   77 project elements rated, 2 involve technology—the identification of  technology development requirements, and the testing and evaluation of  the technology to be used. While the project technologies are collectively  given a ranking with this tool, the PDRI does not represent a rigorous  examination of the demonstrated readiness of each critical technology for  its application in the project. Furthermore, not all EM projects we  examined were using this tool.
DOE’s design reviews, risk assessments, and other actions to monitor  design completion are extensive and certainly have merit. However, we  found that these actions alone do not provide consistent and transparent  assurance that all technologies are sufficiently ready because they do not  use a consistent and systematic method of measurement. DOE’s project  design reviews, for example, do not always clearly distinguish between  technology that has been demonstrated to work as expected in the  intended design versus a judgment that the technology has potential for  reaching a specific level of readiness.
The external review of the technologies for the Mixed Oxide Fuel  Fabrication Facility illustrates the shortfalls in DOE’s current approach to  assessing technology readiness and communicating the results of those  assessments. The report concluded, among other things, that the method  chosen by the contractor is the most rigorous and comprehensive, and  should result in the most successful technology transfer possible.  Furthermore, the review team was very impressed with the rigor with  which designs and design changes were being managed, finding ample  evidence verifying that the exact design process used by the French was  being transferred to the United States facility. Although the external  reviewers seemed to be impressed with many aspects of the design  transfer, and concluded that the technologies should not be problematic,  they had identified some key concerns about technology readiness in the  body of their final report. The reviewers did not explain how they  reconciled their conclusion with their concerns. To reconcile these  differences, we obtained several clarifying statements from DOE’s project  director, technical experts, and one of the study’s authors. These clarifying  statements appear to support the reviewers’ conclusions. However,  without these statements, the level of technological readiness was not  readily evident because the independent review lacked consistent,  systematic criteria and a method for measuring the degree of readiness or  clearly communicating assessment results, and the review was not  transparent.
DOE does not consistently assess technology readiness of project  technologies because its project management guidance lacks  comprehensive standards for systematically measuring and  communicating the readiness of project technologies. Specifically, DOE  lacks consistent metrics for determining technology readiness  departmentwide, terminology to facilitate effective communication, and  oversight protocols for reporting and reviewing technology readiness  levels. DOE project management guidance is contained in two key  documents—DOE Order 413.3A and Manual 413.3-1. Although the manual  requires final designs to be sufficiently complete before beginning  construction, it does not specify how technologies reflected in project  designs are to be assessed for readiness—to determine that they have  been sufficiently demonstrated to work as intended. Consequently, critical  decisions made without standard measures are susceptible to varying  interpretations of the actual technology readiness attained and the level  needed for a project to proceed, which can easily vary among projects and  among officials within a single project.
section_title: Other Federal Agencies Use a Standard Method for Measuring and Communicating Technology Readiness
paragraphs: Other federal agencies have recognized the importance of ensuring that  technologies have been sufficiently demonstrated for their intended  purpose and have issued standard guidance for measuring and  communicating TRLs. In particular, recognizing the need to measure the  readiness level of project technologies, NASA began using a systematic  method of measurement in the mid-1990s. NASA incorporated a structured  TRL approach into guidance on integrated technology planning.
Similarly, to improve DOD management of risk and technology  development, the Deputy Under Secretary of Defense (Science and  Technology) officially endorsed, in a July 2001 memorandum, the use of  TRLs in new major programs. In 2002, DOD issued mandatory procedures  for major defense acquisition programs and major automated information  system acquisition programs, which identified technology readiness as a  principal element of program risk. The procedures require the military  services’ science and technology officials to conduct a systematic  assessment of critical technologies that are identified in major weapon  systems programs before starting engineering and manufacturing  development and production. Using TRLs is the preferred method, and  approval must be obtained from the Deputy Under Secretary if an  equivalent alternative method is used, according to the Deputy Under  Secretary’s memorandum. Importantly, the procedures stated that TRLs  are a measure of demonstrated technical maturity—they do not discuss  the probability of occurrence (i.e., the likelihood of attaining required  maturity) or the impact of not achieving technology maturity.
Both NASA and DOD use a nine-point scale to measure technology  readiness, from a low of TRL 1 (basic principles observed) to a high of  TRL 9 (total system used successfully in project operations). (App. V  contains the definitions of these nine TRLs.) For example, a subsystem  prototype that has been successfully demonstrated in an operational  environment would receive a higher TRL value (i.e., TRL 7) than a  technological component that has been demonstrated in a laboratory test  (i.e., TRL 4). In our previous work, we recommended to the Secretary of  Defense that key project technologies used in weapons systems be  demonstrated in an operational environment, reaching a high maturity  level—analogous to TRL 7—before deciding to commit to a cost, schedule,  and performance baseline for development and production of the weapon  system. In response to our recommendation, DOD has agreed that if a  technology does not achieve a score of TRL 6 or 7, project managers must  develop a plan to bring the technology to the required readiness level  before proceeding to the next project phase.
Use of TRLs is not by itself a cure-all for managing critical technologies,  but TRLs can be used in conjunction with other measures to improve the  way projects are managed. For example, according to studies by NASA,  DOD, and others, TRLs can    provide a common language among the technology developers, engineers  who will adopt/use the technology, and other stakeholders; improve stakeholder communication regarding technology development— a by-product of the discussion among stakeholders that is needed to  negotiate a TRL value; reveal the gap between a technology’s current readiness level and the  readiness level needed for successful inclusion in the intended product; identify at-risk technologies that need increased management attention or  additional resources for technology development to initiate risk-reduction  measures; and increase transparency of critical decisions by identifying key technologies  that have been demonstrated to work or by highlighting still immature or  unproven technologies that might result in high project risk.
Two DOE headquarters offices have attempted to systematically assess  technology readiness. First, under the Office of Nuclear Energy, a DOE  contractor preparing a congressional report used a TRL method to  compare the maturity of advanced fuel cycle technologies. In addition, in  2000, DOE’s Office of Science and Technology, under EM, issued a report  that defined a process for assessing technology maturity of EM projects.  However, according to an EM official, the office decided to discontinue  using this assessment process because it was considered overly  burdensome. As a result, DOE devolved responsibility for managing  technology readiness to the contractor level.
According to several DOE project directors we spoke with, a consistent,  systematic method for assessing technology readiness would help achieve  a number of objectives: that is, standardize terminology, make technology  assessments more transparent, and improve communication among  project stakeholders before they make critical project decisions. DOE  project managers also acknowledged that TRLs could improve project  management departmentwide, and some managers are now attempting to  use this tool to assess technology maturity. The DOE project director for  the Waste Treatment and Immobilization Plant told us that a senior DOE  official encouraged him to begin using TRLs. He is consulting with DOD  officials knowledgeable about using the TRL method and expects to  develop a TRL tool and have TRL determinations for major parts of the  project in 2007. (App. VI compares DOD’s product development process  with DOE’s project management process for major projects.)
section_title: Conclusions
paragraphs: The magnitude of the cost increases and schedule delays for DOE’s major  projects is cause for serious rethinking of how DOE manages them. To its  credit, DOE has completed, or expects to complete prior to construction,  85 to 100 percent of project design work for the 5 projects we reviewed  that have recently begun or are nearing construction. However, DOE has  not systematically addressed another key factor—the readiness level of  the technologies it expects to use in these projects. DOE lacks  comprehensive standards in DOE Order 413.3A and Manual 413.3-1 for  systematically measuring and communicating the readiness of project  technologies. Specifically, the department lacks consistent metrics for  determining technology readiness departmentwide, terminology, and  oversight protocols for reporting and reviewing TRLs. Without consistent  measurement and communication of the readiness of technologies, DOE  does not have a basis for defining the acceptable level of technological  risk for each project, making critical decisions on accepting the validity of  a project’s total estimated cost and schedule, or proceeding with  construction.
Other federal agencies have recognized the need to consistently measure  and communicate technology readiness to help avoid cost increases and  delays that result from relying on immature technologies. DOD, for  example, requires its managers to use a TRL process to measure  technology readiness and generally requires a TRL 7 (as we had  recommended) before system development and demonstration. In  contrast, as DOE’s poor track record for managing the technological  complexity of major projects shows, DOE has not systematically measured  the readiness of critical project technologies before it approves definitive  cost and schedule estimates or begins construction. Furthermore, without  a systematic method for measuring technological readiness, DOE cannot  effectively communicate within the department and to the Congress  whether projects are at risk of experiencing cost increases and schedule  delays associated with technology problems.
section_title: Recommendations for Executive Action
paragraphs: To improve decision making and oversight for major DOE construction  projects, including how project technology readiness is measured and  reported, we recommend that the Secretary of Energy evaluate and  consider adopting a disciplined and consistent approach to assessing TRLs  for projects with critical technologies that includes the following three  actions:    Develop comprehensive standards for systematically measuring and  communicating the readiness of project technologies. At a minimum, these  standards should (1) specify consistent metrics for determining  technology readiness departmentwide, (2) establish terminology that can  be consistently applied across projects, and (3) detail the oversight  protocols to be used in reporting and reviewing TRLs. In preparing these  standards, DOE should consider lessons learned from NASA and DOD,  and its own experience in measuring technology readiness. If DOE’s  evaluation results in the decision to adopt these standards, it should  incorporate them into DOE Order 413.3A and Manual 413.3-1, and provide  the appropriate training to ensure their proper implementation.
Direct DOE Acquisition Executives to ensure that projects with critical  technologies reach a level of readiness commensurate with acceptable  risk—analogous to TRL 7—before deciding to approve the preliminary  design and commit to definitive cost and schedule estimates, and at least  TRL 7 or, if possible, TRL 8 before committing to construction expenses.
Inform the appropriate committees and Members of Congress of any DOE  decision to approve definitive cost and schedule estimates, or to begin  construction, without first having ensured that project technologies are  sufficiently ready (at TRL 7 or 8). This information should include specific  plans for mitigating technology risks, such as developing backup  technologies to offset the effects of a potential technology failure, and  appropriate justification for accepting higher technological risk.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to DOE for its review and comment.  DOE’s written comments are reproduced in appendix VII. DOE agreed  with our recommendations but suggested revisions that would allow it to  first conduct a pilot application on selected projects to better understand  the technology readiness assessment process and evaluate its potential  use. We revised our recommendations to give DOE this flexibility. DOE  also provided detailed technical comments, which we have incorporated  into our report as appropriate.
DOE also expressed several specific concerns with our draft report. First,  DOE stated that while our draft broadly asserts that DOE project  management has led to increases in cost and schedule, our  recommendations are narrowly focused on technology assessment. We  agree that our draft states that DOE project management has led to cost  increases and schedule delays, a conclusion we reached on the basis of  our contact with DOE project directors and our review of numerous  studies and reports on DOE major projects. Our recommendations address  technology assessment, a critical project management activity, because  they were developed primarily on the basis of our specific finding that  DOE lacks a systematic approach to ensure that final project designs,  including critical technologies reflected in these designs, have been  demonstrated to work as intended prior to construction. This report  explains that delaying resolution of technology problems until  construction can potentially lead to significant cost increases and  schedule delays.
Second, DOE stated that our draft report inappropriately characterizes  cost and schedule growth from a small sample of projects by using  preliminary cost and schedule estimates that are intended for internal  DOE planning. To clarify, the scope of our review included an evaluation  of DOE’s major construction projects. In addition, our report explains that  DOE changed its project management policy in 2000 to allow cost and  schedule estimates to be prepared later in the project—at the end of  preliminary design. Prior to this new policy, project directors submitted  cost and schedule estimates earlier in the project development phase—at  the end of conceptual design. For projects under way prior to the policy in  2000, we used post-2000 validated baseline estimates, if available.  Otherwise, we used earlier estimates since these were the only estimates  available and had been previously used by DOE to inform Congress of the  total expected project cost and schedule while seeking initial project  funding. We also note that for the five projects that were started after the  new policy in 2000, we used the validated project baseline estimates  recommended by DOE, if available.
Third, DOE suggested we revise table 3 in our report to more clearly  identify the correlation between cost and schedule growth and technology  maturity. As our report states, the information in table 3 was drawn from  the results of our review of independent studies involving the projects we  reviewed and the results of our interviews with DOE project directors. Our  report explains that cost increases and schedule delays for 6 of the 9  projects shown in the table were due in part to contractors’ poor  management of the development and integration of technologies used in  the project designs.
Finally, DOE stated that it is unclear how the factors cited in appendix IV,  such as communication, and changes in “political will,” among other  things, led to our recommendation to assess technology readiness.  Although not all of the factors cited in our survey have a link to our  recommendation on technology readiness, one factor in particular— absence of communication—is addressed in our recommendation.  Specifically, we recommended that the Secretary of Energy consider  developing comprehensive standards for systematically measuring and  communicating the readiness of project technologies, including the  establishment of terminology that is to be consistently applied across  projects.
We are sending copies of the report to interested congressional  committees, the Secretary of Energy, and the Director of the Office of  Management and Budget. We will make copies available to others on  request. In addition, the report will also be available at no charge on the  GAO Web site at http://www.gao.gov.
If you or your staffs have any questions about this report, please contact  me at (202) 512-3841 or aloisee@gao.gov. Contact points for our Offices of  Congressional Relations and Public Affairs may be found on the last page  of this report. Other staff contributing to the report are listed in appendix  VIII.
section_title: Appendix I: Scope and Methodology
paragraphs: To determine the extent to which the Department of Energy’s (DOE)  major construction projects have experienced cost increases and schedule  delays and the factors that have contributed to these problems, we  identified (1) active DOE major line-item construction projects that have  current total project cost estimates above the $750 million threshold— DOE’s criteria for “major construction projects,” and (2) the projects with  estimates above $400 million—the DOE threshold for major projects until  July 2006. We also identified those projects above $300 million to account  for any projects that may pass the $400 million threshold. In all, we  identified the following 12 projects:    Five of these 12 projects began before DOE moved its requirement for firm  cost and schedule estimates to later in the project: the National Ignition  Facility, the Mixed Oxide Fuel Fabrication Facility, the Pit Disassembly  and Conversion Facility, the Spallation Neutron Source, and the Tritium  Extraction Facility. We used the estimates at the end of conceptual design,  as reported by project directors, for the initial project cost and schedule  estimates.
Four of the remaining 7 projects had cost and schedule estimates  completed at the end of preliminary design, according to the new DOE  guidelines: the Highly Enriched Uranium Materials Facility, Microsystems  and Engineering Sciences Applications, the Depleted Uranium  Hexafluoride 6 Conversion Facilities, and the Linac Coherent Light Source.  For these projects, we considered the estimates as reported by project  directors to be the initial cost and schedule estimates.
One project, the Waste Treatment and Immobilization Plant, began after  DOE moved the requirement for firm cost and schedule estimates to later  in the project. However, DOE initially exempted the contractor from  submitting firm cost and schedule estimates. Therefore, we used the  estimates reported by the project director to be the initial cost and  schedule estimates.
The final 2 projects, although falling under the new DOE requirements,  had yet to complete their preliminary design at the time of our review: the  Chemistry and Metallurgy Research Facility Replacement and the Salt  Waste Processing Facility. For these projects, we considered the cost and  schedule estimates at the end of conceptual design reported by project  directors to be the initial project cost and schedule estimates.
Because we and others have previously expressed concern about the data  reliability of a key DOE project management tracking database—the  Project Assessment and Reporting System—we did not develop  conclusions or findings based on information generated through that  system. Instead, we collected information directly through surveys and  interviews with project site officials.
To identify cost increases and schedule delays, and the factors that may  have contributed to these changes, we surveyed DOE project directors,  interviewed DOE and contractor project personnel, and reviewed project  management documents for 12 major projects. These 12 projects are  managed by DOE’s Office of Science, Office of Environmental  Management (EM), or National Nuclear Security Administration (NNSA).  (See app. II for information on these projects.)
Our survey asked DOE project directors of the 12 projects to identify the  degree to which cost and schedule estimates may have changed and the  reasons for these changes, and to describe the events and conditions that  led to any changes. Eight of the 12 project directors responded that their  projects had experienced cost increases and schedule delays, and 1  project director reported only a schedule delay. For these 9 projects, we  asked project directors to (1) identify the top three events that led to the  cost and schedule delays and (2) indicate to what extent certain factors  may have contributed to the event that led to the largest percentage cost  increase or schedule delay. The factors included in the survey instrument  were based on the results of a National Research Council study that listed  essential or important conditions needed for the successful completion of  major projects. We asked project directors to identify the extent to which  the lack of these conditions may have contributed to any cost and  schedule delays. (App. IV shows key survey results for these 9 projects.)
In addition to reviewing project documentation, we conducted site visits  for the 9 projects that had experienced cost and schedule changes, and we  analyzed (1) studies of these projects completed by DOE’s Office of  Inspector General and (2) external independent project reviews conducted  under the direction of DOE’s Office of Engineering and Construction  Management in Washington, D.C. We interviewed federal project directors  of the 3 projects that had not experienced cost increases or schedule  delays to obtain information on factors they believe are important in  avoiding such increases.
To determine the extent to which DOE ensures that project designs are  sufficiently complete before construction, we obtained additional  information from project directors on 5 projects that were approaching, or  had recently begun, construction. During our review, we obtained  information on the extent project designs were, or are expected to be,  complete before beginning construction, and the actions DOE had taken to  ensure technologies used in these designs are sufficiently ready to begin  construction. For 2 of these 5 projects, we applied a tool we previously  had used to assess DOD programs—the tool enables project directors to  characterize the readiness level of each technology being developed for  use in aircraft and other military applications. In addition, we spoke with  officials from DOE program offices and DOE’s Office of Engineering and  Construction Management in Washington, D.C.
We provided interim briefings to the Subcommittee on Energy and Water  Development, House Committee on Appropriations, on the status of our  work in May and September, 2006. We performed our work between  December 2005 and January 2007, in accordance with generally accepted  government auditing standards.
section_title: Appendix II: Information on the 12 Department of Energy Major Projects Reviewed
paragraphs: Relocate and consolidate mission-critical analytical chemistry, material  characterization, and research and development capabilities to ensure  continuous national security mission support beyond 2010.
Design and construct facilities at Portsmouth, Ohio, and Paducah,  Kentucky, to convert the Department of Energy’s existing inventory of  depleted uranium hexafluoride into a more stable form for disposal or  beneficial reuse.
Project will construct a highly secure, state-of-the-art facility for  consolidating and storing highly enriched uranium, resulting in cost  savings and an increased security posture.
Provide laser-like radiation in the X-ray region of the spectrum that is 10  billion times greater in peak brightness than any existing X-ray light  source. The project will apply these high-brightness X-rays to  experiments in the chemical, material, and biological sciences.
Provide state-of-the-art national complex that will provide for the design,  integration, prototyping, and qualification of microsystems into  components, subsystems, and systems within the nuclear weapons  stockpile.
Facility will combine surplus weapon-grade plutonium oxide with  depleted uranium to form mixed oxide fuel assemblies that will be  irradiated in United States commercial nuclear reactors. Once irradiated  and converted into spent fuel, the resulting plutonium can no longer be  readily used for nuclear weapons.
Provide experimental capability to assess nuclear weapons physics,  providing critical data that will allow the United States to maintain its  technical capabilities in nuclear weapons in the absence of underground  testing, and to advance fusion as an energy source.
Eliminate surplus Russian and United States plutonium and highly  enriched uranium by disassembling surplus nuclear weapons pits and  converting the resulting plutonium metal to a powder form that can later  be fabricated into mixed oxide fuel to produce nuclear fuel assemblies for  use in commercial nuclear reactors.
Meet site cleanup goals and reduce significant environmental and  health/safety risk by construction of a facility to treat large quantities of  waste from reprocessing and nuclear materials production operations at  the Savannah River Site. Process will separate waste, solidify it in glass,  and send it to federal repositories for disposal.
Provide next generation, short-pulse spallation neutron source for  neutron scattering, to be used by researchers from academia, national  and federal labs, and industry for basic and applied research and  technology development in the fields of condensed matter physics,  materials sciences, magnetic materials, polymers and complex fluids,  chemistry, biology, earth sciences, and engineering.
To replenish the tritium needs of the nuclear weapons stockpile, the  facility will extract tritium produced in a commercial nuclear reactor for  use in nuclear weapons development.
The plant will separate high-level from low-level radioactive waste  currently stored in underground tanks, processing and solidifying all high- level waste and a substantial portion of the low-level waste, and will treat  the remaining low-level waste.
section_title: Appendix III: Independent Studies Reviewed
paragraphs: Department of Energy, Office of Inspector General. Audit Report: Status  of the National Ignition Facility Project. DOE/IG-0598. Washington, D.C.:  April 28, 2003.
GAO. Department of Energy: Status of Contract and Project Management  Reforms. GAO-03-570T. Washington, D.C.: March 20, 2003.
GAO. Contract Reform: DOE Has Made Progress, but Actions Needed to  Ensure Initiatives Have Improved Results. GAO-02-798. Washington,  D.C.: September 13, 2002.
GAO. Department of Energy: Follow-up Review of DOE’s National  Ignition Facility. GAO-01-677R. Washington, D.C.: June 1, 2001.
GAO. National Ignition Facility: Management and Oversight Failures  Caused Major Cost Overruns and Schedule Delays. GAO/RCED-00-141  and GAO/RCED-00-271. Washington, D.C.: August 8, 2000.
The Mitre Corporation. NIF Ignition. JSR-05-340. McLean, VA: June 29,  2005.
subsections: section_title: Mixed Oxide Fuel Fabrication Facility
paragraphs: Burns and Roe Enterprises, Inc. External Independent Review of the  Mixed Oxide Fuel Fabrication Facility (MFFF) Project Critical Decision  (CD) 2/3 Baseline: Performance Baseline (CD-2) and Start of  Construction (CD-3) Review. BREI-L-R-06-03. Oradell, NJ: July 7, 2006.
Burns and Roe Enterprises, Inc. External Independent Review of the  Basis of Design for the Aqueuous Polishing Process. BREI-SLP-R-06-01.  Oradell, NJ: March 27, 2006.
Civil Engineering Research Foundation. Independent Research  Assessment of Project Management Factors Affecting Department of  Energy Project Success. Washington, D.C.: July 12, 2004.
Department of Energy, Office of Inspector General. Audit Report: Status  of the Mixed Oxide Fuel Fabrication Facility. DOE/IG-0713. Washington,  D.C.: December 21, 2005.
section_title: Pit Disassembly and Conversion Facility
paragraphs: Department of Energy, Office of Inspector General. Audit Report:  National Nuclear Security Administration’s Pit Disassembly and  Conversion Facility. DOE/IG-0688. Washington, D.C.: May 3, 2005.
Los Alamos National Laboratory. Options for the Development and  Testing of the Pit Disassembly and Conversion Facility Government- Furnished Design. LA-UR-03-3926. Los Alamos, NM: June 11, 2003.
section_title: Waste Treatment and Immobilization Plant
paragraphs: Bechtel National, Inc. Hanford Tank Waste Treatment and  Immobilization Plant, May 2006 Estimate at Completion. Hanford Site,  WA: May 31, 2006.
Bechtel National, Inc. Comprehensive Review of the Hanford Tank Waste  Treatment and Immobilization Plant Estimate at Completion. CCN  132848. Hanford Site, WA: March 31, 2006.
Bechtel National, Inc. Comprehensive Review of the Hanford Waste  Treatment Plant Flowsheet and Throughput. CCN132846. Hanford Site,  WA: March 17, 2006.
Bechtel National, Inc. Hanford Tank Waste Treatment and  Immobilization Plant, December 2005 Estimate at Completion  Executive Summary. Hanford Site, WA: January 30, 2006.
Department of the Army Corp of Engineers. Complete Statement of Kim  Callan, to the Subcommittee on Energy and Water Development,  Committee on Appropriations, United States House of Representatives.  Washington, D.C.: April 6, 2006.
Department of Energy. External Independent Review, Independent Cost  Review, CD-3C Review of the Waste Treatment and Immobilization  Plant Project. Hanford Site, WA: September 2002.
Department of Energy. External Independent Review CD-3B Review of  the Waste Treatment and Immobilization Plant Project. Hanford Site,  WA: April 2002.
GAO. Hanford Waste Treatment Plant, Contractor and DOE Management  Problems Have Led to Higher Costs, Construction Delays, and Safety  Concerns. GAO-06-602T. Washington, D.C.: April 6, 2006.
GAO. Further Actions Are Needed to Strengthen Contract Management  for Major Projects. GAO-05-123. Washington, D.C.: March 18, 2005.
GAO. Nuclear Waste: Absence of Key Management Reforms on Hanford’s  Cleanup Project Adds to Challenges of Achieving Cost and Schedule  Goals. GAO-04-611. Washington, D.C.: June 9, 2004.
GAO. Status of Contract and Project Management Reforms. GAO-03-57T.  Washington, D.C.: March 20, 2003.
GAO. Contract Reform: DOE Has Made Progress, but Actions Needed to  Ensure Initiatives Have Improved Results. GAO-02-798. Washington,  D.C.: September 13, 2002.
GAO. Nuclear Waste: Hanford Tank Waste Program Needs Cost,  Schedule, and Management Changes. GAO/RCED-93-99. Washington,  D.C.: March 8, 1993.
LMI Government Consulting. Hanford Waste Treatment and  Immobilization Plant After-Action Fact-Finding Review. DE535T1.  McLean, VA: January 2006.
LMI Government Consulting. External Independent Review, Follow-up  Review, Waste Treatment and Immobilization Plant (WTP) Out  Briefing. Washington, D.C.: March 14, 2003.
section_title: Spallation Neutron Source
paragraphs: Civil Engineering Research Foundation. Independent Research  Assessment of Project Management Factors Affecting Department of  Energy Project Success. Washington, D.C.: July 12, 2004.
Department of Energy, Office of Inspector General. Audit Report:  Progress of the Spallation Neutron Source Project. DOE/IG-0532.  Washington, D.C.: November 19, 2001.
Department of Energy. Review Committee Report on the Baseline Review  of the Spallation Neutron Source (SNS) Project. Washington, D.C.:   July 15, 1999.
Department of Energy. Technical, Cost, Schedule, and Management  Review of the Spallation Neutron Source Project. Washington, D.C.:  January 28, 1999.
GAO. Department of Energy: Status of Contract and Project Management  Reforms. GAO-03-570T. Washington, D.C.: March 20, 2003.
GAO. Contract Reform: DOE Has Made Progress, but Actions Needed to  Ensure Initiatives Have Improved Results. GAO-02-798. Washington,  D.C.: September 13, 2002.
GAO. Department of Energy: Challenges Exist in Managing the  Spallation Neutron Source Project. GAO/T-RCED-99-103. Washington,  D.C.: March 3, 1999.
section_title: Salt Waste Processing Facility
paragraphs: Department of Energy, Office of Inspector General. Audit Report: Salt  Processing Project at the Savannah River Site. DOE/IG-0565. Washington,  D.C.: August 27, 2002.
Institute for Regulatory Science. Technical Peer Review Report of the  Review Panel on Salt Waste Processing Facility Technology Readiness.  CRTD-Vol. 75. Danvers, MA: October 31, 2003.
section_title: Tritium Extraction Facility
paragraphs: Civil Engineering Research Foundation. Independent Research  Assessment of Project Management Factors Affecting Department of  Energy Project Success. Washington, D.C.: July 12, 2004.
Department of Energy, Office of Inspector General. Audit Report: The  Department of Energy’s Tritium Extraction Facility. DOE/IG-0560.  Washington, D.C.: June 24, 2002.
GAO. Department of Energy: Further Actions Are Needed to Strengthen  Contract Management for Major Projects. GAO-05-123. Washington, D.C.:  March 18, 2005.
GAO. Department of Energy: Status of Contract and Project Management  Reforms. GAO-03-570T. Washington, D.C.: March 20, 2003.
GAO. Contract Reform: DOE Has Made Progress, but Actions Needed to  Ensure Initiatives Have Improved Results. GAO-02-798. Washington,  D.C.: September 13, 2002.
GAO. Nuclear Weapons: Design Reviews of DOE’s Tritium Extraction  Facility. GAO/RCED-98-75. Washington, D.C.: March 31, 1998.
National Nuclear Security Administration. Program Review of the  Estimate to Complete Tritium Extraction Facility (TEF) at Savannah  River Site. Washington, D.C.: August 29, 2002.
section_title: Highly Enriched Uranium Materials Facility
paragraphs: BWXT Y-12. Highly Enriched Uranium Materials Facility Project Causal  Analysis Report. Oak Ridge, TN: March 6, 2006.
Department of Energy. Limited External Independent Review for  Baseline Change Proposal Review. Oak Ridge, TN: August 31, 2004.
Department of Energy, Office of Inspector General. Audit Report, Design  of the Uranium Storage Facility at the Y-12 National Security Complex.  DOE/IG-0643. Washington, D.C.: March 19, 2004.
Department of Energy, Office of Inspector General. Audit Report,  Reestablishment of Enriched Uranium Operations at the Y-12 National  Security Complex. DOE/IG-0640. Washington, D.C.: February 24, 2004.
Department of Energy. External Independent Review – Performance  Baseline Review of the Highly Enriched Uranium Materials Facility  Project. Oak Ridge, TN: June 2003.
section_title: Depleted Uranium Hexafluoride 6 Conversion Facility
paragraphs: Department of Energy. Report on the Independent Project Review of the  Depleted Uranium Hexafluoride Conversion Project. Washington, D.C.:  October 8, 2004.
Department of Energy, Office of Inspector General. Audit Report:  Depleted Uranium Hexafluoride Conversion. DOE/IG-0642. Washington,  D.C.: March 18, 2004.
GAO. Department of Energy: Status of Contract and Project Management  Reforms. GAO-03-570T. Washington, D.C.: March 20, 2003.
LMI Government Consulting. DUF6 Conversion Project CD-3 Corrective  Action Plan Review. DE538T1. McLean, VA: October 2005.
LMI Government Consulting. Construction Readiness EIR (for CD-3) of  the Depleted Uranium Hexafluoride Conversion Project. DE534T1.  McLean, VA: June 2005.
LMI Government Consulting. DUF6 Conversion Project CD-3C  Construction Readiness Review Preliminary Draft. Washington, D.C.:  May 20, 2005.
LMI Government Consulting. DUF6 Limited Conversion Plan Project  External Independent Review for the Office of Engineering and  Construction Management. DE428T1. McLean, VA: June 2004.
section_title: Chemistry and Metallurgy Research Facility Replacement
paragraphs: Jupiter Corporation. External Independent Review of the Chemistry and  Metallurgy Research Building Replacement Project. Approve  Performance Baseline and Approve Start of Construction. CD-2A/CD-3A.  Wheaton, MD: October 14, 2005.
section_title: Appendix IV: Survey Results for Primary Factors Affecting Cost and Schedule on Nine Projects with Cost or Schedule Changes
paragraphs: Absence of open communication, mutual trust, and close coordination  Depleted Uranium Hexafluoride 6 Conversion Facility  Highly Enriched Uranium Materials Facility  Pit Disassembly and Conversion Facility  Waste Treatment and Immobilization Plant  Changes in “political will” during project execution (e.g., project changes  resulting from political decisions—includes politics internal and external to the project )
section_title: Appendix V: Definitions of Technology Readiness Levels
paragraphs: Appendix V: Definitions of Technology  Readiness Levels None.
None.
Research to prove  feasibility.
Desktop, “back of  envelope” environment.
Research to prove  feasibility.
None.
Paper studies indicate  components ought to  work together.
Academic environment.  The emphasis here is  still on understanding  the science but  beginning to think  about possible  applications of the  scientific principles.
Research to prove  feasibility.
No system  components, just  basic laboratory  research equipment  to verify physical  principles.
No attempt at  integration; still trying  to see whether  individual parts of the  technology work. Lab  experiments with  available components  show they will work.
Uses of the observed  properties are  postulated and  experimentation with  potential elements of  subsystem begins. Lab  work to validate pieces  of technology without  trying to integrate.  Emphasis is on  validating the  predictions made  during earlier analytical  studies to ensure that  the technology has a  firm scientific  underpinning.
Demonstrate  technical feasibility  and functionality.
Ad Hoc and available  laboratory  components are  surrogates for system  components that may  require special  handling, calibration,  or alignment to get  them to function. Not  fully functional but  representative of  technically feasible  approach.
Available components  assembled into  subsystem  breadboard.  Interfaces between  components are  realistic.
Tests in controlled  laboratory environment.  Lab work at less than  full subsystem  integration, although  starting to see if  components will work  together.
Demonstrate  technical feasibility  and functionality.
Fidelity of  components and  interfaces are  improved from TRL 4.  Some special  purpose components  combined with  available laboratory  components.  Functionally  equivalent but not of  same material or size.  May include  integration of several  components with  reasonably realistic  support elements to  demonstrate  functionality.
Fidelity of subsystem  mock up improves  (e.g., from  breadboard to  brassboard).  Integration issues  become defined.
Laboratory environment  modified to  approximate  operational  environment. Increases  in accuracy of the  controlled environment  in which it is tested.
Demonstrate  applicability to  intended project and  subsystem  integration.  (Specific to intended  application in  project.)
Subsystem is high  fidelity functional  prototype with (very  near same material  and size of  operational system).  Probably includes the  integration of many  new components and  realistic supporting  elements/subsystems  if needed to  demonstrate full  functionality. Partially  integrated with  existing systems.
Components are  functionally  compatible (and very  near same material  and size of  operational system).  Component  integration into  system is  demonstrated.
Relevant environment  inside or outside the  laboratory, but not the  eventual operating  environment. The  testing environment  does not reach the  level of an operational  environment, although  moving out of  controlled laboratory  environment into  something more closely  approximating the  realities of technology’s  intended use.
Demonstrate  applicability to  intended project and  subsystem  integration.  (Specific to intended  application in  project.)
Prototype improves to  preproduction quality.  Components are  representative of  project components  (material, size, and  function) and  integrated with other  key supporting  elements/subsystems  to demonstrate full  functionality. Accurate  enough  representation to  expect only minor  design changes.
Prototype not  integrated into  intended system but  onto surrogate  system.
Operational  environment, but not  the eventual  environment.  Operational testing of  system in  representational  environment. Prototype  will be exposed to the  true operational  environment on a  surrogate platform,  demonstrator, or test  bed.
Applied/Integrated  into intended project  application.
Components are right  material, size, and  function compatible  with operational  system.
Subsystem  performance meets  intended application  and is fully integrated  into total system.
Demonstration, test,  and evaluation  completed.  Demonstrates system  meets procurement  specifications.  Demonstrated in  eventual environment.
Applied/Integrated  into intended project  application.
Components are  successfully  performing in the  actual environment— proper size, material,  and function.
Subsystem has been  installed and  successfully deployed  in project systems.
Operational testing and  evaluation completed.  Demonstrates that  system is capable of  meeting all mission  requirements.
section_title: Appendix VI: Comparison of DOD’s Product Development Process with DOE’s Project Management Process
subsections: section_title: DOD’s Product Development Process
paragraphs: application) application)
section_title: Appendix VII: Comments from the Department of Energy
section_title: Appendix VIII: GAO Contact and Staff Acknowledgments
subsections: section_title: Staff Acknowledgments
paragraphs: In addition to the individual named above, Michaela Brown, Rudy Chatlos,  James Espinoza, Daniel Feehan (Assistant Director), Joseph Keener,  Thomas Kingham, Matthew Lea, Mehrzad Nadji, Omari Norman,  Christopher Pacheco, Thomas Perry, and Carol Herrnstadt Shulman made                     key contributions to this report.