Department of Energy: Major Construction Projects Need a Consistent Approach for Assessing Technology Readiness to Help Avoid Cost Increases and Delays
Released: 2007-03-27T00:00:00

Letter

Background

To meet its diverse missions, DOE pays its contractors billions of dollars each year to implement hundreds of projects, ranging from hazardous waste cleanups at sites in the weapons complex to the construction of scientific facilities. Many of these complex, unique projects are designed to meet defense, energy research, environmental, and fissile materials disposition goals. They often rely on technologies that are unproven in operational conditions. In recent years, DOE’s budget has been dominated by the monumental task of environmental restoration and waste management to repair damage caused by the past production of nuclear weapons.

DOE has long had a poor track record for developing designs and cost estimates and managing projects. We reported in 1997 that from 1980 to 1996, 31 of DOE’s 80 major projects were terminated prior to completion, after expenditures of over $10 billion; 15 of the projects were completed, but most of them were finished behind schedule and with cost overruns; and the remaining 34 ongoing projects also were experiencing schedule slippage or cost overruns. In addition, for over a decade, DOE’s Office of Inspector General, the National Academy of Sciences, and others have identified problems with DOE’s management of major construction projects. Projects were late or never finished; project costs increased by millions and sometimes billions of dollars; and environmental conditions at the sites did not significantly improve. According to the National Research Council, DOE’s construction and environmental remediation projects take much longer and cost about 50 percent more than comparable projects by other federal agencies or projects in the private sector. A 2004 assessment of departmental project management completed by the Civil Engineering Research Foundation recommended, among other things, that DOE develop a core group of highly qualified project directors and require peer reviews for first-of-a-kind and technically complex projects when the projects’ preliminary baselines are approved.

To address project management issues, DOE began a series of reforms in the 1990s that included efforts to strengthen project management practices. To guide these reforms, the department formed the Office of Engineering and Construction Management in 1999. The reforms instituted to date have included planning, organizing, and tracking project activities, costs, and schedules; training to ensure that federal project managers had the required expertise to manage projects; increasing emphasis on independent reviews; and strengthening project reporting and oversight.

Most Major Projects Have Exceeded Original Costs and Are Years Late, Principally Because of Ineffective DOE Project Oversight and Contractor Management

The estimated costs of many of the DOE major construction projects we reviewed have significantly exceeded original estimates and schedules have slipped. On the basis of our analysis of independent project studies and interviews with project directors, cost growth and schedule slippage occurred principally because of ineffective DOE project oversight and poor contractor project management. Furthermore, unreliable initial cost and schedule estimates resulting from a now-defunct policy may have been a contributing factor, according to DOE project officials. Although external factors, such as additional security and safety requirements, contributed to cost growth and delays, the management of these requirements was complicated by ineffective and untimely DOE communication and decision making.

Eight of the 12 DOE projects we reviewed had increases in estimates of total project cost (TPC) ranging from $79.0 million to $7.9 billion. As table 1 shows, the percentage of cost increase for these 8 projects ranged from 2 percent to over 200 percent.

In addition, as shown in table 2, 9 of the 12 projects experienced schedule delays ranging from 9 months to more than 11 years. Of the 9 projects, 7 had schedule delays of at least 2 years or more.

As table 3 shows, ineffective DOE project oversight and poor contractor management were frequently cited reasons for cost increases and schedule delays for the projects we reviewed, according to our review of independent studies of the 9 projects experiencing cost growth and schedule delays and our follow-up interviews with DOE project directors. Project officials, in commenting on our draft report, were concerned that table 3 might misrepresent the overall successful execution and completion of some projects, such as the Spallation Neutron Source, and that some problems may have already been addressed. Nevertheless, to clarify our main purpose for table 3, our intent is to show broad categories of major reasons for cost increases and schedule delays, regardless of when they occurred or whether they have been adequately addressed.

The DOE project oversight issues mentioned in table 3 include the following: inadequate systems for measuring contractor performance; approval of construction activities before final designs were sufficiently insufficient DOE staffing and experience; inadequate use of project management controls; lack of headquarters assistance and oversight support of field project directors; failure to detect contractor performance problems, including inadequate federal inspection activities; and poor government cost estimates, including inadequate funding for contingencies.

DOE’s lack of adequate systems to measure contractor performance was cited in a December 2005 DOE Inspector General review of the Mixed Oxide Fuel Fabrication Facility. The Inspector General criticized DOE’s NNSA for failing to approve a baseline against which to measure contractor performance and relying on outdated cost plans. According to the report, NNSA relied on confusing and misleading information detailed in the monthly project reports to monitor progress and track costs— reports that the contractor acknowledged as being “useless for evaluating performance or managing the project.” Furthermore, although the contractor reported unfavorable cost and schedule variances for months, these variances were inaccurate and meaningless because performance was being compared against a 2-year-old plan. NNSA, in commenting on our draft report, stated that project oversight and contractor management problems identified in previous GAO, Inspector General, and other independent assessments have led to extensive improvements to the project, and that major findings identified during a recent independent review have been successfully addressed.

Similarly, DOE’s approval of construction activities before final designs were sufficiently complete has contributed significantly to project cost growth and schedule delays. As we have previously reported, the accelerated fast-track, design-build approach used for the Waste Treatment and Immobilization Plant, a highly complex first-of-a-kind nuclear facility, resulted in significant cost increases and schedule delays. DOE also allowed the contractor on another project, the Tritium Extraction Facility, to begin construction before the final design was completed to meet schedule commitments. According to a 2002 DOE Inspector General report on the project, this revised acquisition strategy of simultaneous design and construction directly resulted in at least $12 million in project overruns.

The contractor management issues mentioned in table 3 include the poor management of technological challenges, among other contractor performance issues, according to DOE project directors. Cost increases and schedule delays for 6 of the 9 projects were due in part to contractors’ poor management of the development and integration of technologies used in project designs by, among other things, not accurately anticipating the cost and time that would be required to carry out the highly complex tasks involved. For example: The National Ignition Facility had over $1 billion in cost overruns and years of schedule delays, in large part because of technology integration problems. The requirements for the National Ignition Facility—the use of 192 high-power laser beams focused on a single target in a “clean room” environment—had not been attempted before on such a large scale. According to the DOE project director, early incorrect assumptions about the original facility design and the amount of work necessary to integrate the technologies and assemble the technical components contributed to about half of the project’s cost increases and schedule delays.

The design of the Mixed Oxide Fuel Fabrication Facility has presented technical challenges in adapting the design of a similar plant in France to the design needs of this project. Although the technological challenge related to adopting the process designs from the French designs was not the primary contributor to the project’s cost increases and schedule delays, according to NNSA officials, it has affected the project’s complexity. The basic technology—combining plutonium oxide with depleted uranium to form fuel assemblies for use in commercial power reactors—has been previously demonstrated in France. However, the DOE project director told us that the DOE facility design must, among other things, account for processing surplus weapon-grade plutonium, a different type of material than processed in the French facility, and must be adapted to satisfy U.S. regulatory and other local requirements. In addition, the DOE facility faced the technological challenge of reducing the scale of components used in the French facility. Although definitive cost estimates are not yet available, expected costs for completing this project have grown by about $3.3 billion since 2002, and the schedule has been extended by more than 11 years, in part because the contractor did not initially understand the project’s complexity and underestimated the level of effort needed to complete the work. NNSA explained that the capability of the reference plants currently in operation in France, and by extension, the Mixed Oxide Fuel Fabrication Facility process design, is currently being demonstrated by several prototype fuel assemblies manufactured with weapon-grade plutonium oxide, which are currently being successfully used in a reactor in South Carolina.

For the Waste Treatment and Immobilization Plant, a technology application used on the project had not been tested before construction. Filters, widely used in the water treatment industry, were being designed for the project to concentrate and remove radioactive particles in liquid waste, a new application for the filters. Although tests are currently under way to demonstrate the effectiveness of this application, project officials conceded that these filters may still not be appropriate for the project.

Other contractor performance problems are illustrated by two examples. First, DOE cited the contractor working on the Highly Enriched Uranium Materials Facility for inadequate quality assurance that resulted in concrete work that did not meet design specifications. The subsequent suspension of construction activities and rework added to the project’s estimated cost and schedule. Second, the DOE project director of the Depleted Uranium Hexafluoride 6 Conversion Facility told us that the project was delayed 2 years because the contractor (1) did not have experience in government contracts, (2) underestimated the design effort needed, and (3) failed to properly integrate the operations of three separate organizations it managed.

As table 3 shows, external factors were cited as also contributing to cost growth and schedule delays, such as additional work to implement requirements for higher levels of safety and security in project operations, among other things. For example, design rework for 4 of the projects occurred in response to external safety oversight recommendations by the Defense Nuclear Facilities Safety Board that large DOE construction projects meet a certain level of personnel safety, and that their designs be robust enough to withstand certain seismic events. In addition, owing to new security requirements implemented after September 11, 2001, project officials on the Highly Enriched Uranium Materials Facility had to redesign some aspects of the project to ensure that heightened security measures were addressed.

While DOE faced additional requirements for safety and security, it did not always reach timely decisions on how to implement these requirements, which contributed significantly to cost increases and schedule delays for the Salt Waste Processing Facility. The DOE project director for this project told us the Defense Nuclear Facilities Safety Board had expressed concerns in June 2004, 5 months after the preliminary design was started, that the facility design might not ensure nuclear wastes would be adequately contained in the event of earthquakes. However, DOE did not decide how to address this concern until 17 months later, as the project continued to move forward with the existing project design. According to the project director, better and more timely discussions between site officials and headquarters to decide on the actions needed to adequately address these safety and security requirements might have hastened resolution of the problem, and up to 1 year of design rework might have been avoided. The delay, the director told us, added $180 million to the total project cost and extended the schedule by 26 months. In commenting on our draft report, EM officials noted that it is now requiring a more rigorous safety analysis earlier in the decision-making process.

Other external factors also contributed significantly to cost increases and delays for 2 interrelated projects we reviewed—the Mixed Oxide Fuel Fabrication Facility and the Pit Disassembly and Conversion Facility. Project officials for these projects told us that 25 to 50 percent of the cost increases and over 70 percent of the schedule delays they experienced were the direct result of Office of Management and Budget funding constraints and restrictions resulting from international agreements with Russia. That is, work that is delayed to a subsequent year because of funding constraints and other work restrictions can delay project completion, which likely increases total project costs. Similarly, Office of Science officials, commenting on our draft report, stated that external factors caused the largest percentage cost increase and schedule delay for the Spallation Neutron Source, including a reduced level of funding appropriated at a time when project activities and costs were increasing considerably. However, congressional funding was reduced in fiscal year 2000 because of concerns about poor project oversight and management in the early stages of this project.

DOE officials also explained that a now-defunct policy may have contributed to increased costs and delays for several projects we examined. Until 2000, DOE required contractors to prepare cost and schedule estimates early in the project, before preliminary designs were completed. These estimates were used to establish a baseline for measuring contractor performance and tracking any cost increases or schedule delays. However, these estimates often were based on early conceptual designs and, thus, were subject to significant change as more detailed designs were developed. To improve the reliability of these estimates, DOE issued a new order in October 2000 that required the preparation of a cost estimate range at the start of preliminary design, and delayed the requirement for a definitive cost and schedule baseline estimate until after the preliminary design was completed. Consequently, DOE officials explained, the new policy should result in improved estimates and a more accurate measure of cost and schedule performance.

We also sent a survey to DOE project directors for all 12 projects asking them to identify key events that led to the greatest cost increases or schedule delays, and the major factors contributing to these key events. However, no individual factors were identified as being major contributors to the cost increases or schedule delays. In responding to our survey, DOE project directors cited several factors that affected changes in cost and schedule. However, when asked to rate the relative significance of these factors for their impact on cost and schedule changes, the project directors generally did not judge them to be significant contributors to the changes. The most frequently cited factors were an absence of open communication, mutual trust, and close coordination; changes in “political will” during project execution (e.g., project changes resulting from political decisions, both internal and external to the project); interruptions in project funding; and project managers’ lack of adequate professional experience. (For detailed survey results covering these four factors, see app. IV.)

In contrast to the cost increases and schedule delays incurred on most of the projects we reviewed, 3 projects had not yet experienced cost increases or schedule delays—Microsystems and Engineering Sciences Applications, the Linac Coherent Light Source, and the Chemistry and Metallurgy Research Facility Replacement. DOE project officials identified key conditions that they believed helped avoid those cost increases and delays. These conditions included active oversight—that is, the DOE project directors were never “blindsided” by contractor issues; a lack of technological complexity; an effective system to measure contractor performance; effective communication with and integration of all stakeholders; and sustained leadership.

However, we observed that the Linac Coherent Light Source and the Chemistry and Metallurgy Research Facility Replacement facilities are still in a relatively early stage in the project development process, and thus it may be too early to gauge the overall success of either project. Additionally, because none of these 3 projects are highly technologically complex, they may be less susceptible to the types of problems associated with other projects we reviewed that experienced cost increases and delays.

DOE Does Not Consistently Measure Technology Readiness to Ensure That Critical Technologies Will Work as Intended before Construction Begins

Although DOE requires its final designs to be sufficiently complete before beginning construction, it has not systematically ensured that the critical technologies reflected in project designs are technologically ready. Recognizing that a lack of technology readiness can result in cost overruns and schedule delays, other federal agencies, such as NASA and DOD, have issued guidance for measuring and communicating technology readiness.

DOE Does Not Consistently Assess Technology Readiness

Only 1 of the 5 projects we reviewed to determine how DOE ensures that project designs are sufficiently complete before construction—projects that were approaching or had recently begun construction—had a systematic assessment of technology readiness to determine whether the project components would work individually or collectively as expected in the intended design. Specifically, the DOE project director for the Pit Disassembly and Conversion Facility systematically measured and assessed readiness levels for each critical component of the overall project. The assessment was based on a method developed by NASA, that is, rating each technology from 0 to 10 in terms of relative maturity. Because the project has not yet begun construction, we could not determine whether the technology readiness assessment has helped project managers to avoid cost increases or schedule delays during construction. However, according to DOE and contractor officials responsible for the project, the assessment helped focus management attention during project design on critical technologies that may require additional resources to ensure that they are sufficiently ready before construction begins. In reviewing the assessment, however, we noted that project officials had not updated the assessment tool for this project for over 3 years. DOE’s project director acknowledged the delay in updating the assessment and responded that he plans to begin updating the assessment annually.

The other 4 projects did not have systematic assessments of technological readiness. Therefore, the risk associated with the technology may not be clearly and consistently understood across all levels of management. Formally approving the project’s cost and schedule estimates as accurate and complete, or proceeding into construction, without having clearly assessed evidence of technology readiness can result in cost overruns and schedule delays.

DOE’s experience with the Waste Treatment and Immobilization Plant is a case in point. Specifically, technology known as “pulse jet mixers” was used in the design of a subsystem intended to prepare radioactive material for processing. However, this technology had not been used previously in this application, and it did not work in tests as expected, even after construction had already begun. Consequently, DOE incurred about $225 million in redesign costs and over 1 year in schedule delays, according to the DOE project director.

Over the past several years, we and others have stressed the importance of assessing technology readiness to complete projects successfully, while avoiding cost increases and schedule delays. Specifically, by 1999, we reported that organizations using best practices recognize that delaying the resolution of technology problems until production or construction can result in at least a 10-fold cost increase. Furthermore, we reported that delaying the resolution until after the start of production could increase costs by 100-fold. Reporting on similar concerns, the National Research Council has identified factors common to large construction projects—in the areas of cost, schedule, and scope—that help to ensure projects are completed successfully. Among key technical conditions for defining project scope, the council stated, is a project plan that is based on employing the best available, state-of-the-art technology, but not experimental or unproven technology. As such, employing a consistent, systematic method for measuring the extent to which technology is still experimental or unproven is of critical importance.

An assessment of technology readiness is even more crucial at certain points in the life of a project—particularly as DOE decides to accept a project’s (1) preliminary design and formally approve the project’s cost and schedule estimates as accurate and complete and (2) final design as sufficiently complete so that resources can be committed toward procurement and construction. Proceeding through these critical decision points without a credible and complete technology readiness assessment can lead to problems later in the project. Specifically, if DOE proceeds with the project when technologies are not yet ready, there is less certainty that the technologies specified in the preliminary or final designs will work as intended. Project managers may then need to modify or replace these technologies to make them work properly, which can result in costly and time-consuming redesign work.

Moreover, modifying the design of a facility after construction has already begun can be expensive and time consuming. First, changes to an already designed work plan are not necessarily subject to competition because the new work can occur through “change orders”—that is, modifications to existing contracts. These change orders can be expensive, according to DOE project directors. Second, worker productivity can be lost if, for example, extra downtime results from delays to interrelated construction work. Finally, tearing down and rebuilding items already constructed, such as concrete floors, walls, and doors, might be necessary to accommodate a design change.

DOE’s experience in the predecessor project to the Salt Waste Processing Facility—the In-Tank Precipitation (ITP) project process—at the Savannah River Site illustrates the potential consequences of proceeding with technology that is not sufficiently ready. As we reported in 2000, the ITP project was selected in 1983 as the preferred method for separating highly radioactive material from 34 million gallons of liquid stored at the Savannah River site—a step considered necessary to effectively handle this large quantity of waste. A 1983 test using the ITP technology on a tank containing 500,000 gallons of waste resulted in a significant buildup of benzene—a highly explosive and hazardous compound. The buildup of benzene was more than the tank instruments could register. Nevertheless, project managers decided to proceed with the project. In 1985, DOE estimated that it would take about 3 years and $32 million to construct the ITP facility. After a number of delays, the ITP facility was constructed and began start-up operations in 1995, which were halted because of safety concerns about the amount of benzene that the facility generated. In 1998, after about a decade of delays and costs of almost $500 million, DOE suspended the project because it did not work as safely and efficiently as designed. This suspension put an effective remedy for treating high-level waste at the Savannah River Site years behind schedule. DOE then directed its contractor to begin a process to identify and select an alternative technology, which has developed into the current project intended to treat this waste—the Salt Waste Processing Facility project.

In response to our concerns about the 4 projects without systematic assessments of technology readiness, DOE project directors explained that they have alternative methods for assessing readiness. They are required to submit a project execution plan, which includes an assessment of risks, including technological risks, and a plan for mitigating risks. They also rely upon independent reviews, including extensive design reviews, before making critical decisions to accept designs, and cost and schedule estimates, or to proceed with construction. For example, DOE’s Office of Engineering and Construction Management formally reviews major projects in an effort to ensure that the designs are sufficiently complete to begin construction. Specifically, an external independent readiness review is performed, often using the services of various independent technical experts, that, at a minimum, is intended to verify the readiness of the project to proceed into construction or to identify remedial action. Finally, several DOE project directors stated that they intentionally have avoided using fast-track, design-build approaches because of the many problems it posed for the Waste Treatment and Immobilization Plant project. The DOE project directors of the 5 DOE projects that are nearing, or have recently begun construction, told us they have completed, or expect to complete prior to construction, 85 to 100 percent of their projects’ final design.

In addition to following the more standard approaches for managing projects, such as preparing risk assessment plans, some DOE offices have developed their own tools for assessing the readiness of projects. For example, DOE’s Office of Environmental Management (EM) uses a Product Definition Rating Index (PDRI) as a tool to assess how well a project is planned, and whether it is ready to proceed to the next project phase. Project elements rated include cost, schedule, scope/technical, management planning and control, and external factors. Among the 77 project elements rated, 2 involve technology—the identification of technology development requirements, and the testing and evaluation of the technology to be used. While the project technologies are collectively given a ranking with this tool, the PDRI does not represent a rigorous examination of the demonstrated readiness of each critical technology for its application in the project. Furthermore, not all EM projects we examined were using this tool.

DOE’s design reviews, risk assessments, and other actions to monitor design completion are extensive and certainly have merit. However, we found that these actions alone do not provide consistent and transparent assurance that all technologies are sufficiently ready because they do not use a consistent and systematic method of measurement. DOE’s project design reviews, for example, do not always clearly distinguish between technology that has been demonstrated to work as expected in the intended design versus a judgment that the technology has potential for reaching a specific level of readiness.

The external review of the technologies for the Mixed Oxide Fuel Fabrication Facility illustrates the shortfalls in DOE’s current approach to assessing technology readiness and communicating the results of those assessments. The report concluded, among other things, that the method chosen by the contractor is the most rigorous and comprehensive, and should result in the most successful technology transfer possible. Furthermore, the review team was very impressed with the rigor with which designs and design changes were being managed, finding ample evidence verifying that the exact design process used by the French was being transferred to the United States facility. Although the external reviewers seemed to be impressed with many aspects of the design transfer, and concluded that the technologies should not be problematic, they had identified some key concerns about technology readiness in the body of their final report. The reviewers did not explain how they reconciled their conclusion with their concerns. To reconcile these differences, we obtained several clarifying statements from DOE’s project director, technical experts, and one of the study’s authors. These clarifying statements appear to support the reviewers’ conclusions. However, without these statements, the level of technological readiness was not readily evident because the independent review lacked consistent, systematic criteria and a method for measuring the degree of readiness or clearly communicating assessment results, and the review was not transparent.

DOE does not consistently assess technology readiness of project technologies because its project management guidance lacks comprehensive standards for systematically measuring and communicating the readiness of project technologies. Specifically, DOE lacks consistent metrics for determining technology readiness departmentwide, terminology to facilitate effective communication, and oversight protocols for reporting and reviewing technology readiness levels. DOE project management guidance is contained in two key documents—DOE Order 413.3A and Manual 413.3-1. Although the manual requires final designs to be sufficiently complete before beginning construction, it does not specify how technologies reflected in project designs are to be assessed for readiness—to determine that they have been sufficiently demonstrated to work as intended. Consequently, critical decisions made without standard measures are susceptible to varying interpretations of the actual technology readiness attained and the level needed for a project to proceed, which can easily vary among projects and among officials within a single project.

Other Federal Agencies Use a Standard Method for Measuring and Communicating Technology Readiness

Other federal agencies have recognized the importance of ensuring that technologies have been sufficiently demonstrated for their intended purpose and have issued standard guidance for measuring and communicating TRLs. In particular, recognizing the need to measure the readiness level of project technologies, NASA began using a systematic method of measurement in the mid-1990s. NASA incorporated a structured TRL approach into guidance on integrated technology planning.

Similarly, to improve DOD management of risk and technology development, the Deputy Under Secretary of Defense (Science and Technology) officially endorsed, in a July 2001 memorandum, the use of TRLs in new major programs. In 2002, DOD issued mandatory procedures for major defense acquisition programs and major automated information system acquisition programs, which identified technology readiness as a principal element of program risk. The procedures require the military services’ science and technology officials to conduct a systematic assessment of critical technologies that are identified in major weapon systems programs before starting engineering and manufacturing development and production. Using TRLs is the preferred method, and approval must be obtained from the Deputy Under Secretary if an equivalent alternative method is used, according to the Deputy Under Secretary’s memorandum. Importantly, the procedures stated that TRLs are a measure of demonstrated technical maturity—they do not discuss the probability of occurrence (i.e., the likelihood of attaining required maturity) or the impact of not achieving technology maturity.

Both NASA and DOD use a nine-point scale to measure technology readiness, from a low of TRL 1 (basic principles observed) to a high of TRL 9 (total system used successfully in project operations). (App. V contains the definitions of these nine TRLs.) For example, a subsystem prototype that has been successfully demonstrated in an operational environment would receive a higher TRL value (i.e., TRL 7) than a technological component that has been demonstrated in a laboratory test (i.e., TRL 4). In our previous work, we recommended to the Secretary of Defense that key project technologies used in weapons systems be demonstrated in an operational environment, reaching a high maturity level—analogous to TRL 7—before deciding to commit to a cost, schedule, and performance baseline for development and production of the weapon system. In response to our recommendation, DOD has agreed that if a technology does not achieve a score of TRL 6 or 7, project managers must develop a plan to bring the technology to the required readiness level before proceeding to the next project phase.

Use of TRLs is not by itself a cure-all for managing critical technologies, but TRLs can be used in conjunction with other measures to improve the way projects are managed. For example, according to studies by NASA, DOD, and others, TRLs can provide a common language among the technology developers, engineers who will adopt/use the technology, and other stakeholders; improve stakeholder communication regarding technology development— a by-product of the discussion among stakeholders that is needed to negotiate a TRL value; reveal the gap between a technology’s current readiness level and the readiness level needed for successful inclusion in the intended product; identify at-risk technologies that need increased management attention or additional resources for technology development to initiate risk-reduction measures; and increase transparency of critical decisions by identifying key technologies that have been demonstrated to work or by highlighting still immature or unproven technologies that might result in high project risk.

Two DOE headquarters offices have attempted to systematically assess technology readiness. First, under the Office of Nuclear Energy, a DOE contractor preparing a congressional report used a TRL method to compare the maturity of advanced fuel cycle technologies. In addition, in 2000, DOE’s Office of Science and Technology, under EM, issued a report that defined a process for assessing technology maturity of EM projects. However, according to an EM official, the office decided to discontinue using this assessment process because it was considered overly burdensome. As a result, DOE devolved responsibility for managing technology readiness to the contractor level.

According to several DOE project directors we spoke with, a consistent, systematic method for assessing technology readiness would help achieve a number of objectives: that is, standardize terminology, make technology assessments more transparent, and improve communication among project stakeholders before they make critical project decisions. DOE project managers also acknowledged that TRLs could improve project management departmentwide, and some managers are now attempting to use this tool to assess technology maturity. The DOE project director for the Waste Treatment and Immobilization Plant told us that a senior DOE official encouraged him to begin using TRLs. He is consulting with DOD officials knowledgeable about using the TRL method and expects to develop a TRL tool and have TRL determinations for major parts of the project in 2007. (App. VI compares DOD’s product development process with DOE’s project management process for major projects.)

Conclusions

The magnitude of the cost increases and schedule delays for DOE’s major projects is cause for serious rethinking of how DOE manages them. To its credit, DOE has completed, or expects to complete prior to construction, 85 to 100 percent of project design work for the 5 projects we reviewed that have recently begun or are nearing construction. However, DOE has not systematically addressed another key factor—the readiness level of the technologies it expects to use in these projects. DOE lacks comprehensive standards in DOE Order 413.3A and Manual 413.3-1 for systematically measuring and communicating the readiness of project technologies. Specifically, the department lacks consistent metrics for determining technology readiness departmentwide, terminology, and oversight protocols for reporting and reviewing TRLs. Without consistent measurement and communication of the readiness of technologies, DOE does not have a basis for defining the acceptable level of technological risk for each project, making critical decisions on accepting the validity of a project’s total estimated cost and schedule, or proceeding with construction.

Other federal agencies have recognized the need to consistently measure and communicate technology readiness to help avoid cost increases and delays that result from relying on immature technologies. DOD, for example, requires its managers to use a TRL process to measure technology readiness and generally requires a TRL 7 (as we had recommended) before system development and demonstration. In contrast, as DOE’s poor track record for managing the technological complexity of major projects shows, DOE has not systematically measured the readiness of critical project technologies before it approves definitive cost and schedule estimates or begins construction. Furthermore, without a systematic method for measuring technological readiness, DOE cannot effectively communicate within the department and to the Congress whether projects are at risk of experiencing cost increases and schedule delays associated with technology problems.

Recommendations for Executive Action

To improve decision making and oversight for major DOE construction projects, including how project technology readiness is measured and reported, we recommend that the Secretary of Energy evaluate and consider adopting a disciplined and consistent approach to assessing TRLs for projects with critical technologies that includes the following three actions: Develop comprehensive standards for systematically measuring and communicating the readiness of project technologies. At a minimum, these standards should (1) specify consistent metrics for determining technology readiness departmentwide, (2) establish terminology that can be consistently applied across projects, and (3) detail the oversight protocols to be used in reporting and reviewing TRLs. In preparing these standards, DOE should consider lessons learned from NASA and DOD, and its own experience in measuring technology readiness. If DOE’s evaluation results in the decision to adopt these standards, it should incorporate them into DOE Order 413.3A and Manual 413.3-1, and provide the appropriate training to ensure their proper implementation.

Direct DOE Acquisition Executives to ensure that projects with critical technologies reach a level of readiness commensurate with acceptable risk—analogous to TRL 7—before deciding to approve the preliminary design and commit to definitive cost and schedule estimates, and at least TRL 7 or, if possible, TRL 8 before committing to construction expenses.

Inform the appropriate committees and Members of Congress of any DOE decision to approve definitive cost and schedule estimates, or to begin construction, without first having ensured that project technologies are sufficiently ready (at TRL 7 or 8). This information should include specific plans for mitigating technology risks, such as developing backup technologies to offset the effects of a potential technology failure, and appropriate justification for accepting higher technological risk.

Agency Comments and Our Evaluation

We provided a draft of this report to DOE for its review and comment. DOE’s written comments are reproduced in appendix VII. DOE agreed with our recommendations but suggested revisions that would allow it to first conduct a pilot application on selected projects to better understand the technology readiness assessment process and evaluate its potential use. We revised our recommendations to give DOE this flexibility. DOE also provided detailed technical comments, which we have incorporated into our report as appropriate.

DOE also expressed several specific concerns with our draft report. First, DOE stated that while our draft broadly asserts that DOE project management has led to increases in cost and schedule, our recommendations are narrowly focused on technology assessment. We agree that our draft states that DOE project management has led to cost increases and schedule delays, a conclusion we reached on the basis of our contact with DOE project directors and our review of numerous studies and reports on DOE major projects. Our recommendations address technology assessment, a critical project management activity, because they were developed primarily on the basis of our specific finding that DOE lacks a systematic approach to ensure that final project designs, including critical technologies reflected in these designs, have been demonstrated to work as intended prior to construction. This report explains that delaying resolution of technology problems until construction can potentially lead to significant cost increases and schedule delays.

Second, DOE stated that our draft report inappropriately characterizes cost and schedule growth from a small sample of projects by using preliminary cost and schedule estimates that are intended for internal DOE planning. To clarify, the scope of our review included an evaluation of DOE’s major construction projects. In addition, our report explains that DOE changed its project management policy in 2000 to allow cost and schedule estimates to be prepared later in the project—at the end of preliminary design. Prior to this new policy, project directors submitted cost and schedule estimates earlier in the project development phase—at the end of conceptual design. For projects under way prior to the policy in 2000, we used post-2000 validated baseline estimates, if available. Otherwise, we used earlier estimates since these were the only estimates available and had been previously used by DOE to inform Congress of the total expected project cost and schedule while seeking initial project funding. We also note that for the five projects that were started after the new policy in 2000, we used the validated project baseline estimates recommended by DOE, if available.

Third, DOE suggested we revise table 3 in our report to more clearly identify the correlation between cost and schedule growth and technology maturity. As our report states, the information in table 3 was drawn from the results of our review of independent studies involving the projects we reviewed and the results of our interviews with DOE project directors. Our report explains that cost increases and schedule delays for 6 of the 9 projects shown in the table were due in part to contractors’ poor management of the development and integration of technologies used in the project designs.

Finally, DOE stated that it is unclear how the factors cited in appendix IV, such as communication, and changes in “political will,” among other things, led to our recommendation to assess technology readiness. Although not all of the factors cited in our survey have a link to our recommendation on technology readiness, one factor in particular— absence of communication—is addressed in our recommendation. Specifically, we recommended that the Secretary of Energy consider developing comprehensive standards for systematically measuring and communicating the readiness of project technologies, including the establishment of terminology that is to be consistently applied across projects.

We are sending copies of the report to interested congressional committees, the Secretary of Energy, and the Director of the Office of Management and Budget. We will make copies available to others on request. In addition, the report will also be available at no charge on the GAO Web site at http://www.gao.gov.

If you or your staffs have any questions about this report, please contact me at (202) 512-3841 or aloisee@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Other staff contributing to the report are listed in appendix VIII.

Appendix I: Scope and Methodology

To determine the extent to which the Department of Energy’s (DOE) major construction projects have experienced cost increases and schedule delays and the factors that have contributed to these problems, we identified (1) active DOE major line-item construction projects that have current total project cost estimates above the $750 million threshold— DOE’s criteria for “major construction projects,” and (2) the projects with estimates above $400 million—the DOE threshold for major projects until July 2006. We also identified those projects above $300 million to account for any projects that may pass the $400 million threshold. In all, we identified the following 12 projects: Five of these 12 projects began before DOE moved its requirement for firm cost and schedule estimates to later in the project: the National Ignition Facility, the Mixed Oxide Fuel Fabrication Facility, the Pit Disassembly and Conversion Facility, the Spallation Neutron Source, and the Tritium Extraction Facility. We used the estimates at the end of conceptual design, as reported by project directors, for the initial project cost and schedule estimates.

Four of the remaining 7 projects had cost and schedule estimates completed at the end of preliminary design, according to the new DOE guidelines: the Highly Enriched Uranium Materials Facility, Microsystems and Engineering Sciences Applications, the Depleted Uranium Hexafluoride 6 Conversion Facilities, and the Linac Coherent Light Source. For these projects, we considered the estimates as reported by project directors to be the initial cost and schedule estimates.

One project, the Waste Treatment and Immobilization Plant, began after DOE moved the requirement for firm cost and schedule estimates to later in the project. However, DOE initially exempted the contractor from submitting firm cost and schedule estimates. Therefore, we used the estimates reported by the project director to be the initial cost and schedule estimates.

The final 2 projects, although falling under the new DOE requirements, had yet to complete their preliminary design at the time of our review: the Chemistry and Metallurgy Research Facility Replacement and the Salt Waste Processing Facility. For these projects, we considered the cost and schedule estimates at the end of conceptual design reported by project directors to be the initial project cost and schedule estimates.

Because we and others have previously expressed concern about the data reliability of a key DOE project management tracking database—the Project Assessment and Reporting System—we did not develop conclusions or findings based on information generated through that system. Instead, we collected information directly through surveys and interviews with project site officials.

To identify cost increases and schedule delays, and the factors that may have contributed to these changes, we surveyed DOE project directors, interviewed DOE and contractor project personnel, and reviewed project management documents for 12 major projects. These 12 projects are managed by DOE’s Office of Science, Office of Environmental Management (EM), or National Nuclear Security Administration (NNSA). (See app. II for information on these projects.)

Our survey asked DOE project directors of the 12 projects to identify the degree to which cost and schedule estimates may have changed and the reasons for these changes, and to describe the events and conditions that led to any changes. Eight of the 12 project directors responded that their projects had experienced cost increases and schedule delays, and 1 project director reported only a schedule delay. For these 9 projects, we asked project directors to (1) identify the top three events that led to the cost and schedule delays and (2) indicate to what extent certain factors may have contributed to the event that led to the largest percentage cost increase or schedule delay. The factors included in the survey instrument were based on the results of a National Research Council study that listed essential or important conditions needed for the successful completion of major projects. We asked project directors to identify the extent to which the lack of these conditions may have contributed to any cost and schedule delays. (App. IV shows key survey results for these 9 projects.)

In addition to reviewing project documentation, we conducted site visits for the 9 projects that had experienced cost and schedule changes, and we analyzed (1) studies of these projects completed by DOE’s Office of Inspector General and (2) external independent project reviews conducted under the direction of DOE’s Office of Engineering and Construction Management in Washington, D.C. We interviewed federal project directors of the 3 projects that had not experienced cost increases or schedule delays to obtain information on factors they believe are important in avoiding such increases.

To determine the extent to which DOE ensures that project designs are sufficiently complete before construction, we obtained additional information from project directors on 5 projects that were approaching, or had recently begun, construction. During our review, we obtained information on the extent project designs were, or are expected to be, complete before beginning construction, and the actions DOE had taken to ensure technologies used in these designs are sufficiently ready to begin construction. For 2 of these 5 projects, we applied a tool we previously had used to assess DOD programs—the tool enables project directors to characterize the readiness level of each technology being developed for use in aircraft and other military applications. In addition, we spoke with officials from DOE program offices and DOE’s Office of Engineering and Construction Management in Washington, D.C.

We provided interim briefings to the Subcommittee on Energy and Water Development, House Committee on Appropriations, on the status of our work in May and September, 2006. We performed our work between December 2005 and January 2007, in accordance with generally accepted government auditing standards.

Appendix II: Information on the 12 Department of Energy Major Projects Reviewed

Relocate and consolidate mission-critical analytical chemistry, material characterization, and research and development capabilities to ensure continuous national security mission support beyond 2010.

Design and construct facilities at Portsmouth, Ohio, and Paducah, Kentucky, to convert the Department of Energy’s existing inventory of depleted uranium hexafluoride into a more stable form for disposal or beneficial reuse.

Project will construct a highly secure, state-of-the-art facility for consolidating and storing highly enriched uranium, resulting in cost savings and an increased security posture.

Provide laser-like radiation in the X-ray region of the spectrum that is 10 billion times greater in peak brightness than any existing X-ray light source. The project will apply these high-brightness X-rays to experiments in the chemical, material, and biological sciences.

Provide state-of-the-art national complex that will provide for the design, integration, prototyping, and qualification of microsystems into components, subsystems, and systems within the nuclear weapons stockpile.

Facility will combine surplus weapon-grade plutonium oxide with depleted uranium to form mixed oxide fuel assemblies that will be irradiated in United States commercial nuclear reactors. Once irradiated and converted into spent fuel, the resulting plutonium can no longer be readily used for nuclear weapons.

Provide experimental capability to assess nuclear weapons physics, providing critical data that will allow the United States to maintain its technical capabilities in nuclear weapons in the absence of underground testing, and to advance fusion as an energy source.

Eliminate surplus Russian and United States plutonium and highly enriched uranium by disassembling surplus nuclear weapons pits and converting the resulting plutonium metal to a powder form that can later be fabricated into mixed oxide fuel to produce nuclear fuel assemblies for use in commercial nuclear reactors.

Meet site cleanup goals and reduce significant environmental and health/safety risk by construction of a facility to treat large quantities of waste from reprocessing and nuclear materials production operations at the Savannah River Site. Process will separate waste, solidify it in glass, and send it to federal repositories for disposal.

Provide next generation, short-pulse spallation neutron source for neutron scattering, to be used by researchers from academia, national and federal labs, and industry for basic and applied research and technology development in the fields of condensed matter physics, materials sciences, magnetic materials, polymers and complex fluids, chemistry, biology, earth sciences, and engineering.

To replenish the tritium needs of the nuclear weapons stockpile, the facility will extract tritium produced in a commercial nuclear reactor for use in nuclear weapons development.

The plant will separate high-level from low-level radioactive waste currently stored in underground tanks, processing and solidifying all high- level waste and a substantial portion of the low-level waste, and will treat the remaining low-level waste.

Appendix III: Independent Studies Reviewed

Department of Energy, Office of Inspector General. Audit Report: Status of the National Ignition Facility Project. DOE/IG-0598. Washington, D.C.: April 28, 2003.

GAO. Department of Energy: Status of Contract and Project Management Reforms. GAO-03-570T. Washington, D.C.: March 20, 2003.

GAO. Contract Reform: DOE Has Made Progress, but Actions Needed to Ensure Initiatives Have Improved Results. GAO-02-798. Washington, D.C.: September 13, 2002.

GAO. Department of Energy: Follow-up Review of DOE’s National Ignition Facility. GAO-01-677R. Washington, D.C.: June 1, 2001.

GAO. National Ignition Facility: Management and Oversight Failures Caused Major Cost Overruns and Schedule Delays. GAO/RCED-00-141 and GAO/RCED-00-271. Washington, D.C.: August 8, 2000.

The Mitre Corporation. NIF Ignition. JSR-05-340. McLean, VA: June 29, 2005.

Mixed Oxide Fuel Fabrication Facility

Burns and Roe Enterprises, Inc. External Independent Review of the Mixed Oxide Fuel Fabrication Facility (MFFF) Project Critical Decision (CD) 2/3 Baseline: Performance Baseline (CD-2) and Start of Construction (CD-3) Review. BREI-L-R-06-03. Oradell, NJ: July 7, 2006.

Burns and Roe Enterprises, Inc. External Independent Review of the Basis of Design for the Aqueuous Polishing Process. BREI-SLP-R-06-01. Oradell, NJ: March 27, 2006.

Civil Engineering Research Foundation. Independent Research Assessment of Project Management Factors Affecting Department of Energy Project Success. Washington, D.C.: July 12, 2004.

Department of Energy, Office of Inspector General. Audit Report: Status of the Mixed Oxide Fuel Fabrication Facility. DOE/IG-0713. Washington, D.C.: December 21, 2005.

Pit Disassembly and Conversion Facility

Department of Energy, Office of Inspector General. Audit Report: National Nuclear Security Administration’s Pit Disassembly and Conversion Facility. DOE/IG-0688. Washington, D.C.: May 3, 2005.

Los Alamos National Laboratory. Options for the Development and Testing of the Pit Disassembly and Conversion Facility Government- Furnished Design. LA-UR-03-3926. Los Alamos, NM: June 11, 2003.

Waste Treatment and Immobilization Plant

Bechtel National, Inc. Hanford Tank Waste Treatment and Immobilization Plant, May 2006 Estimate at Completion. Hanford Site, WA: May 31, 2006.

Bechtel National, Inc. Comprehensive Review of the Hanford Tank Waste Treatment and Immobilization Plant Estimate at Completion. CCN 132848. Hanford Site, WA: March 31, 2006.

Bechtel National, Inc. Comprehensive Review of the Hanford Waste Treatment Plant Flowsheet and Throughput. CCN132846. Hanford Site, WA: March 17, 2006.

Bechtel National, Inc. Hanford Tank Waste Treatment and Immobilization Plant, December 2005 Estimate at Completion Executive Summary. Hanford Site, WA: January 30, 2006.

Department of the Army Corp of Engineers. Complete Statement of Kim Callan, to the Subcommittee on Energy and Water Development, Committee on Appropriations, United States House of Representatives. Washington, D.C.: April 6, 2006.

Department of Energy. External Independent Review, Independent Cost Review, CD-3C Review of the Waste Treatment and Immobilization Plant Project. Hanford Site, WA: September 2002.

Department of Energy. External Independent Review CD-3B Review of the Waste Treatment and Immobilization Plant Project. Hanford Site, WA: April 2002.

GAO. Hanford Waste Treatment Plant, Contractor and DOE Management Problems Have Led to Higher Costs, Construction Delays, and Safety Concerns. GAO-06-602T. Washington, D.C.: April 6, 2006.

GAO. Further Actions Are Needed to Strengthen Contract Management for Major Projects. GAO-05-123. Washington, D.C.: March 18, 2005.

GAO. Nuclear Waste: Absence of Key Management Reforms on Hanford’s Cleanup Project Adds to Challenges of Achieving Cost and Schedule Goals. GAO-04-611. Washington, D.C.: June 9, 2004.

GAO. Status of Contract and Project Management Reforms. GAO-03-57T. Washington, D.C.: March 20, 2003.

GAO. Contract Reform: DOE Has Made Progress, but Actions Needed to Ensure Initiatives Have Improved Results. GAO-02-798. Washington, D.C.: September 13, 2002.

GAO. Nuclear Waste: Hanford Tank Waste Program Needs Cost, Schedule, and Management Changes. GAO/RCED-93-99. Washington, D.C.: March 8, 1993.

LMI Government Consulting. Hanford Waste Treatment and Immobilization Plant After-Action Fact-Finding Review. DE535T1. McLean, VA: January 2006.

LMI Government Consulting. External Independent Review, Follow-up Review, Waste Treatment and Immobilization Plant (WTP) Out Briefing. Washington, D.C.: March 14, 2003.

Spallation Neutron Source

Civil Engineering Research Foundation. Independent Research Assessment of Project Management Factors Affecting Department of Energy Project Success. Washington, D.C.: July 12, 2004.

Department of Energy, Office of Inspector General. Audit Report: Progress of the Spallation Neutron Source Project. DOE/IG-0532. Washington, D.C.: November 19, 2001.

Department of Energy. Review Committee Report on the Baseline Review of the Spallation Neutron Source (SNS) Project. Washington, D.C.: July 15, 1999.

Department of Energy. Technical, Cost, Schedule, and Management Review of the Spallation Neutron Source Project. Washington, D.C.: January 28, 1999.

GAO. Department of Energy: Status of Contract and Project Management Reforms. GAO-03-570T. Washington, D.C.: March 20, 2003.

GAO. Contract Reform: DOE Has Made Progress, but Actions Needed to Ensure Initiatives Have Improved Results. GAO-02-798. Washington, D.C.: September 13, 2002.

GAO. Department of Energy: Challenges Exist in Managing the Spallation Neutron Source Project. GAO/T-RCED-99-103. Washington, D.C.: March 3, 1999.

Salt Waste Processing Facility

Department of Energy, Office of Inspector General. Audit Report: Salt Processing Project at the Savannah River Site. DOE/IG-0565. Washington, D.C.: August 27, 2002.

Institute for Regulatory Science. Technical Peer Review Report of the Review Panel on Salt Waste Processing Facility Technology Readiness. CRTD-Vol. 75. Danvers, MA: October 31, 2003.

Tritium Extraction Facility

Civil Engineering Research Foundation. Independent Research Assessment of Project Management Factors Affecting Department of Energy Project Success. Washington, D.C.: July 12, 2004.

Department of Energy, Office of Inspector General. Audit Report: The Department of Energy’s Tritium Extraction Facility. DOE/IG-0560. Washington, D.C.: June 24, 2002.

GAO. Department of Energy: Further Actions Are Needed to Strengthen Contract Management for Major Projects. GAO-05-123. Washington, D.C.: March 18, 2005.

GAO. Department of Energy: Status of Contract and Project Management Reforms. GAO-03-570T. Washington, D.C.: March 20, 2003.

GAO. Contract Reform: DOE Has Made Progress, but Actions Needed to Ensure Initiatives Have Improved Results. GAO-02-798. Washington, D.C.: September 13, 2002.

GAO. Nuclear Weapons: Design Reviews of DOE’s Tritium Extraction Facility. GAO/RCED-98-75. Washington, D.C.: March 31, 1998.

National Nuclear Security Administration. Program Review of the Estimate to Complete Tritium Extraction Facility (TEF) at Savannah River Site. Washington, D.C.: August 29, 2002.

Highly Enriched Uranium Materials Facility

BWXT Y-12. Highly Enriched Uranium Materials Facility Project Causal Analysis Report. Oak Ridge, TN: March 6, 2006.

Department of Energy. Limited External Independent Review for Baseline Change Proposal Review. Oak Ridge, TN: August 31, 2004.

Department of Energy, Office of Inspector General. Audit Report, Design of the Uranium Storage Facility at the Y-12 National Security Complex. DOE/IG-0643. Washington, D.C.: March 19, 2004.

Department of Energy, Office of Inspector General. Audit Report, Reestablishment of Enriched Uranium Operations at the Y-12 National Security Complex. DOE/IG-0640. Washington, D.C.: February 24, 2004.

Department of Energy. External Independent Review – Performance Baseline Review of the Highly Enriched Uranium Materials Facility Project. Oak Ridge, TN: June 2003.

Depleted Uranium Hexafluoride 6 Conversion Facility

Department of Energy. Report on the Independent Project Review of the Depleted Uranium Hexafluoride Conversion Project. Washington, D.C.: October 8, 2004.

Department of Energy, Office of Inspector General. Audit Report: Depleted Uranium Hexafluoride Conversion. DOE/IG-0642. Washington, D.C.: March 18, 2004.

GAO. Department of Energy: Status of Contract and Project Management Reforms. GAO-03-570T. Washington, D.C.: March 20, 2003.

LMI Government Consulting. DUF6 Conversion Project CD-3 Corrective Action Plan Review. DE538T1. McLean, VA: October 2005.

LMI Government Consulting. Construction Readiness EIR (for CD-3) of the Depleted Uranium Hexafluoride Conversion Project. DE534T1. McLean, VA: June 2005.

LMI Government Consulting. DUF6 Conversion Project CD-3C Construction Readiness Review Preliminary Draft. Washington, D.C.: May 20, 2005.

LMI Government Consulting. DUF6 Limited Conversion Plan Project External Independent Review for the Office of Engineering and Construction Management. DE428T1. McLean, VA: June 2004.

Chemistry and Metallurgy Research Facility Replacement

Jupiter Corporation. External Independent Review of the Chemistry and Metallurgy Research Building Replacement Project. Approve Performance Baseline and Approve Start of Construction. CD-2A/CD-3A. Wheaton, MD: October 14, 2005.

Appendix IV: Survey Results for Primary Factors Affecting Cost and Schedule on Nine Projects with Cost or Schedule Changes

Absence of open communication, mutual trust, and close coordination Depleted Uranium Hexafluoride 6 Conversion Facility Highly Enriched Uranium Materials Facility Pit Disassembly and Conversion Facility Waste Treatment and Immobilization Plant Changes in “political will” during project execution (e.g., project changes resulting from political decisions—includes politics internal and external to the project )

Appendix V: Definitions of Technology Readiness Levels

Appendix V: Definitions of Technology Readiness Levels None.

None.

Research to prove feasibility.

Desktop, “back of envelope” environment.

Research to prove feasibility.

None.

Paper studies indicate components ought to work together.

Academic environment. The emphasis here is still on understanding the science but beginning to think about possible applications of the scientific principles.

Research to prove feasibility.

No system components, just basic laboratory research equipment to verify physical principles.

No attempt at integration; still trying to see whether individual parts of the technology work. Lab experiments with available components show they will work.

Uses of the observed properties are postulated and experimentation with potential elements of subsystem begins. Lab work to validate pieces of technology without trying to integrate. Emphasis is on validating the predictions made during earlier analytical studies to ensure that the technology has a firm scientific underpinning.

Demonstrate technical feasibility and functionality.

Ad Hoc and available laboratory components are surrogates for system components that may require special handling, calibration, or alignment to get them to function. Not fully functional but representative of technically feasible approach.

Available components assembled into subsystem breadboard. Interfaces between components are realistic.

Tests in controlled laboratory environment. Lab work at less than full subsystem integration, although starting to see if components will work together.

Demonstrate technical feasibility and functionality.

Fidelity of components and interfaces are improved from TRL 4. Some special purpose components combined with available laboratory components. Functionally equivalent but not of same material or size. May include integration of several components with reasonably realistic support elements to demonstrate functionality.

Fidelity of subsystem mock up improves (e.g., from breadboard to brassboard). Integration issues become defined.

Laboratory environment modified to approximate operational environment. Increases in accuracy of the controlled environment in which it is tested.

Demonstrate applicability to intended project and subsystem integration. (Specific to intended application in project.)

Subsystem is high fidelity functional prototype with (very near same material and size of operational system). Probably includes the integration of many new components and realistic supporting elements/subsystems if needed to demonstrate full functionality. Partially integrated with existing systems.

Components are functionally compatible (and very near same material and size of operational system). Component integration into system is demonstrated.

Relevant environment inside or outside the laboratory, but not the eventual operating environment. The testing environment does not reach the level of an operational environment, although moving out of controlled laboratory environment into something more closely approximating the realities of technology’s intended use.

Demonstrate applicability to intended project and subsystem integration. (Specific to intended application in project.)

Prototype improves to preproduction quality. Components are representative of project components (material, size, and function) and integrated with other key supporting elements/subsystems to demonstrate full functionality. Accurate enough representation to expect only minor design changes.

Prototype not integrated into intended system but onto surrogate system.

Operational environment, but not the eventual environment. Operational testing of system in representational environment. Prototype will be exposed to the true operational environment on a surrogate platform, demonstrator, or test bed.

Applied/Integrated into intended project application.

Components are right material, size, and function compatible with operational system.

Subsystem performance meets intended application and is fully integrated into total system.

Demonstration, test, and evaluation completed. Demonstrates system meets procurement specifications. Demonstrated in eventual environment.

Applied/Integrated into intended project application.

Components are successfully performing in the actual environment— proper size, material, and function.

Subsystem has been installed and successfully deployed in project systems.

Operational testing and evaluation completed. Demonstrates that system is capable of meeting all mission requirements.

Appendix VI: Comparison of DOD’s Product Development Process with DOE’s Project Management Process

DOD’s Product Development Process

application) application)

Appendix VII: Comments from the Department of Energy

Appendix VIII: GAO Contact and Staff Acknowledgments

Staff Acknowledgments

In addition to the individual named above, Michaela Brown, Rudy Chatlos, James Espinoza, Daniel Feehan (Assistant Director), Joseph Keener, Thomas Kingham, Matthew Lea, Mehrzad Nadji, Omari Norman, Christopher Pacheco, Thomas Perry, and Carol Herrnstadt Shulman made key contributions to this report.