Project and Program Management: DOE Needs to Revise Requirements and Guidance for Cost Estimating and Related Reviews
Released: 2014-11-25T00:00:00

Letter

Background

This section describes: (1) NNSA organization and management, (2) DOE and NNSA cost estimating requirements and guidance for projects, (3) DOE and NNSA cost estimating requirements and guidance for programs, (4) our 2009 cost estimating best practices, and (5) federal standards for internal controls.

NNSA Organization and Management

To fund NNSA’s projects and programs, the President requested approximately $11.7 billion for NNSA in the fiscal year 2015 budget submission to Congress, the majority of which is intended to fund program operations. Work activities on both projects and programs are largely carried out by management and operating (M&O) contractors at NNSA’s eight government-owned, contractor-operated sites—collectively referred to as the nuclear security enterprise. Among other things, these contractors operate and maintain the government-owned facilities and infrastructure deemed necessary to support the nuclear security enterprise and to support the capabilities to conduct scientific, technical, engineering, and production activities.

As shown in figure 1, the nuclear security enterprise sites include national research and development laboratories, as well as production plants. Specifically, NNSA manages three national nuclear weapons design laboratories—Lawrence Livermore National Laboratory in California, Los Alamos National Laboratory in New Mexico, and Sandia National Laboratories in New Mexico and California. It also manages four nuclear weapons production plants—the Pantex Plant in Texas, the Y-12 National Security Complex in Tennessee, the Kansas City Plant in Missouri, and Tritium Operations at DOE’s Savannah River Site in South Carolina. NNSA also manages the Nevada National Security Site, formerly known as the Nevada Test Site. Various headquarters organizations within NNSA develop policies and NNSA site offices, collocated with NNSA’s sites, conduct day-to-day oversight of the M&O contractors, and evaluate the M&O contractors’ performance in carrying out the sites’ missions.

The Secretary of Energy is authorized under the National Nuclear Security Administration Act to establish policy and provide direction to NNSA.authority to establish NNSA-specific policies, unless disapproved by the The NNSA Administrator, however, is also vested with the Secretary of Energy. NNSA does this through the issuance of Policy Letters. According to NNSA officials responsible for project management, DOE directives and orders and NNSA Policy Letters, including business operating procedures, collectively establish requirements for the development of NNSA project and program cost estimates, and we refer to them as such throughout this document.

DOE and NNSA Cost Estimating Requirements and Guidance for Projects

NNSA is required to manage its projects, including the development of cost estimates, in accordance with DOE Order 413.3B. The purpose of the order is to provide program and project management direction for the acquisition of capital assets, and it includes requirements for developing project cost estimates. The order also provides management direction for NNSA and other DOE offices with the goal of delivering projects within the original performance baseline that are fully capable of meeting mission performance and other requirements such as environmental, safety, and health standards.or critical decision points—that span the life of a project. These critical decision (CD) points are: The order defines five major milestones—

CD-0: Approve mission need.

CD-1: Approve alternative selection and cost range.

CD-2: Approve project performance baseline.

CD-3: Approve start of construction.

CD-4: Approve start of operations or project completion.

The order specifies requirements that must be met, along with the documentation necessary, to move a project past each CD point. In addition, the order requires senior management to review the supporting documentation and decide whether to approve the project at each CD.

DOE also provides suggested approaches for meeting the requirements contained in Order 413.3B through a series of guides, such as guides for project reviews and risk management.

The order includes several cost estimating requirements for projects. Specifically, the order requires: a cost estimate be provided at each CD point; the degree of rigor and detail for a cost estimate be carefully defined, depending on the degree of confidence in project scale and scope that is reasonable to expect at that stage; a cost estimate range for CD-0 and 1 explicitly noting any relevant caveats concerning uncertainties; and a program sponsor never be the sole cost estimator at any critical decision given the inherent conflict of interest, and a second cost estimator should come from outside of the line manager’s chain of command to avoid conflict of interest.

The order also establishes requirements for the review of cost estimates for certain projects at CD-0 through CD-3, depending on the estimated cost of the project. Table 1 shows these requirements and an NNSA requirement for review.

As a supplement to Order 413.3B, DOE provides guidance for the development of cost estimates in its Cost Estimating Guide, which was issued in May 2011. According to the guide, its purpose is to provide uniform guidance and best practices that describe recommended methods and procedures that projects and programs could use for preparing cost estimates. The guide states that it is based on accepted industry practices and processes, including practices outlined in our 2009 Cost Estimating and Assessment Guide, to meet federal and DOE requirements.

DOE and NNSA Cost Estimating Requirements and Guidance for Programs

DOE and NNSA cost estimating requirements and guidance for programs are primarily defined in four documents.

DOE Order 130.1. This order requires all DOE and NNSA program budget requests be based on cost estimates that have been thoroughly reviewed and deemed reasonable prior to their inclusion in department budgets.

DOE Cost Estimating Guide 413.3-21. DOE’s cost estimating guide is primarily intended for use in managing the phases involved in the acquisition of a capital asset project. However, the guide states that it also could be used by DOE and NNSA programs in preparing cost estimates.

NNSA Planning, Programming, Budgeting, and Evaluation (PPBE)

Requirements. NNSA’s PPBE process provides a framework for the agency to plan, prioritize, fund, and evaluate its program activities. This process includes a budget validation step to review the process contractors follow in developing program cost estimates for budgeting purposes.

NNSA Phase 6.X Process. NNSA manages life extension programs, including the B61 LEP, according to a process called Phase 6.X that was developed by DOE in collaboration with the Department of Defense. The Phase 6.X process provides NNSA with a framework to conduct and manage LEP activities, including the development and documentation of a program cost estimate.

GAO Cost Estimating Best Practices

Drawing from federal cost estimating organizations and industry, our cost estimating guide provides best practices about the processes, procedures, and practices needed for ensuring development of high- quality—that is, reliable cost estimates. A high-quality cost estimate helps ensure that management is given the information it needs to make informed decisions. The guide identifies the following four characteristics of a high-quality cost estimate. Specifically, such an estimate is:

Credible when it has been cross-checked with an ICE, the level of confidence associated with the point estimate has been identified through the use of risk and uncertainty analysis, and a sensitivity analysis has been conducted;

Well-documented when supporting documentation is accompanied by a narrative explaining the process, sources, and methods used to create the estimate and contains the underlying data used to develop the estimate;

Accurate when it is not overly conservative or too optimistic and based on an assessment of the costs most likely to be incurred; and

Comprehensive when it accounts for all possible costs associated with a project, it contains a cost estimating structure in sufficient detail to ensure that costs are neither omitted nor double-counted, and the estimating teams’ composition is commensurate with the assignment.

To develop a cost estimate that embodies these four characteristics, our cost estimating guide lays out 12 best practice steps. For example, one step—determining the estimating structure—includes the need to develop a “product-oriented” work breakdown structure (WBS) that reflects the requirements and basis for identifying resources and tasks necessary to accomplish the project’s objectives. A product-oriented WBS is organized to reflect the cost, schedule, and technical performance of project components. Such a WBS allows a project to track cost by defined deliverables, promote accountability by identifying work products that are independent of one another, and provides a basis for identifying resources and tasks for developing a cost estimate. Table 2 shows the four characteristics of a high-quality cost estimate with the corresponding steps.

Federal Standards for Internal Controls

In addition to the requirements for cost estimating established by DOE and NNSA, according to Standards for Internal Control in the Federal Government, federal agencies are to employ internal control activities, such as functional reviews by management of projects and programs to compare actual performance to planned or expected results throughout the organization and analyze significant differences. Such reviews, which may include reviews of program cost estimates, are to help ensure that management’s directives are carried out and to determine if agencies are effectively and efficiently using resources.

DOE and NNSA Cost Estimating Requirements and Guidance Generally Do Not Reflect Best Practices for Developing Cost Estimates

DOE and NNSA cost estimating requirements and guidance for projects and programs generally do not reflect best practices for developing cost estimates. For projects, DOE requires one best practice, and DOE’s cost estimating guide does not fully reflect cost estimating best practices, and is not mandatory. For programs, DOE does not require the use of any of the best practices to develop cost estimates, and NNSA has taken steps to close gaps in the existing cost estimating framework for programs, such as creating a cost analysis office, but these steps have not included a requirement to use best practices.

Projects Are Not Required to Meet Most Best Practice Steps, and Guidance Describes Most Best Practice Steps but Is Not Mandatory

DOE and NNSA cost estimating requirements and guidance for projects generally do not reflect best practices for developing cost estimates. DOE’s 2010 project management order requires the use of only 1 of the 12 cost estimating best practice steps. Specifically, the order requires an ICE be prepared at CD-2 and CD-3 for projects with an estimated cost of $100 million or greater. The order requires the development of an ICE at CD-3 if warranted by risk and performance indicators or as designated by DOE or NNSA management. In addition, NNSA’s 2014 requirement for an ICE or independent cost review could subject additional projects with an estimate of a cost of less than $100 million to an ICE, but this would depend on whether NNSA chooses to conduct an ICE rather than the less rigorous independent cost review. None of the other cost estimating requirements in the order, such as the need for a cost estimate at each CD point, ensure that project cost estimates will be prepared in accordance with cost estimating best practices. For example, the order does not require any of the other 11 best practice steps such as conducting a risk and uncertainty analysis, identifying ground rules and assumptions, documenting the estimate, developing a point estimate, or determining the estimating structure. According to the DOE officials responsible for developing DOE’s project management order, DOE has chosen to not require all cost estimating best practices in the order and instead includes suggested approaches for developing cost estimates in the DOE cost estimating guide that accompanies the order. However, because neither DOE nor NNSA requires the use of most cost estimating best practices for its projects, it is unlikely that NNSA and its contractors will consistently develop reliable cost estimates.

DOE’s 2011 cost estimating guide describes most best practices, but it is not mandatory and it is not referenced in the order. We found that the guide fully or substantially describes 10 of the 12 best practices. However, the guide only partially or minimally contains information about the other 2 best practice steps—determining the estimating structure and conducting a sensitivity analysis. For the estimating structure, the guide only partially describes how to develop this structure, which primarily involves establishing an adequate WBS. For example, we found that the guide describes the need for a WBS, but it does not describe the need for a product-oriented WBS and does not provide enough information on how to create a WBS. For the sensitivity analysis, the guide only minimally describes the information needed. For example, the guide suggests briefing management on the results of a sensitivity analysis, but it does not address the best practice steps on how to perform a sensitivity analysis. As a result, DOE and NNSA have not provided its contractors with all the detailed guidance needed to consistently develop reliable cost estimates. When we discussed our analysis of this guide and the shortcomings we found in regard to the guide with the DOE officials responsible for developing the guide, they said that they would review our analysis and consider revisions to the guide as appropriate. Appendix I provides a comparison of the extent to which DOE’s cost guide contains information on the 12 best practice steps.

In addition to the shortcomings in the description for two of the best practices, the use of the guide is not mandatory and, as a result, neither NNSA staff nor its contractors who develop cost estimates are required to use it. The guide states that it does not impose any new cost estimating requirements or establish departmental policy; it describes only suggested recommended approaches for preparing cost estimates. Further, DOE’s cost estimating guide is not referenced in the order. To supplement DOE’s project management order, DOE has issued a series of guides that provide suggested approaches to meeting the requirements, some of which are specified in the order. There are a total of 18 separate guides, including guides that provide suggested approaches on topics such as systems engineering, earned value management systems, and project closeout. The order references some of these guides as appropriate so that the user of the order is made aware of the existence of the suggested approaches that are included in the guides. However, because the cost guide was issued in 2011, a year after DOE’s latest version of the order was issued, the order contains no reference to this guide that a preparer of a cost estimate could use in developing an estimate, or that a reviewer could use in reviewing one, and users of the order may not be aware of the guide’s availability and may not benefit from its usefulness.

Programs Are Not Required to Meet Any Cost Estimating Best Practices

DOE and NNSA programs are not required to meet any cost estimating best practices. NNSA officials explained that NNSA cost estimating practices for programs are limited, de-centralized, and inconsistent, and are not governed by a cost estimating policy or single set of NNSA requirements and guidance. According to these officials, each NNSA program office uses different practices and procedures for the development of cost estimates that are included in the NNSA annual budget. In addition, these officials also explained that, while there are no specific requirements on how program cost estimates should be developed or reviewed, all NNSA programs must follow DOE’s budget formulation order and NNSA’s PPBE process, but the order does not require, and the PPBE process does not direct the use of best practices in developing cost estimates. In 2012, we reported on NNSA’s PPBE process, and we noted significant deficiencies in NNSA’s implementation of its PPBE process.ensure the development of fully credible program cost estimates, and that the review of the cost estimates supporting DOE’s budget was not sufficiently thorough to ensure the credibility and reliability of NNSA’s budget estimates. We also found that NNSA was not following DOE’s budget formulation order that requires that budget requests be based on Specifically, we found that the process does not cost estimates that have been thoroughly reviewed and deemed reasonable by the cognizant program organization. NNSA officials told us that they do not follow DOE’s budget formulation order because it expired in 2003. Additionally, according to NNSA officials, NNSA’s trust in its contractors minimizes the need for formal review of its budget estimates.

In our 2012 report on the NNSA’s PPBE process we recommended that, among other things, DOE update the departmental order for budget reviews, improve the formal process for reviewing budget estimates, and reinstitute an independent analytical capability to provide senior decision makers with independent program reviews. The agency agreed in principle with these recommendations, but it has not taken action to fully implement them. According to NNSA officials, as of August 2014, NNSA has suspended the budget review process and several options for replacing this process are under review.

In the absence of a DOE or NNSA requirement for programs to follow cost estimating best practices, we found that managers for the programs we reviewed—the B61 LEP and the Plutonium Disposition Program— used different processes in developing cost estimates for these programs. Specifically:

B61 LEP. NNSA managers for the B61 LEP use the Phase 6.X Process which, among other things, involves the direction to establish a cost and schedule baseline for the program. However, according to the NNSA B61 LEP project manager, since the Phase 6.X Process does not define how cost estimates should be developed, the B61 management team developed an approach for developing cost estimates for the program using various sources, including direction under the Phase 6.X Process, as well as DOE’s project management order and cost guide and our cost-estimating guide. This effort resulted in the management team producing a document that defines the strategy and provides guidance for completing the cost estimate for the B61 LEP. We reviewed this document and found that it does not stipulate that NNSA program managers or its contractors must follow any DOE or NNSA requirements or guidance for the development of a program cost estimate when developing the estimate for the B61 LEP.

Plutonium Disposition Program. In February 2014, we found that NNSA’s life-cycle cost estimate for the Plutonium Disposition Program did not follow all key steps for developing high-quality cost estimates in part because it did not have a requirement to develop a life-cycle cost estimate. According to NNSA officials, DOE’s project management order includes requirements for development of cost and schedule estimates for a capital asset project, such as the MOX facility or Waste Solidification Building, but does not specify equivalent requirements for a program like Plutonium Disposition, which includes multiple projects as well as supporting activities. As a result, when developing the life-cycle cost estimate for the Plutonium Disposition Program, NNSA officials used an ad hoc approach to adapt DOE requirements for managing projects in DOE’s project management order. NNSA officials also said that its April 2013 life-cycle cost estimate did not include all the steps of a high-quality, reliable estimate in part because NNSA considered the estimate to be draft and therefore had not fully implemented plans for developing it. We recommended in this report that DOE conduct a root cause analysis of the Plutonium Disposition Program’s cost increases and ensure that future estimates of the program’s life-cycle cost and schedule for the program meet all best practices for reliable estimates. We also recommended that DOE revise its project management order to require life-cycle cost estimates covering the full cost of programs that include both construction projects and other efforts and activities not related to construction. DOE concurred with recommendations to analyze the programs cost increases and revise and update the program life-cycle cost estimate following best practices. DOE did not agree to update its project management order to require life-cycle cost estimates of programs. However, we continue to believe that this recommendation has merit and should be fully implemented.

NNSA took some actions to improve independent review and analysis of program estimates. For example, in April 2013, NNSA created the Office of Program Review and Analysis. According to NNSA, this office is intended to improve NNSA’s ability to plan and budget by providing senior leadership independent advice on resource allocations to ensure the best use of the agency’s resources, including evaluating cost estimates of NNSA projects and programs.

DOE and NNSA began taking some actions to improve cost estimating. For example, DOE OAPM has embarked on a Cost Estimating and Scheduling Initiative to systematically improve DOE’s policies and guidance. Some of the near term efforts include development of a Life- Cycle Cost handbook, a Key Performance Parameters and Statement of Work handbook, and Analysis of Alternatives guidance. In addition, NNSA’s Office of Defense Programs has taken steps to fill the existing gaps in the cost estimating framework for programs. For example, in 2011, Defense Programs established an Office of Cost Policy and Analysis to provide it with a cost analysis capability. In March 2014, NNSA’s Office of Program Integration issued a cost estimating improvement plan, which includes proposed guidance for conducting cost estimate briefings to the Office of Defense Programs Assistant Deputy Administrator, establishing a defense programs database, and implementing various process improvements to improve cost accounting and performance. While these efforts are intended to improve, among other things, cost estimating practices, none of these efforts include establishing requirements to follow cost estimating best practices.

Project Reviews Indicate Cost Estimating Weaknesses and Program Reviews Are Not Required and Therefore the Extent of Weaknesses Is Largely Unknown

NNSA’s project and program reviews for fiscal years 2009 to 2014 identified cost estimating weaknesses that can be attributed to not following best practices. DOE and NNSA require reviews of projects, including reviews of cost estimates at various CD points and at the discretion of project managers; however, because DOE and NNSA do not require reviews of program cost estimates, the extent of weaknesses in program cost estimates is largely unknown.

Project Reviews Identified Cost Estimating Weaknesses That Can Be Attributed to Not Following Best Practices

Of the 50 NNSA project reviews conducted from February 2009 through February 2014, 39 identified a total of 113 cost estimating weaknesses. We determined that 71 of the 113 weaknesses—or about 63 percent— can be attributed to not following 4 of the 12 best practice steps. These four steps are: determining the estimating structure, identifying ground rules and assumptions, conducting risk and uncertainty analysis, and documenting the estimate.

For example, as part of this review, we examined NNSA’s 2012 independent review of the Nuclear Material Safeguards and Security Upgrades Project at Los Alamos National Laboratory and it identified three weaknesses, one of which can be attributed to not following the best practice of updating the estimate to reflect actual costs and changes. In particular, the review found deficiencies in the estimate that could lead to an inaccurate cost and schedule estimate for the project. In another example, a 2012 review of NNSA’s Waste Solidification Building identified four weaknesses, including not following the best practice of conducting risk and uncertainty analysis. Several other reviews that we examined found similar weaknesses in project cost estimating. For example, the 2012 Independent Project Review of the MOX Fuel Fabrication Facility at the Savannah River Site identified flaws in the project’s cost estimate that include: (1) basic assumptions concerning project reserves and contingency cost; (2) data reliability of cost information considering the design maturity of the project; (3) the sensitivity analysis; and (4) the project risk analysis. The review also concluded that the most significant risk to the project’s success was NNSA’s under-estimation of the cost to complete the facility.

We also found that the remaining 42 weaknesses could be attributed to not following seven other best practice steps. These best practices are developing an estimating plan, defining program characteristics, developing a point estimate and comparing it to an ICE, conducting a sensitivity analysis, presenting the estimate to management for approval, and updating the estimate to reflect actual costs and changes.

Appendix II includes the cost estimating weaknesses we identified for each of the project reviews we assessed and the best practice step not followed to which the weakness can be attributed to not following.

Program Reviews Are Limited to LEPs So the Extent of Cost Estimating Weaknesses in NNSA Programs Is Largely Unknown

DOE and NNSA do not require reviews of programs, including reviews of program cost estimates. As a result, reviews of cost estimates for programs are limited, and the extent to which program cost estimates have weaknesses is largely unknown. While program reviews are not required, we identified and analyzed three LEP program reviews from the beginning of fiscal year 2009 through February 2014, two of which were reviews of the B61 LEP. These identified several weaknesses in the cost estimates for this program that can be attributed to not following three best practice steps: (1) obtaining data, (2) defining program characteristics, and (3) determining the estimating structure. For example, both reviews identified weaknesses in obtaining the data needed to compile the B61 cost estimate. In addition, one of the reviews noted that the estimate was not based on a sound program definition, while the other review stated that a standard WBS was not used to develop the B61 LEP cost estimate. Both reviews concluded that the B61 cost estimate is inaccurate, with one review noting that the program will cost approximately $3.6 billion more than NNSA’s 2011 $6.5 billion Weapon Design and Cost Report estimate.

While we did not identify any additional reviews of cost estimates for NNSA programs, we reported in February 2014 on the results of our review of the cost estimates for the Plutonium Disposition Program. We concluded that the life-cycle cost estimate for the overall program was not reliable and found that it did not fully reflect the characteristics of high- quality, reliable estimates as established by best practices. Specifically, in developing its April 2013 draft life-cycle cost estimate of $24.2 billion for the Plutonium Disposition Program, we found that NNSA followed several cost estimating best practices; including obtaining the data, defining the estimate’s purpose, and defining the program’s characteristics. However, we also found that NNSA did not follow other key steps such as conducting an independent cost estimate and, as a result, the estimate was not reliable. We recommended, among other things, that NNSA revise and update the program’s life-cycle estimate using cost estimating best practices, including conducting an independent cost estimate. NNSA agreed with this recommendation.

Without a requirement for conducting program reviews, NNSA does not have the appropriate internal controls necessary for assessing program performance. According to the Standards for Internal Control in the Federal Government, federal agencies are to employ internal control activities, such as functional reviews by management of projects and programs to compare actual performance to planned or expected results throughout the organization and analyze significant differences. Such reviews are to help ensure that management’s directives are carried out and to determine if agencies are effectively and efficiently using resources.

Conclusions

DOE and NNSA have taken action in recent years to improve cost- estimating practices, including the corrective actions that were implemented as part of the department’s 2008 root-cause analysis, which DOE later reported as having effectively mitigated most of the root causes of the most significant contract and project management challenges. Nonetheless, NNSA continues to struggle with significant cost overruns on its major projects. Because DOE does not require the use of most of the 12 cost estimating best practices for its projects and programs, it is unlikely that NNSA and its contractors will consistently develop reliable cost estimates. In addition, while DOE has developed a cost estimating guide, it does not fully describe all 12 cost estimating best practices. As a result, DOE and NNSA have not provided its contractors with all the detailed guidance needed to consistently develop reliable cost estimates. Also, because DOE Order 413.3B has not been updated since 2010, it omits reference to the supplemental cost estimating guide; users of the order may not be aware of the guide’s availability and may not benefit from its usefulness. Finally, without a requirement for conducting reviews of programs with project-like characteristics, including the life-cycle cost estimates of these programs, neither DOE nor NNSA have appropriate internal controls to assess the quality of program performance over time.

Recommendations for Executive Action

To enhance NNSA’s ability to develop reliable cost estimates for its projects and for its programs that have project-like characteristics, we recommend the Secretary of Energy take the following five actions:

Revise DOE’s project management order to require that DOE, NNSA, and its contractors develop cost estimates in accordance with the 12 cost estimating best practices.

Revise DOE’s cost estimating guide so that it fully reflects the 12 cost estimating best practices.

Revise DOE’s project management order to include references to the DOE cost estimating guide, where applicable.

Revise DOE directives that apply to programs to require that DOE and NNSA and its contractors develop cost estimates in accordance with the 12 cost estimating best practices, including developing life-cycle cost estimates for programs.

Revise DOE requirements and guidance that apply to programs to ensure that program reviews are conducted periodically, including reviews of the life-cycle cost estimates for programs.

Agency Comments

We provided DOE with a draft of this report for its review and comment. In its written comments, reproduced in appendix III, DOE agreed with the report’s recommendations. In regard to the report’s first recommendation—revise DOE’s project management order to require that DOE, NNSA, and its contractors develop cost estimates in accordance with the 12 cost estimating best practices—DOE stated in its written comments that DOE’s order for project management (DOE O 413.3B) will be assessed for revision following the issuance of the revision to DOE-STD-1189, Integration of Safety into the Design Process, currently scheduled for November 2016. DOE stated that it has established a log of issues to be addressed when revising Order 413.3B, which includes this recommendation. DOE also stated that this recommendation will be fully considered during this revision process. In the interim, DOE stated that its cost estimating guide incorporates the 12 cost estimating best practices, albeit not in the same format as our guidance. DOE also stated that it has begun internal efforts for the publication of a Departmental Cost Estimating and Schedule Policy and that the policy will complement existing cost estimating guidance and will incorporate our cost estimating guidance. DOE stated that the time frame for this policy to be issued is still to be determined. Further, DOE stated that DOE’s Office of Acquisition and Project Management will continue to incorporate the 12 cost estimating best practices in its independent cost estimating activities. Additionally, DOE explained that the curriculum of the Project Management Career Development Program requires cost and schedule estimating courses and that this training incorporates the 12 cost estimating best practices. We are pleased that DOE agreed with our recommendation and that it has interim measures to improve cost estimating before it plans to implement the recommendation. However, while these may be useful interim measures, the unspecified, open-ended date for updating the project management order that contains requirements (i.e., sometime after November 2016) and the statement that the cost estimating best practices will be considered, not incorporated, may indicate DOE’s lack of urgency or concern about the need to implement this recommendation.

In regard to the report’s second recommendation—revise DOE’s cost estimating guide so that it fully reflects the 12 cost estimating best practices—DOE stated in its written comments that it will begin updating the cost estimating guide in the first quarter of fiscal year 2015. The update efforts are to elevate the significance of the 12 cost estimating best practices within the content of the guide, although DOE did not specify when it might complete this update. We are pleased that DOE agreed with our recommendation and that it plans to take action during the current quarter of this fiscal year. However, we are concerned that in its written comments DOE did not specify whether it plans to revise the guide to better include the two areas in the guide we found were deficient and that the lack of a completion date may indicate DOE’s lack of urgency or concern about the need to implement this recommendation.

In regard to the report’s third recommendation—revise DOE’s project management order to include references to the DOE cost estimating guide, where applicable—DOE stated in its written comments that its project management order will be assessed for revision following the issuance of the revision to DOE-STD-1189, currently scheduled for November 2016, and that this recommendation will be fully considered during the order revision process. In the interim, DOE stated that its directives website points to the guides that accompany Order 413.3B as the best practices guidance for implementation of Order 413.3B requirements. We are pleased that DOE agreed with our recommendation and that its website includes links to the guides that accompany the project management order. However, our report noted the deficiencies associated with the existing cost estimating guide, and DOE did not specify plans for addressing those deficiencies. In addition, we are concerned that the unspecified, open-ended date for updating the project management order that contains requirements (i.e., sometime after November 2016) and the statement that referencing the cost estimating guide will be considered, not completed, may indicate DOE’s lack of urgency or concern about the need to implement this recommendation.

In regard to the report’s fourth recommendation—revise DOE directives that apply to programs to require that DOE and NNSA and its contractors develop cost estimates in accordance with the 12 cost estimating best practices, including developing lifecycle cost estimates for programs— DOE stated in its written comments that it is in the process of substantially revising the existing 1995 DOE Order 130.1, Budget Formulation, and that as part of this effort, DOE will assess the requirement for program cost estimates and will revise the order to provide more specificity on the cost estimating requirements. Further, DOE stated that the revised order will (1) define which DOE and NNSA program budget requests require cost estimates and (2) clarify that cost estimates for program budget submissions shall be conducted in accordance with the DOE cost estimating guide (or its successor policy). DOE estimated that it would complete this process in September 2016. We are pleased that DOE agreed with our recommendation.

In regard to the report’s fifth recommendation—revise DOE requirements and guidance that apply to programs to ensure that program reviews are conducted periodically, including reviews of the lifecycle cost estimates for programs—DOE again stated that it is in the process of substantially revising the existing DOE Order 130.1, and that as part of this effort, the department will assess requirements for program reviews and the linkage between program reviews and the budget formulation process. Further, DOE stated that the revised order will clarify requirements for program reviews and specify how such reviews—to include lifecycle cost estimates—can best support the budget formulation process. NNSA is to review these requirements and adjust NNSA-specific policies and guidance as appropriate. DOE estimated that it would complete this process in September 2016. We are pleased that DOE agreed with our recommendation.

We are sending copies of this report to the appropriate congressional committees; the Secretary of Energy; the Director, Office of Management and Budget; and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov.

If you or your staff members have any questions about this report, please contact me at (202) 512-3841 or trimbled@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix IV.

Appendix I: Comparison of DOE Cost Guide to GAO’s Best Practices for Cost Estimating

Appendix I: Comparison of DOE Cost Guide to GAO’s Best Practices for Cost Estimating 12 key steps/best practices Define estimate’s purpose, scope, and schedule.

Determine the estimating approach—work breakdown structure (WBS).

Identify the ground rules and assumptions.

Develop the point estimate and compare it to an independent cost estimate.

Update the estimate to reflect actual costs and changes.

Appendix II: Cost Estimating Weaknesses Identified in All Completed NNSA Project Reviews February 2009 - February 2014 and GAO’s Best Practices for Cost Estimating

GAO 12 steps (see table legend)

Legend: GAO 12 Best Practice Cost Estimating Steps

Step Description 1.

Summary of associated tasks Determine purpose, scope, required level of detail of estimate, as well as who will receive estimate.

Determine cost estimating team, schedule, and outline tasks in writing.

Identify technical characteristics of planned investment, quality of data needed, and plan for documenting and updating information.

Define the elements of the cost estimate, including best method for estimating costs and potential cross-checks, and standardized structure.

Define what the estimate will include and exclude, key assumptions (such as life-cycle of investment), schedule or budget constraints, and other elements that affect estimate. Assumptions should be measurable, specific, and consistent with historical data. Assumptions should be based on expert and technical judgment.

Create data collection plan, identify sources, collect valid and useful data, analyze data for cost drivers and other factors, and assess data for reliability and accuracy.

Develop a point estimate and compare it to an independent cost estimate.

Develop cost estimation model and calculate estimate, in constant dollars for investments that occur over multiple years, and other cross checks and validation, and compare estimate to an independent estimate and previous estimates. Update as more data are available.

Test the sensitivity of cost elements to changes in input values, ground rules, and assumptions.

Step Description 9.

Summary of associated tasks Determine which cost elements pose technical, cost, or schedule risks; analyze those risks; and recommend a plan to track and mitigate risks. A range of potential costs, based on risks and uncertainties, should be identified around a point estimate.

Document all steps used to develop the estimate so it can be recreated, describing methodology, data, assumptions, and results of risk, uncertainty, and sensitivity analysis.

Develop briefing on results, including information on estimation methods and risks, making content clear and complete so those unfamiliar with analysis can comprehend estimate and have confidence in it.

As technical aspects of project change, the complete cost estimate should be regularly updated and, as project moves forward, cost and schedule estimates should be tracked.

Appendix III: Comments from the Department of Energy

Appendix IV: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the individual named above, Daniel Feehan, Assistant Director; Antoinette Capaccio; Jennifer Echard; Mark Braza; Mike Meleady; Alison O’Neill; and Peter Ruedel made key contributions to this report.