Project and Program Management: DOE Needs to Revise Requirements and Guidance for Cost Estimating and Related Reviews
Released: 2014-11-25T00:00:00


Highlights: section_title: Why GAO Did This Study
paragraphs: NNSA is responsible for the nation's nuclear security programs. These programs often include the design and construction of large projects to meet program needs. NNSA has a history of struggling to complete these and other projects and programs within cost estimates.

Senate Report 112-73 mandated that GAO report on NNSA's cost estimating practices. This report examines: (1) the extent to which DOE and NNSA cost estimating requirements and guidance for projects and programs reflect best practices and (2) the extent to which recent NNSA project and program reviews identified cost estimating weaknesses, and the extent to which the weaknesses can be attributed to not following best practices. GAO reviewed DOE and NNSA cost estimating practices and compared them to best practices, NNSA project and program reviews, and two programs selected among the largest and most complex NNSA programs. GAO also interviewed DOE and NNSA officials about requirements and guidance for cost estimates.

section_title: What GAO Found
paragraphs: The Department of Energy's (DOE) and its National Nuclear Security Administration's (NNSA) cost estimating requirements and guidance for projects and programs do not fully reflect best practices for developing cost estimates. In regard to cost estimating requirements for projects, DOE's 2010 project management order requires 1 of the 12 cost estimating best practices—conducting an independent cost estimate—for larger projects at certain stages of development. In contrast, DOE's 2011 cost estimating guide describes recommended approaches for using 10 of 12 best practices and partially contains information about the other 2. Furthermore, because DOE's cost estimating guide was issued in 2011—after DOE's 2010 project order was issued—it is not referenced in the order. As a result, users of the order may not be aware of the guide's availability and may not benefit from its usefulness. In addition, although NNSA programs are required to follow DOE's budget formulation order and NNSA's budget process, both of which require the development of cost estimates, neither the order nor the process requires the use of best practices in developing the estimates. In February 2014, for example, GAO found that NNSA's lifecycle cost estimate for the Plutonium Disposition Program did not follow all key steps for developing high-quality cost estimates, in part because the agency did not have a requirement to develop a life-cycle cost estimate. In the absence of a requirement for using best practices, it is unlikely that DOE, NNSA, and their contractors will consistently develop reliable cost estimates.

NNSA's project and program reviews issued during fiscal year 2009 through March 2014 identified cost estimating weaknesses that can be attributed to not following best practices. DOE and NNSA require independent project reviews, including reviews of cost estimates at certain stages of development and at the discretion of project managers. Of the 50 reviews GAO analyzed, 39 identified a total of 113 cost estimating weaknesses. GAO determined that 71 of the 113 weaknesses—or about 63 percent—can be attributed to not following four best practices: (1) determining the estimating structure, (2) identifying ground rules and assumptions, (3) conducting risk and uncertainty analysis, and (4) documenting the estimate. Neither DOE nor NNSA, however, requires reviews of program cost estimates. Of the three program reviews conducted during fiscal years 2009 to 2013, two were of the B61 Life Extension Program, which is to extend the operational life of this nuclear weapon. Both reviews identified weaknesses in the cost estimates that can be attributed to not following three best practices: (1) determining the estimating structure, (2) defining program characteristics, and (3) obtaining data. In addition, a February 2014 GAO report on NNSA's program to dispose of weapons-grade plutonium found that NNSA did not follow several cost estimating best practices, such as conducting an independent cost estimate and as a result, the program cost estimate was not reliable. While the program reviews and GAO's February 2014 report indicate weaknesses in a few program cost estimates, the extent of program cost estimate weaknesses is largely unknown because neither DOE nor NNSA requires reviews of program cost estimates. Without a requirement for conducting independent program reviews, NNSA does not have the internal control necessary for assessing program performance over time.

section_title: What GAO Recommends
paragraphs: GAO recommends that DOE, among other things, revise DOE requirements and guidance that apply to projects and programs to ensure: (1) the guidance is referenced in the project management order, (2) DOE and NNSA and its contractors develop cost estimates in accordance with the 12 best practices, (3) its cost estimating guide fully reflects best practices, and (4) independent reviews of programs are conducted periodically. DOE agreed with the report and its recommendations.

section_title: Letter
paragraphs: The National Nuclear Security Administration (NNSA)—a  semiautonomous agency within the Department of Energy (DOE)—is  responsible for managing the nation’s nuclear security missions. These  missions include maintaining the safety, security, and effectiveness of the  U.S. nuclear weapons stockpile and reducing the threat posed by nuclear  proliferation. To achieve its missions, NNSA manages numerous  programs such as the Stockpile Life Extension Program and the  Plutonium Disposition Program. These and other programs frequently  rely on the acquisition of capital assets to meet program needs. For  example, to meet the needs of the Plutonium Disposition Program, NNSA  is currently managing the construction of two capital asset projects—the  Mixed Oxide (MOX) Fuel Fabrication Facility and the Waste Solidification   NNSA has a  Building—at DOE’s Savannah River Site in South Carolina.history of struggling to complete these and other projects within their  initial cost and schedule estimates. Since 1990, DOE’s management of  major contracts and projects, including those executed by NNSA, has  been on our list of areas at high risk for fraud, waste, abuse, and  mismanagement. We have also recently reported that cost estimates for  several NNSA projects were inaccurate, and significant cost increases  were likely. For example, in 2014, we reported that the estimated cost of  the MOX facility had increased from $4.8 billion in April 2007 to $7.7  billion in 2012.DOE and NNSA conduct an analysis of the root causes of the increase  and identify and prioritize recommended solutions. NNSA concurred with  this recommendation and stated that it was planning to conduct such an  analysis.

NNSA’s other program offices include Emergency Operations, Defense Nuclear Security,  Counterterrorism and Counterproliferation, Infrastructure and Environment, Management  and Administration, and the Office of the Administrator.

Science Campaign and the Directed Stockpile Work program. The  second category consists of programs that have defined start and end  points and a total cost that includes all costs incurred to complete the  program. Programs in this category include the B61 Life Extension  Program (LEP) and the Plutonium Disposition Program. Programs in this  second category are the programs discussed in this report.

In April 2008, to assess the underlying causes of its contract and project  management challenges, DOE conducted a root-cause analysis, which  involved identifying the most significant issues that were impeding the  department’s ability to complete projects within their cost, scope, and  schedule estimates. The analysis identified an inadequate cost estimating  capability as one of the top five reasons that DOE was unable to  successfully complete these projects. The analysis also identified DOE’s  lack of cost estimating policy or standards, personnel without the  appropriate skills, and databases that lack current or historical information  as root causes contributing to cost estimating problems. In July 2008,  DOE published a corrective action plan to address the deficiencies  identified in the root-cause analysis. The plan included actions to address  cost estimating issues by establishing “a federal independent government  cost estimating capability.” In line with these efforts, DOE and NNSA have  undertaken a number of activities to improve their cost estimating  practices, including revising DOE’s project management order, issuing a  supplemental cost estimating guide, and strengthening training courses  and certification requirements. In February 2011, DOE issued a report  documenting the actions taken and progress made to implement  corrective actions. In the report, DOE stated that the actions taken  effectively mitigated most of the root causes of their most significant  contract and project management challenges and decreased the inherent  risks involved with managing large, technically complex projects.

The Senate Report accompanying a proposed bill for the National  Defense Authorization Act for Fiscal Year 2013 mandated that GAO  report on NNSA’s cost estimating practices and provide  recommendations, if appropriate, to help NNSA conform to cost  estimating best practices for projects and programs. This report  examines: (1) the extent to which DOE and NNSA cost estimating  requirements and guidance for its projects and programs reflect best  practices for developing and reviewing cost estimates, and (2) the extent  to which NNSA project and program reviews identified weaknesses in  cost estimates, and the extent to which any weaknesses can be attributed  to not following cost estimating best practices. In May 2013, we testified  before the U.S. Senate Subcommittee on Strategic Forces, Committee on  Armed Services, relying on prior work and on preliminary observations  from our work in this area.

To examine the extent to which DOE and NNSA cost estimating  requirements and guidance for its projects and programs reflect best  practices for developing and reviewing cost estimates, we reviewed DOE  and NNSA requirements and guidance related to cost estimating for  projects and programs and compared them with the best practices  identified in our 2009 Cost Estimating and Assessment Guide. This guide is a compilation of cost estimating best practices drawn from across  industry and government. We also interviewed DOE and NNSA officials at  headquarters and NNSA project and program managers and their  contractors at NNSA’s Los Alamos and Sandia National Laboratories in  New Mexico to discuss the requirements and guidance they follow in  preparing, documenting, and reviewing project and program cost  estimates. These NNSA sites were chosen because both sites have  recently completed or have ongoing projects and are involved in the  execution of several NNSA programs. We focused on projects and  programs managed by NNSA’s Offices of Defense Programs and  Defense Nuclear Nonproliferation because they manage a diverse array  of programs and manage NNSA’s largest ongoing capital asset projects— these two offices account for approximately 85 percent of NNSA’s fiscal  year 2015 budget request. We also analyzed NNSA program-specific  cost-estimating policies and guidance for the B61 LEP. We focused on  the B61 LEP and the Plutonium Disposition Program because they  represent two of the largest and most complex NNSA programs and have  defined start and end points and a total cost that includes all costs  incurred to complete the program.

To determine the extent to which recent NNSA project and program  reviews identified any weaknesses associated with the cost estimates for  projects and programs and whether any of these weaknesses can be  attributed to not following cost estimating best practices, we obtained  project and program reviews completed from February 2009 through  February 28, 2014. We limited our assessment to those reviews  conducted since February 2009 to coincide with NNSA’s issuance of a  requirement in February 2009 to conduct independent cost estimates  (ICE) or independent reviews for certain projects. The reviews included  independent cost assessments, independent project reviews, peer  reviews, and external independent reviews conducted primarily by DOE’s  Office of Acquisition and Project Management (OAPM), NNSA’s Office of  Acquisition and Project Management, and U.S. Army Corps of Engineers, and other subject matter experts including former federal contractors.We obtained and assessed a total of 50 reviews of 21 projects and the  results from two program reviews. We analyzed these reviews to  determine whether they included any references to weaknesses in the  cost estimates associated with these projects and programs, and we  determined whether the weaknesses could be attributed to not following  specific cost estimating best practices. When a weakness was noted, we  determined whether the weakness fit into any of the 12 best practices as  described in the Cost Estimating and Assessment Guide. As a measure  to collect reliable data, two GAO analysts reviewed each report  independently and compared their responses.

We conducted this performance audit from November 2012 to November  2014, in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: This section describes: (1) NNSA organization and management, (2)  DOE and NNSA cost estimating requirements and guidance for projects,  (3) DOE and NNSA cost estimating requirements and guidance for  programs, (4) our 2009 cost estimating best practices, and (5) federal  standards for internal controls.
subsections: section_title: NNSA Organization and Management
paragraphs: To fund NNSA’s projects and programs, the President requested  approximately $11.7 billion for NNSA in the fiscal year 2015 budget  submission to Congress, the majority of which is intended to fund  program operations. Work activities on both projects and programs are  largely carried out by management and operating (M&O) contractors at  NNSA’s eight government-owned, contractor-operated sites—collectively  referred to as the nuclear security enterprise. Among other things, these  contractors operate and maintain the government-owned facilities and  infrastructure deemed necessary to support the nuclear security  enterprise and to support the capabilities to conduct scientific, technical,  engineering, and production activities.

As shown in figure 1, the nuclear security enterprise sites include national  research and development laboratories, as well as production plants.  Specifically, NNSA manages three national nuclear weapons design  laboratories—Lawrence Livermore National Laboratory in California, Los  Alamos National Laboratory in New Mexico, and Sandia National  Laboratories in New Mexico and California. It also manages four nuclear  weapons production plants—the Pantex Plant in Texas, the Y-12 National  Security Complex in Tennessee, the Kansas City Plant in Missouri, and  Tritium Operations at DOE’s Savannah River Site in South Carolina.  NNSA also manages the Nevada National Security Site, formerly known  as the Nevada Test Site. Various headquarters organizations within  NNSA develop policies and NNSA site offices, collocated with NNSA’s  sites, conduct day-to-day oversight of the M&O contractors, and evaluate  the M&O contractors’ performance in carrying out the sites’ missions.

The Secretary of Energy is authorized under the National Nuclear  Security Administration Act to establish policy and provide direction to  NNSA.authority to establish NNSA-specific policies, unless disapproved by the   The NNSA Administrator, however, is also vested with the  Secretary of Energy. NNSA does this through the issuance of Policy  Letters. According to NNSA officials responsible for project  management, DOE directives and orders and NNSA Policy Letters,  including business operating procedures, collectively establish  requirements for the development of NNSA project and program cost  estimates, and we refer to them as such throughout this document.

section_title: DOE and NNSA Cost Estimating Requirements and Guidance for Projects
paragraphs: NNSA is required to manage its projects, including the development of  cost estimates, in accordance with DOE Order 413.3B. The purpose of  the order is to provide program and project management direction for the  acquisition of capital assets, and it includes requirements for developing  project cost estimates. The order also provides management direction for  NNSA and other DOE offices with the goal of delivering projects within  the original performance baseline that are fully capable of meeting  mission performance and other requirements such as environmental,  safety, and health standards.or critical decision points—that span the life of a project. These critical  decision (CD) points are:   The order defines five major milestones—

CD-0: Approve mission need.

CD-1: Approve alternative selection and cost range.

CD-2: Approve project performance baseline.

CD-3: Approve start of construction.

CD-4: Approve start of operations or project completion.

The order specifies requirements that must be met, along with the  documentation necessary, to move a project past each CD point. In  addition, the order requires senior management to review the supporting  documentation and decide whether to approve the project at each CD.

DOE also provides suggested approaches for meeting the requirements  contained in Order 413.3B through a series of guides, such as guides for  project reviews and risk management.

The order includes several cost estimating requirements for projects.  Specifically, the order requires: a cost estimate be provided at each CD point;  the degree of rigor and detail for a cost estimate be carefully defined,  depending on the degree of confidence in project scale and scope  that is reasonable to expect at that stage; a cost estimate range for CD-0 and 1 explicitly noting any relevant  caveats concerning uncertainties; and a program sponsor never be the sole cost estimator at any critical  decision given the inherent conflict of interest, and a second cost  estimator should come from outside of the line manager’s chain of  command to avoid conflict of interest.

The order also establishes requirements for the review of cost estimates  for certain projects at CD-0 through CD-3, depending on the estimated  cost of the project. Table 1 shows these requirements and an NNSA  requirement for review.

As a supplement to Order 413.3B, DOE provides guidance for the  development of cost estimates in its Cost Estimating Guide, which was  issued in May 2011. According to the guide, its purpose is to provide  uniform guidance and best practices that describe recommended  methods and procedures that projects and programs could use for  preparing cost estimates. The guide states that it is based on accepted  industry practices and processes, including practices outlined in our 2009  Cost Estimating and Assessment Guide, to meet federal and DOE  requirements.

section_title: DOE and NNSA Cost Estimating Requirements and Guidance for Programs
paragraphs: DOE and NNSA cost estimating requirements and guidance for programs  are primarily defined in four documents.

DOE Order 130.1. This order requires all DOE and NNSA program  budget requests be based on cost estimates that have been  thoroughly reviewed and deemed reasonable prior to their inclusion in  department budgets.

DOE Cost Estimating Guide 413.3-21. DOE’s cost estimating guide is  primarily intended for use in managing the phases involved in the  acquisition of a capital asset project. However, the guide states that it  also could be used by DOE and NNSA programs in preparing cost  estimates.

NNSA Planning, Programming, Budgeting, and Evaluation (PPBE)

Requirements. NNSA’s PPBE process provides a framework for the  agency to plan, prioritize, fund, and evaluate its program activities.  This process includes a budget validation step to review the process  contractors follow in developing program cost estimates for budgeting  purposes.

NNSA Phase 6.X Process. NNSA manages life extension programs,  including the B61 LEP, according to a process called Phase 6.X that  was developed by DOE in collaboration with the Department of  Defense. The Phase 6.X process provides NNSA with a framework to conduct and manage LEP activities, including the development and  documentation of a program cost estimate.

section_title: GAO Cost Estimating Best Practices
paragraphs: Drawing from federal cost estimating organizations and industry, our cost  estimating guide provides best practices about the processes,  procedures, and practices needed for ensuring development of high- quality—that is, reliable cost estimates. A high-quality cost estimate  helps ensure that management is given the information it needs to make  informed decisions. The guide identifies the following four characteristics  of a high-quality cost estimate. Specifically, such an estimate is:

Credible when it has been cross-checked with an ICE, the level of  confidence associated with the point estimate has been identified  through the use of risk and uncertainty analysis, and a sensitivity  analysis has been conducted;

Well-documented when supporting documentation is accompanied  by a narrative explaining the process, sources, and methods used to  create the estimate and contains the underlying data used to develop  the estimate;

Accurate when it is not overly conservative or too optimistic and  based on an assessment of the costs most likely to be incurred; and

Comprehensive when it accounts for all possible costs associated  with a project, it contains a cost estimating structure in sufficient detail  to ensure that costs are neither omitted nor double-counted, and the  estimating teams’ composition is commensurate with the assignment.

To develop a cost estimate that embodies these four characteristics, our  cost estimating guide lays out 12 best practice steps. For example, one  step—determining the estimating structure—includes the need to develop  a “product-oriented” work breakdown structure (WBS) that reflects the  requirements and basis for identifying resources and tasks necessary to  accomplish the project’s objectives. A product-oriented WBS is organized  to reflect the cost, schedule, and technical performance of project  components. Such a WBS allows a project to track cost by defined  deliverables, promote accountability by identifying work products that are  independent of one another, and provides a basis for identifying  resources and tasks for developing a cost estimate. Table 2 shows the  four characteristics of a high-quality cost estimate with the corresponding  steps.

section_title: Federal Standards for Internal Controls
paragraphs: In addition to the requirements for cost estimating established by DOE  and NNSA, according to Standards for Internal Control in the Federal  Government, federal agencies are to employ internal control activities,  such as functional reviews by management of projects and programs to  compare actual performance to planned or expected results throughout  the organization and analyze significant differences. Such reviews, which  may include reviews of program cost estimates, are to help ensure that  management’s directives are carried out and to determine if agencies are  effectively and efficiently using resources.

section_title: DOE and NNSA Cost Estimating Requirements and Guidance Generally Do Not Reflect Best Practices for Developing Cost Estimates
paragraphs: DOE and NNSA cost estimating requirements and guidance for projects  and programs generally do not reflect best practices for developing cost  estimates. For projects, DOE requires one best practice, and DOE’s cost  estimating guide does not fully reflect cost estimating best practices, and  is not mandatory. For programs, DOE does not require the use of any of  the best practices to develop cost estimates, and NNSA has taken steps  to close gaps in the existing cost estimating framework for programs,  such as creating a cost analysis office, but these steps have not included  a requirement to use best practices.
subsections: section_title: Projects Are Not Required to Meet Most Best Practice Steps, and Guidance Describes Most Best Practice Steps but Is Not Mandatory
paragraphs: DOE and NNSA cost estimating requirements and guidance for projects  generally do not reflect best practices for developing cost estimates.  DOE’s 2010 project management order requires the use of only 1 of the  12 cost estimating best practice steps. Specifically, the order requires an  ICE be prepared at CD-2 and CD-3 for projects with an estimated cost of  $100 million or greater. The order requires the development of an ICE at  CD-3 if warranted by risk and performance indicators or as designated by  DOE or NNSA management. In addition, NNSA’s 2014 requirement for an  ICE or independent cost review could subject additional projects with an  estimate of a cost of less than $100 million to an ICE, but this would  depend on whether NNSA chooses to conduct an ICE rather than the less  rigorous independent cost review. None of the other cost estimating  requirements in the order, such as the need for a cost estimate at each  CD point, ensure that project cost estimates will be prepared in  accordance with cost estimating best practices. For example, the order  does not require any of the other 11 best practice steps such as  conducting a risk and uncertainty analysis, identifying ground rules and  assumptions, documenting the estimate, developing a point estimate, or  determining the estimating structure. According to the DOE officials  responsible for developing DOE’s project management order, DOE has  chosen to not require all cost estimating best practices in the order and  instead includes suggested approaches for developing cost estimates in  the DOE cost estimating guide that accompanies the order. However,  because neither DOE nor NNSA requires the use of most cost estimating  best practices for its projects, it is unlikely that NNSA and its contractors  will consistently develop reliable cost estimates.

DOE’s 2011 cost estimating guide describes most best practices, but it is  not mandatory and it is not referenced in the order. We found that the  guide fully or substantially describes 10 of the 12 best practices.  However, the guide only partially or minimally contains information about  the other 2 best practice steps—determining the estimating structure and  conducting a sensitivity analysis. For the estimating structure, the guide  only partially describes how to develop this structure, which primarily  involves establishing an adequate WBS. For example, we found that the  guide describes the need for a WBS, but it does not describe the need for  a product-oriented WBS and does not provide enough information on how  to create a WBS. For the sensitivity analysis, the guide only minimally  describes the information needed. For example, the guide suggests  briefing management on the results of a sensitivity analysis, but it does  not address the best practice steps on how to perform a sensitivity  analysis. As a result, DOE and NNSA have not provided its contractors  with all the detailed guidance needed to consistently develop reliable cost  estimates. When we discussed our analysis of this guide and the  shortcomings we found in regard to the guide with the DOE officials  responsible for developing the guide, they said that they would review our  analysis and consider revisions to the guide as appropriate. Appendix I  provides a comparison of the extent to which DOE’s cost guide contains  information on the 12 best practice steps.

In addition to the shortcomings in the description for two of the best  practices, the use of the guide is not mandatory and, as a result, neither  NNSA staff nor its contractors who develop cost estimates are required to  use it. The guide states that it does not impose any new cost estimating  requirements or establish departmental policy; it describes only  suggested recommended approaches for preparing cost estimates.  Further, DOE’s cost estimating guide is not referenced in the order. To  supplement DOE’s project management order, DOE has issued a series  of guides that provide suggested approaches to meeting the  requirements, some of which are specified in the order. There are a total  of 18 separate guides, including guides that provide suggested  approaches on topics such as systems engineering, earned value  management systems, and project closeout. The order references some  of these guides as appropriate so that the user of the order is made  aware of the existence of the suggested approaches that are included in  the guides. However, because the cost guide was issued in 2011, a year  after DOE’s latest version of the order was issued, the order contains no  reference to this guide that a preparer of a cost estimate could use in  developing an estimate, or that a reviewer could use in reviewing one,  and users of the order may not be aware of the guide’s availability and  may not benefit from its usefulness.

section_title: Programs Are Not Required to Meet Any Cost Estimating Best Practices
paragraphs: DOE and NNSA programs are not required to meet any cost estimating  best practices. NNSA officials explained that NNSA cost estimating  practices for programs are limited, de-centralized, and inconsistent, and  are not governed by a cost estimating policy or single set of NNSA  requirements and guidance. According to these officials, each NNSA  program office uses different practices and procedures for the  development of cost estimates that are included in the NNSA annual  budget. In addition, these officials also explained that, while there are no  specific requirements on how program cost estimates should be  developed or reviewed, all NNSA programs must follow DOE’s budget  formulation order and NNSA’s PPBE process, but the order does not  require, and the PPBE process does not direct the use of best practices  in developing cost estimates. In 2012, we reported on NNSA’s PPBE  process, and we noted significant deficiencies in NNSA’s implementation  of its PPBE process.ensure the development of fully credible program cost estimates, and that  the review of the cost estimates supporting DOE’s budget was not  sufficiently thorough to ensure the credibility and reliability of NNSA’s  budget estimates. We also found that NNSA was not following DOE’s  budget formulation order that requires that budget requests be based on   Specifically, we found that the process does not  cost estimates that have been thoroughly reviewed and deemed  reasonable by the cognizant program organization. NNSA officials told us  that they do not follow DOE’s budget formulation order because it expired  in 2003. Additionally, according to NNSA officials, NNSA’s trust in its  contractors minimizes the need for formal review of its budget estimates.

In our 2012 report on the NNSA’s PPBE process we recommended that,  among other things, DOE update the departmental order for budget  reviews, improve the formal process for reviewing budget estimates, and  reinstitute an independent analytical capability to provide senior decision  makers with independent program reviews. The agency agreed in  principle with these recommendations, but it has not taken action to fully  implement them. According to NNSA officials, as of August 2014, NNSA  has suspended the budget review process and several options for  replacing this process are under review.

In the absence of a DOE or NNSA requirement for programs to follow  cost estimating best practices, we found that managers for the programs  we reviewed—the B61 LEP and the Plutonium Disposition Program— used different processes in developing cost estimates for these programs.  Specifically:

B61 LEP. NNSA managers for the B61 LEP use the Phase 6.X  Process which, among other things, involves the direction to establish  a cost and schedule baseline for the program. However, according to  the NNSA B61 LEP project manager, since the Phase 6.X Process  does not define how cost estimates should be developed, the B61  management team developed an approach for developing cost  estimates for the program using various sources, including direction  under the Phase 6.X Process, as well as DOE’s project management  order and cost guide and our cost-estimating guide. This effort  resulted in the management team producing a document that defines  the strategy and provides guidance for completing the cost estimate  for the B61 LEP. We reviewed this document and found that it does  not stipulate that NNSA program managers or its contractors must  follow any DOE or NNSA requirements or guidance for the  development of a program cost estimate when developing the  estimate for the B61 LEP.

Plutonium Disposition Program. In February 2014, we found that  NNSA’s life-cycle cost estimate for the Plutonium Disposition Program  did not follow all key steps for developing high-quality cost estimates  in part because it did not have a requirement to develop a life-cycle  cost estimate. According to NNSA officials, DOE’s project  management order includes requirements for development of cost  and schedule estimates for a capital asset project, such as the MOX  facility or Waste Solidification Building, but does not specify equivalent  requirements for a program like Plutonium Disposition, which includes  multiple projects as well as supporting activities. As a result, when  developing the life-cycle cost estimate for the Plutonium Disposition  Program, NNSA officials used an ad hoc approach to adapt DOE  requirements for managing projects in DOE’s project management  order. NNSA officials also said that its April 2013 life-cycle cost  estimate did not include all the steps of a high-quality, reliable  estimate in part because NNSA considered the estimate to be draft  and therefore had not fully implemented plans for developing it. We  recommended in this report that DOE conduct a root cause analysis  of the Plutonium Disposition Program’s cost increases and ensure  that future estimates of the program’s life-cycle cost and schedule for  the program meet all best practices for reliable estimates. We also  recommended that DOE revise its project management order to  require life-cycle cost estimates covering the full cost of programs that  include both construction projects and other efforts and activities not  related to construction. DOE concurred with recommendations to  analyze the programs cost increases and revise and update the  program life-cycle cost estimate following best practices. DOE did not  agree to update its project management order to require life-cycle cost  estimates of programs. However, we continue to believe that this  recommendation has merit and should be fully implemented.

NNSA took some actions to improve independent review and analysis of  program estimates. For example, in April 2013, NNSA created the Office  of Program Review and Analysis. According to NNSA, this office is  intended to improve NNSA’s ability to plan and budget by providing senior  leadership independent advice on resource allocations to ensure the best  use of the agency’s resources, including evaluating cost estimates of  NNSA projects and programs.

DOE and NNSA began taking some actions to improve cost estimating.  For example, DOE OAPM has embarked on a Cost Estimating and  Scheduling Initiative to systematically improve DOE’s policies and  guidance. Some of the near term efforts include development of a Life- Cycle Cost handbook, a Key Performance Parameters and Statement of  Work handbook, and Analysis of Alternatives guidance. In addition,  NNSA’s Office of Defense Programs has taken steps to fill the existing  gaps in the cost estimating framework for programs. For example, in  2011, Defense Programs established an Office of Cost Policy and  Analysis to provide it with a cost analysis capability. In March 2014,  NNSA’s Office of Program Integration issued a cost estimating  improvement plan, which includes proposed guidance for conducting cost  estimate briefings to the Office of Defense Programs Assistant Deputy  Administrator, establishing a defense programs database, and  implementing various process improvements to improve cost accounting  and performance. While these efforts are intended to improve, among  other things, cost estimating practices, none of these efforts include  establishing requirements to follow cost estimating best practices.

section_title: Project Reviews Indicate Cost Estimating Weaknesses and Program Reviews Are Not Required and Therefore the Extent of Weaknesses Is Largely Unknown
paragraphs: NNSA’s project and program reviews for fiscal years 2009 to 2014  identified cost estimating weaknesses that can be attributed to not  following best practices. DOE and NNSA require reviews of projects,  including reviews of cost estimates at various CD points and at the  discretion of project managers; however, because DOE and NNSA do not  require reviews of program cost estimates, the extent of weaknesses in  program cost estimates is largely unknown.
subsections: section_title: Project Reviews Identified Cost Estimating Weaknesses That Can Be Attributed to Not Following Best Practices
paragraphs: Of the 50 NNSA project reviews conducted from February 2009 through  February 2014, 39 identified a total of 113 cost estimating weaknesses.  We determined that 71 of the 113 weaknesses—or about 63 percent— can be attributed to not following 4 of the 12 best practice steps. These  four steps are: determining the estimating structure,  identifying ground rules and assumptions,  conducting risk and uncertainty analysis, and documenting the estimate.

For example, as part of this review, we examined NNSA’s 2012  independent review of the Nuclear Material Safeguards and Security  Upgrades Project at Los Alamos National Laboratory and it identified  three weaknesses, one of which can be attributed to not following the  best practice of updating the estimate to reflect actual costs and changes.  In particular, the review found deficiencies in the estimate that could lead  to an inaccurate cost and schedule estimate for the project. In another  example, a 2012 review of NNSA’s Waste Solidification Building identified  four weaknesses, including not following the best practice of conducting  risk and uncertainty analysis. Several other reviews that we examined  found similar weaknesses in project cost estimating. For example, the  2012 Independent Project Review of the MOX Fuel Fabrication Facility at  the Savannah River Site identified flaws in the project’s cost estimate that  include: (1) basic assumptions concerning project reserves and  contingency cost; (2) data reliability of cost information considering the  design maturity of the project; (3) the sensitivity analysis; and (4) the  project risk analysis. The review also concluded that the most significant  risk to the project’s success was NNSA’s under-estimation of the cost to  complete the facility.

We also found that the remaining 42 weaknesses could be attributed to  not following seven other best practice steps. These best practices are developing an estimating plan, defining program characteristics, developing a point estimate and comparing it to an ICE,  conducting a sensitivity analysis, presenting the estimate to management for approval, and updating the estimate to reflect actual costs and changes.

Appendix II includes the cost estimating weaknesses we identified for  each of the project reviews we assessed and the best practice step not  followed to which the weakness can be attributed to not following.

section_title: Program Reviews Are Limited to LEPs So the Extent of Cost Estimating Weaknesses in NNSA Programs Is Largely Unknown
paragraphs: DOE and NNSA do not require reviews of programs, including reviews of  program cost estimates. As a result, reviews of cost estimates for  programs are limited, and the extent to which program cost estimates  have weaknesses is largely unknown. While program reviews are not  required, we identified and analyzed three LEP program reviews from the  beginning of fiscal year 2009 through February 2014, two of which were  reviews of the B61 LEP. These identified several weaknesses in the cost  estimates for this program that can be attributed to not following three  best practice steps: (1) obtaining data, (2) defining program  characteristics, and (3) determining the estimating structure. For example,  both reviews identified weaknesses in obtaining the data needed to  compile the B61 cost estimate. In addition, one of the reviews noted that  the estimate was not based on a sound program definition, while the  other review stated that a standard WBS was not used to develop the  B61 LEP cost estimate. Both reviews concluded that the B61 cost  estimate is inaccurate, with one review noting that the program will cost  approximately $3.6 billion more than NNSA’s 2011 $6.5 billion Weapon  Design and Cost Report estimate.

While we did not identify any additional reviews of cost estimates for  NNSA programs, we reported in February 2014 on the results of our  review of the cost estimates for the Plutonium Disposition Program. We  concluded that the life-cycle cost estimate for the overall program was not  reliable and found that it did not fully reflect the characteristics of high- quality, reliable estimates as established by best practices. Specifically, in  developing its April 2013 draft life-cycle cost estimate of $24.2 billion for  the Plutonium Disposition Program, we found that NNSA followed several  cost estimating best practices; including obtaining the data, defining the  estimate’s purpose, and defining the program’s characteristics. However,  we also found that NNSA did not follow other key steps such as  conducting an independent cost estimate and, as a result, the estimate  was not reliable. We recommended, among other things, that NNSA  revise and update the program’s life-cycle estimate using cost estimating  best practices, including conducting an independent cost estimate. NNSA  agreed with this recommendation.

Without a requirement for conducting program reviews, NNSA does not  have the appropriate internal controls necessary for assessing program  performance. According to the Standards for Internal Control in the  Federal Government, federal agencies are to employ internal control  activities, such as functional reviews by management of projects and  programs to compare actual performance to planned or expected results  throughout the organization and analyze significant differences. Such  reviews are to help ensure that management’s directives are carried out  and to determine if agencies are effectively and efficiently using  resources.

section_title: Conclusions
paragraphs: DOE and NNSA have taken action in recent years to improve cost- estimating practices, including the corrective actions that were  implemented as part of the department’s 2008 root-cause analysis, which  DOE later reported as having effectively mitigated most of the root causes  of the most significant contract and project management challenges.  Nonetheless, NNSA continues to struggle with significant cost overruns  on its major projects. Because DOE does not require the use of most of  the 12 cost estimating best practices for its projects and programs, it is  unlikely that NNSA and its contractors will consistently develop reliable  cost estimates. In addition, while DOE has developed a cost estimating  guide, it does not fully describe all 12 cost estimating best practices. As a  result, DOE and NNSA have not provided its contractors with all the  detailed guidance needed to consistently develop reliable cost estimates.  Also, because DOE Order 413.3B has not been updated since 2010, it  omits reference to the supplemental cost estimating guide; users of the  order may not be aware of the guide’s availability and may not benefit  from its usefulness. Finally, without a requirement for conducting reviews  of programs with project-like characteristics, including the life-cycle cost  estimates of these programs, neither DOE nor NNSA have appropriate  internal controls to assess the quality of program performance over time.

section_title: Recommendations for Executive Action
paragraphs: To enhance NNSA’s ability to develop reliable cost estimates for its  projects and for its programs that have project-like characteristics, we  recommend the Secretary of Energy take the following five actions:

Revise DOE’s project management order to require that DOE, NNSA,  and its contractors develop cost estimates in accordance with the 12  cost estimating best practices.

Revise DOE’s cost estimating guide so that it fully reflects the 12 cost  estimating best practices.

Revise DOE’s project management order to include references to the  DOE cost estimating guide, where applicable.

Revise DOE directives that apply to programs to require that DOE and  NNSA and its contractors develop cost estimates in accordance with  the 12 cost estimating best practices, including developing life-cycle  cost estimates for programs.

Revise DOE requirements and guidance that apply to programs to  ensure that program reviews are conducted periodically, including  reviews of the life-cycle cost estimates for programs.

section_title: Agency Comments
paragraphs: We provided DOE with a draft of this report for its review and comment. In  its written comments, reproduced in appendix III, DOE agreed with the  report’s recommendations. In regard to the report’s first  recommendation—revise DOE’s project management order to require  that DOE, NNSA, and its contractors develop cost estimates in  accordance with the 12 cost estimating best practices—DOE stated in its  written comments that DOE’s order for project management (DOE O  413.3B) will be assessed for revision following the issuance of the  revision to DOE-STD-1189, Integration of Safety into the Design Process,  currently scheduled for November 2016. DOE stated that it has  established a log of issues to be addressed when revising Order 413.3B,  which includes this recommendation. DOE also stated that this  recommendation will be fully considered during this revision process. In  the interim, DOE stated that its cost estimating guide incorporates the 12  cost estimating best practices, albeit not in the same format as our  guidance. DOE also stated that it has begun internal efforts for the  publication of a Departmental Cost Estimating and Schedule Policy and  that the policy will complement existing cost estimating guidance and will  incorporate our cost estimating guidance. DOE stated that the time frame  for this policy to be issued is still to be determined.  Further, DOE stated  that DOE’s Office of Acquisition and Project Management will continue to  incorporate the 12 cost estimating best practices in its independent cost  estimating activities. Additionally, DOE explained that the curriculum of  the Project Management Career Development Program requires cost and  schedule estimating courses and that this training incorporates the 12  cost estimating best practices.  We are pleased that DOE agreed with our  recommendation and that it has interim measures to improve cost  estimating before it plans to implement the recommendation.  However,  while these may be useful interim measures, the unspecified, open-ended  date for updating the project management order that contains  requirements (i.e., sometime after November 2016) and the statement  that the cost estimating best practices will be considered, not  incorporated, may indicate DOE’s lack of urgency or concern about the  need to implement this recommendation.

In regard to the report’s second recommendation—revise DOE’s cost  estimating guide so that it fully reflects the 12 cost estimating best  practices—DOE stated in its written comments that it will begin updating  the cost estimating guide in the first quarter of fiscal year 2015. The  update efforts are to elevate the significance of the 12 cost estimating  best practices within the content of the guide, although DOE did not  specify when it might complete this update.  We are pleased that DOE  agreed with our recommendation and that it plans to take action during  the current quarter of this fiscal year. However, we are concerned that in  its written comments DOE did not specify whether it plans to revise the  guide to better include the two areas in the guide we found were deficient  and that the lack of a completion date may indicate DOE’s lack of urgency  or concern about the need to implement this recommendation.

In regard to the report’s third recommendation—revise DOE’s project  management order to include references to the DOE cost estimating  guide, where applicable—DOE stated in its written comments that its  project management order will be assessed for revision following the  issuance of the revision to DOE-STD-1189, currently scheduled for  November 2016, and that this recommendation will be fully considered  during the order revision process. In the interim, DOE stated that its  directives website points to the guides that accompany Order 413.3B as  the best practices guidance for implementation of Order 413.3B  requirements. We are pleased that DOE agreed with our recommendation  and that its website includes links to the guides that accompany the  project management order. However, our report noted the deficiencies  associated with the existing cost estimating guide, and DOE did not  specify plans for addressing those deficiencies. In addition, we are  concerned that the unspecified, open-ended date for updating the project  management order that contains requirements (i.e., sometime after  November 2016) and the statement that referencing the cost estimating  guide will be considered, not completed, may indicate DOE’s lack of  urgency or concern about the need to implement this recommendation.

In regard to the report’s fourth recommendation—revise DOE directives  that apply to programs to require that DOE and NNSA and its contractors  develop cost estimates in accordance with the 12 cost estimating best  practices, including developing lifecycle cost estimates for programs— DOE stated in its written comments that it is in the process of  substantially revising the existing 1995 DOE Order 130.1, Budget  Formulation, and that as part of this effort, DOE will assess the  requirement for program cost estimates and will revise the order to  provide more specificity on the cost estimating requirements. Further,  DOE stated that the revised order will (1) define which DOE and NNSA  program budget requests require cost estimates and (2) clarify that cost  estimates for program budget submissions shall be conducted in  accordance with the DOE cost estimating guide (or its successor policy).  DOE estimated that it would complete this process in September 2016.  We are pleased that DOE agreed with our recommendation.

In regard to the report’s fifth recommendation—revise DOE requirements  and guidance that apply to programs to ensure that program reviews are  conducted periodically, including reviews of the lifecycle cost estimates  for programs—DOE again stated that it is in the process of substantially  revising the existing DOE Order 130.1, and that as part of this effort, the  department will assess requirements for program reviews and the linkage  between program reviews and the budget formulation process. Further,  DOE stated that the revised order will clarify requirements for program  reviews and specify how such reviews—to include lifecycle cost  estimates—can best support the budget formulation process. NNSA is to  review these requirements and adjust NNSA-specific policies and  guidance as appropriate. DOE estimated that it would complete this  process in September 2016.  We are pleased that DOE agreed with our  recommendation.

We are sending copies of this report to the appropriate congressional  committees; the Secretary of Energy; the Director, Office of Management  and Budget; and other interested parties. In addition, the report is  available at no charge on the GAO website at http://www.gao.gov.

If you or your staff members have any questions about this report, please  contact me at (202) 512-3841 or trimbled@gao.gov. Contact points for  our Offices of Congressional Relations and Public Affairs may be found  on the last page of this report. GAO staff who made major contributions to  this report are listed in appendix IV.

section_title: Appendix I: Comparison of DOE Cost Guide to GAO’s Best Practices for Cost Estimating
paragraphs: Appendix I: Comparison of DOE Cost Guide to  GAO’s Best Practices for Cost Estimating 12 key steps/best practices  Define estimate’s purpose, scope, and schedule.

Determine the estimating approach—work  breakdown structure (WBS).

Identify the ground rules and assumptions.

Develop the point estimate and compare it to an  independent cost estimate.

Update the estimate to reflect actual costs and  changes.

section_title: Appendix II: Cost Estimating Weaknesses Identified in All Completed NNSA Project Reviews February 2009 - February 2014 and GAO’s Best Practices for Cost Estimating
paragraphs: GAO 12 steps (see table legend)
subsections: section_title: Legend: GAO 12 Best Practice Cost Estimating Steps
subsections: section_title: Step Description 1.
paragraphs: Summary of associated tasks  Determine purpose, scope, required level of detail of estimate, as well as who will  receive estimate.

Determine cost estimating team, schedule, and outline tasks in writing.

Identify technical characteristics of planned investment, quality of data needed, and  plan for documenting and updating information.

Define the elements of the cost estimate, including best method for estimating costs and  potential cross-checks, and standardized structure.

Define what the estimate will include and exclude, key assumptions (such as life-cycle  of investment), schedule or budget constraints, and other elements that affect estimate.  Assumptions should be measurable, specific, and consistent with historical data.  Assumptions should be based on expert and technical judgment.

Create data collection plan, identify sources, collect valid and useful data, analyze data  for cost drivers and other factors, and assess data for reliability and accuracy.

Develop a point estimate and  compare it to an independent cost  estimate.

Develop cost estimation model and calculate estimate, in constant dollars for  investments that occur over multiple years, and other cross checks and validation, and  compare estimate to an independent estimate and previous estimates. Update as more  data are available.

Test the sensitivity of cost elements to changes in input values, ground rules, and  assumptions.

section_title: Step Description 9.
paragraphs: Summary of associated tasks  Determine which cost elements pose technical, cost, or schedule risks; analyze those  risks; and recommend a plan to track and mitigate risks. A range of potential costs,  based on risks and uncertainties, should be identified around a point estimate.

Document all steps used to develop the estimate so it can be recreated, describing  methodology, data, assumptions, and results of risk, uncertainty, and sensitivity  analysis.

Develop briefing on results, including information on estimation methods and risks,  making content clear and complete so those unfamiliar with analysis can comprehend  estimate and have confidence in it.

As technical aspects of project change, the complete cost estimate should be regularly  updated and, as project moves forward, cost and schedule estimates should be tracked.

section_title: Appendix III: Comments from the Department of Energy

section_title: Appendix IV: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact

section_title: Staff Acknowledgments
paragraphs: In addition to the individual named above, Daniel Feehan, Assistant  Director; Antoinette Capaccio; Jennifer Echard; Mark Braza; Mike  Meleady; Alison O’Neill; and Peter Ruedel made key contributions to this  report.


Question-summary annotations: questions: question: What is the problem with DOE's cost estimating guidance?
answer: The Department of Energy's (DOE) and its National Nuclear Security Administration's (NNSA) cost estimating requirements and guidance for projects and programs do not fully reflect best practices for developing cost estimates.
child_questions: question: What are the best practices that are reflected in the project management order?
answer: In regard to cost estimating requirements for projects, DOE's 2010 project management order requires 1 of the 12 cost estimating best practices—conducting an independent cost estimate—for larger projects at certain stages of development.

question: What best practices are reflected in the cost estimating guide?
answer: In contrast, DOE's 2011 cost estimating guide describes recommended approaches for using 10 of 12 best practices and partially contains information about the other 2. Furthermore, because DOE's cost estimating guide was issued in 2011—after DOE's 2010 project order was issued—it is not referenced in the order.

question: What is the result of this oversight?
answer: As a result, users of the order may not be aware of the guide's availability and may not benefit from its usefulness. In addition, although NNSA programs are required to follow DOE's budget formulation order and NNSA's budget process, both of which require the development of cost estimates, neither the order nor the process requires the use of best practices in developing the estimates. In the absence of a requirement for using best practices, it is unlikely that DOE, NNSA, and their contractors will consistently develop reliable cost estimates.
child_questions: question: What is one example?
answer: In February 2014, for example, GAO found that NNSA's lifecycle cost estimate for the Plutonium Disposition Program did not follow all key steps for developing high-quality cost estimates, in part because the agency did not have a requirement to develop a life-cycle cost estimate.

questions: question: What did Senate Report 112-73 mandate?
answer: Senate Report 112-73 mandated that GAO report on NNSA's cost estimating practices.

question: What does the report examine?
answer: This report examines: (1) the extent to which DOE and NNSA cost estimating requirements and guidance for projects and programs reflect best practices and (2) the extent to which recent NNSA project and program reviews identified cost estimating weaknesses, and the extent to which the weaknesses can be attributed to not following best practices.
child_questions: question: What was GAO's methodology?
answer: GAO reviewed DOE and NNSA cost estimating practices and compared them to best practices, NNSA project and program reviews, and two programs selected among the largest and most complex NNSA programs. GAO also interviewed DOE and NNSA officials about requirements and guidance for cost estimates.