DOE and NNSA Project Management: Analysis of Alternatives Could Be Improved by Incorporating Best Practices
Published: 2014-12-11T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: NNSA manages numerous programs that require the design and construction of one-of-a-kind facilities. NNSA's selection of preferred alternatives for these projects is governed by DOE requirements and guidance related to conducting an AOA. In recent years, NNSA has incurred substantial cost increases and schedule delays for such projects.
GAO was mandated to review the AOA process applied by NNSA. This report (1) identifies and describes AOA best practices, (2) determines the extent to which DOE requirements and guidance for conducting an AOA conform to AOA best practices, and (3) determines the extent to which NNSA conformed to best practices and followed certain DOE requirements and guidance in conducting the AOAs for recent NNSA projects.
To do this work, GAO examined relevant AOA guidance from the public and private sectors and DOE's AOA requirements and guidance, sought input from AOA experts, and interviewed agency officials. GAO reviewed three AOAs that NNSA completed or was scheduled to complete between November 2010 and September 2014.
section_title: What GAO Found
paragraphs: GAO identified 24 best practices for analysis of alternatives (AOA)—a process that is a key first step in capital asset acquisition. The process entails identifying, analyzing, and selecting a preferred alternative to best meet the mission need by comparing the operational effectiveness, costs, and risks of potential alternatives. Because no single set of best practices for AOAs was broadly recognized by government and private-sector entities, GAO developed a set of practices by reviewing AOA policies and guidance used by seven public and private-sector entities with experience in the AOA process, and verified these practices with subject matter experts. These best practices include, among other things, defining functional requirements based on mission need, conducting the AOA without a predetermined solution, including the status-quo alternative, and conducting an independent review of the entire AOA process. These practices can be applied to a wide range of activities, projects, and programs.
The Department of Energy's (DOE) requirements and guidance—found in the agency's orders and associated guides—for conducting an AOA do not conform to the 24 best practices GAO identified. Therefore, DOE does not have assurance that applying these requirements and guidance may lead to reliable AOAs. GAO's review of DOE's requirements for AOAs found that they conform to only 1 of the 24 best practices: the practice of defining functional requirements based on mission need. GAO's review of DOE's requirements combined with associated guidance—which includes nonmandatory approaches for meeting requirements—found that they conform to 9 of the 24 best practices. For example, DOE's guidance suggests identifying and considering at least three viable alternatives, including the status quo—a best practice that is not included in the requirements. Federal standards for internal control related to risk assessment call for agency management to decide on actions to mitigate identified risks. Without developing a reliable AOA process, DOE may not be successful in mitigating the risk it has identified related to this process.
DOE's requirements and guidance for AOAs also apply to the National Nuclear Security Administration (NNSA), a separately organized agency in DOE. For three recent NNSA projects that GAO reviewed, NNSA did not conform to most of the best practices GAO identified for conducting AOAs, therefore, raising concerns about the reliability of these AOAs. Specifically, for the three projects' AOAs, NNSA conformed to 6, 8, and 11 of the 24 identified best practices, respectively. For all three projects, NNSA consistently followed the one DOE requirement that conformed to a best practice—to define functional requirements based on mission need. DOE and NNSA officials acknowledge that unreliable AOAs are a risk factor for major cost increases and schedule delays for NNSA projects. As GAO has previously reported, NNSA has spent billions of dollars designing and partially constructing projects with an estimated cost of $750 million or more, only to later reassess alternatives. Without a process to develop reliable AOAs, NNSA may continue on this path and continue to have limited assurance that it is selecting alternatives that best meet its mission needs and will not result in major cost increases and schedule delays in the future.
section_title: What GAO Recommends
paragraphs: GAO recommends that DOE incorporate best practices into its AOA requirements. DOE agreed with this recommendation.

section_title: Letter
paragraphs: To accomplish its nuclear security and nonproliferation mission, the  National Nuclear Security Administration (NNSA)—a separately  organized agency within the Department of Energy (DOE)—manages  numerous programs that require NNSA to design and construct one-of-a- kind facilities through a process known as capital asset acquisition.  When NNSA is acquiring such assets, the use of the analysis of  alternatives (AOA) process included in DOE’s requirements and guidance  is a key first step to help ensure that the selected alternative best meets  the agency’s mission need. In 2009, we defined the AOA process as an  analytical study that is intended to compare the operational effectiveness,  cost, and risks of a number of potential alternatives to address valid  needs and shortfalls in operational capability. This process helps ensure  that the best alternative that satisfies the mission need is chosen on the  basis of the selection criteria, such as safety, cost, or schedule.
In recent years, as we previously reported, NNSA has used its  requirements and guidance to select preferred alternatives for many  projects, and then spent billions of dollars designing and partially  constructing several major capital asset projects (facilities with an  estimated cost of $750 million or more), only to later reassess alternatives  for each project. These major projects are the Chemistry and Metallurgy  Research Replacement Nuclear Facility at NNSA’s Los Alamos National  Laboratory in New Mexico, the Mixed Oxide Fuel Fabrication Facility and  the Pit Disassembly and Conversion Facility at NNSA’s Savannah River  Site in South Carolina, and the Uranium Processing FacilityY-12 site in Tennessee, all projects for which NNSA has recently  reassessed or is now in the process of reassessing alternatives. Cost  increases and schedule delays for these projects led to the reassessment  of alternatives and increased scrutiny of NNSA’s project management  process, including requirements and guidance for the AOA.
Senate Report 113-44 accompanying the National Defense Authorization  Act for Fiscal Year 2014 mandated that GAO review the AOA process  applied by NNSA for capital asset acquisition. mandate, this report (1) identifies and describes AOA best practices, (2)  determines the extent to which DOE’s requirements and guidance for  conducting an AOA conform to AOA best practices, and (3) determines  the extent to which NNSA conformed to best practices and followed  certain DOE requirements and guidance in conducting the AOAs for  recent NNSA projects.
S. Rep. No. 113-44, at 258-259 (2013); Pub. L. No. 113-66, 127 Stat. 672 (codified in  scattered sections of 10, 50, and 37 U.S.C.). related to the AOA process included in our Cost Estimating Guide and  from seven additional entities and compiled a draft set of practices  commonly mentioned across these different entities’ AOA policies and  guidance. We sent this draft set of AOA best practices to the experts for  review and received comments by e-mail. We also received comments on  this set of best practices during a GAO semiannual meeting, in which  some of these experts participated, including officials from DOE and  NNSA. We finalized the set of 24 best practices for the AOA process by  incorporating the experts’ comments. We developed this set of best  practices because no one single set of practices existed that was broadly  recognized by government and private-sector entities. We grouped these  24 best practices into four categories: (1) general principles, (2)  identifying alternatives, (3) analyzing alternatives, and (4) selecting a  preferred alternative.
To determine the extent to which DOE’s requirements and guidance for  conducting an AOA conform to AOA best practices, we examined DOE’s  project management order, its associated guides and other  documentation, and interviewed DOE and NNSA officials to identify the  process NNSA is required to follow when conducting an AOA. We then  confirmed these requirements with DOE and NNSA officials. We  performed two sets of analysis: we compared the best practices to (1)  DOE’s requirements and (2) DOE’s requirements combined with the  guidance. We used a five-point scoring system to determine the extent to  which DOE’s AOA process conforms to best practices. We first used this scoring system to determine how well DOE’s requirements and  guidance conform to each best practice. We then used the average of the  scores for the best practices in each of the four categories to determine  an overall score for each category, and we then used the average of the  scores for the four categories as the final score for the overall DOE AOA  process. If the score for each best practice, the average score for each  category, or the final score for the AOA process was “fully met” or  “substantially met,” we concluded that the AOA process conformed to  best practices and therefore could be considered reliable. In contrast, if  the score was “partially met,” “minimally met,” or “not met,” we concluded  that the AOA process did not conform to best practices and therefore  could not be considered reliable.
To determine the extent to which NNSA conformed to best practices and  followed certain DOE requirements and guidance in conducting the AOAs  for recent NNSA projects, we chose projects for which NNSA completed  the AOA process since November 2010, when DOE’s most recent version  of the project management order came into effect, or for which NNSA was  scheduled to complete the AOA process by the end of fiscal year 2014.  We identified these projects by examining DOE’s data from its February  2014 Monthly Project Portfolio Status Report. The three projects that met  our criteria were (1) the High Explosive Science, Technology and  Engineering Project at NNSA’s Pantex site in Texas, (2) the Radioactive  Liquid Waste Treatment Facility at NNSA’s Los Alamos National  Laboratory in New Mexico, and (3) the Uranium Processing Facility at  NNSA’s Y-12 site in Tennessee.and NNSA officials, reviewed recent GAO reports, and compared the data  in the project status report with information received from NNSA on the  projects to assess the reliability of the data in this status report, and we  determined that the data were sufficiently reliable to ensure these projects  met our criteria. For each of these projects, we reviewed project  documentation, and we interviewed NNSA and contractor officials in   We checked with knowledgeable DOE  charge of the AOA process to compare the process followed by NNSA  with best practices. In April 2014, as we were conducting our  engagement, NNSA released a peer review report that recommended a  new alternative for the Uranium Processing Facility, and we examined  whether this peer review reflected any characteristics of an AOA. In  addition, we reviewed project documentation and interviewed NNSA and  contractor officials in charge of the AOA process to determine the extent  to which NNSA conformed to certain DOE requirements and guidance— those DOE requirements and guidance that conformed to best  practices—in conducting the AOAs for these projects. Appendix I  presents a more detailed description of our scope and methodology.
We conducted this performance audit from February 2014 to December  2014 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: This section describes DOE’s requirements and guidance for AOA; DOE  and NNSA offices responsible for conducting AOAs for capital asset  projects; and NNSA’s recent projects that have completed, or are nearing  completion of, an AOA.
subsections: section_title: DOE’s Requirements and Guidance for Analysis of Alternatives
paragraphs: DOE’s mandatory requirements and optional guidance for identifying,  analyzing, and selecting alternatives when conducting AOAs also apply to  NNSA. DOE’s requirements for the AOA process are outlined in Order  413.3B governing the Program and Project Management for the  Acquisition of Capital Assets (hereafter referred to as Order 413.3B).  This order includes requirements for the acquisition of capital assets, with  the stated goal of delivering fully capable projects within the planned cost,  schedule, and performance baseline. In addition to the order’s  requirements, DOE has guidance for identifying, analyzing, and selecting  alternatives (hereafter referred to as “guidance”) that is found throughout  seven guides associated with the order. DOE states at the beginning of  each of these guides that they include nonmandatory approaches for  meeting requirements, that guides are not requirements documents, and  that they are not to be construed as requirements in any audit for  appraisal of compliance with the parent policy or order.
DOE’s Order 413.3B establishes five critical decision processes of project  development that each end with a major approval milestone—or “critical  decision point”; these decision processes cover the life of a project. Under  Order 413.3B, an AOA occurs during the span of the first two critical  decision processes—the preconceptual design process (CD-0) and the  conceptual design process (CD-1)—with the majority of the AOA being  conducted during the conceptual design process and ending with CD-1  approval. Figure 1 illustrates when DOE conducts the analysis of  alternatives as part of its project management process.
DOE’s Order 413.3B and its associated guides include the AOA-related  requirements and guidance listed below, among others. Appendix II  includes a full description of DOE’s requirements and guidance for  conducting an AOA.
As part of the preconceptual design (CD-0) approval process, the  mission need—which DOE defines in Order 413.3B as a credible  gap between current capabilities and those required to meet the  goals articulated in the strategic plan—and functional  requirements—the general parameters that the selected  alternative must have to address the mission need—must be  identified. The mission need must be independent of a particular  alternative, and the program office responsible for the capital  asset project must explore a variety of alternatives.
As part of the conceptual design (CD-1) approval process, reliable  cost and schedule range estimates for the alternatives considered  must be developed, and whatever figure or range is provided at  the CD-0 and CD-1 stages must explicitly note relevant caveats  concerning risks and uncertainties inherent in early estimates. In  addition, a conceptual design report must be developed that  includes, among other things, a clear and concise description of  the alternatives analyzed, the basis for the selected alternative,  how the selected alternative meets the mission need, the  functional requirements that define the alternative and  demonstrate that the alternative can be successful, and life-cycle  cost assumptions.
During the CD-0 and CD-1 approval process, several independent  reviews must be conducted, depending on the estimated cost of  the project, related to two aspects of the AOA process: validation  of (1) the mission need statement and (2) the cost estimates.
As a project moves toward approval of CD-2, if the top end of the  approved CD-1 cost range for the selected alternative grows by  more than 50 percent, the program office must reassess  alternatives by conducting another AOA and obtaining another  CD-1 approval.
DOE’s guides suggest summarizing a planned approach to  conduct an analysis of alternatives; considering at least three  viable alternatives for analysis, including one that represents the  status quo; developing cost estimates that are explicit ranges  instead of point estimates; including life-cycle cost estimates of  the alternatives being considered; quantifying the benefits of  alternatives over their life cycle; adjusting life-cycle cost and  benefit estimates for risk; considering various selection criteria for  the alternatives; weighting the selection criteria for relative  importance; and comparing alternatives using net present value.The guidance suggests that, at the end of the analysis, the AOA  team present the recommended alternative based on the  preceding analysis in an integrated form, summarizing why an  alternative is preferred and supporting the recommendation of the  preferred alternative with facts from the analysis.  In addition, DOE’s guidance suggests that the program office  review the alternative selection and cost range to assess whether  the AOA process (1) evaluates a range of appropriate attributes  for each alternative, including cost, risks, safety, technology and  regulatory requirements, and (2) is reasonable and provides best  value to the government.
Order 413.3B also requires prior to CD-1 approval that DOE conduct a  separate analysis of alternatives under the National Environmental Policy  Act of 1969 (NEPA). Under NEPA, federal agencies must identify and  assess the likely environmental effects of proposed projects, and  reasonable alternatives, using an environmental assessment or, if the  projects likely would significantly affect the quality of the human  environment, a more detailed environmental impact statement.  Recognizing the need for better integration between the AOA conducted  under Order 413.3B and the AOA conducted under NEPA, in early 2012,  DOE established a multidisciplinary team of NEPA-compliance and  program- and project-management specialists, and in June 2012, the  Secretary of Energy issued a memo highlighting the importance of  integrating these two AOAs. DOE officials who were part of this team  stated that DOE has not yet decided how to connect the NEPA process  with the AOA process under Order 413.3B.
section_title: DOE and NNSA Offices Responsible for Conducting AOAs for Capital Asset Projects
paragraphs: Within DOE and NNSA, several groups are responsible for various  aspects of conducting an AOA:
DOE Office of Acquisition and Project Management (DOE APM).  This office is responsible for, according to DOE APM officials, writing  DOE’s project management Order 413.3B and its associated guides,  including requirements and guidance for conducting an AOA, and for  reviewing some DOE projects.
NNSA Office of Acquisition and Project Management (NNSA  APM). This office is responsible for managing construction of capital  asset projects within approved cost and schedule estimates, and for  conducting at least yearly reviews of the construction projects to  evaluate technical, cost, scope, and other aspects of the projects.  According to NNSA APM’s Concept of Operations, this office leads  the development of the analysis of alternatives for NNSA projects.
NNSA program offices. Program offices are responsible for the  projects that fall within their portfolios. For projects, according to  NNSA APM’s Concept of Operations, the program offices lead the  development of the mission needs and functional requirements,  support the analysis of alternatives, and lead the selection of the  preferred alternative.
NNSA Office of Cost Estimating and Program Evaluation. The  National Defense Authorization Act for Fiscal Year 2014 mandated  the permanent establishment of this office to provide an independent  review of cost estimates and to advise the NNSA Administrator on  AOA policies and procedures, among other responsibilities. NNSA  officials stated that they expected this office to work with other NNSA  offices to define its roles and responsibilities as they relate to  conducting AOAs during fiscal year 2015.
section_title: NNSA’s Recent Projects That Have Completed, or Are Nearing Completion of, an AOA
paragraphs: The following describes the three recent NNSA projects that we reviewed  that had completed, or are nearing completion of an AOA.
subsections: section_title: High Explosive Science, Technology and Engineering Project
paragraphs: This project is intended to replace and upgrade existing facilities used for  NNSA’s high explosive manufacturing support, surveillance, testing, and  technology development program. NNSA began conducting the AOA in  November 2011 and expects to complete the AOA process by obtaining  CD-1 approval in December 2014. As of September 2014, NNSA  estimated that the total project cost would range from $100 million to  $155 million at an 85 percent confidence level.
section_title: Radioactive Liquid Waste Treatment Facility
paragraphs: This project is intended to replace an existing facility at the Los Alamos  National Laboratory to treat and dispose of two types of radioactive  waste. According to this project’s mission need statement document, the  existing facility was nearing its designed life span and had suffered  numerous failures. NNSA completed an initial AOA in 2006, selecting the  alternative to place the treatment capabilities for both types of waste in  the same structure. NNSA estimated in 2006 that the total project cost  would likely range between $82 million and $104 million. After significant  cost increases were incurred during the preliminary design process of the  project, NNSA completed a reassessment of alternatives in September  2013, which we examine in this report. NNSA indicated in this  reassessment that the existing facility was beyond its designed life span.  The selected alternative for this reassessment separates treatment  capabilities into two separate structures, and NNSA estimated that the  revised total project cost would likely range between $168 million and  $220 million.
section_title: Uranium Processing Facility
paragraphs: This project is intended to replace existing facilities for enriched uranium  capabilities that, according to this project’s mission need statement  document, require intensive maintenance and are experiencing escalating  operating costs. NNSA completed an initial AOA in 2007, selecting the  alternative to construct a new building with full capability to process  enriched uranium. NNSA estimated in 2007 that the total project cost  would likely range between $1.4 billion and $3.5 billion. During the  preliminary design phase, the estimated cost of the project increased  significantly, and NNSA completed a reassessment of alternatives in June  2012. This second AOA, which we examine in this report, selected a new  alternative to construct the same building, but included fewer capabilities,  while deferring significant portions of the project’s original scope to be  added in the facility at a later time. The approved estimated cost  increased to a range of $4.2 billion to $6.5 billion.
This independent cost estimate was requested under the National Defense  Authorization Act of 2013. The $9.8 billion is a point estimate and not the full cost range.  The top end of the Department of Defense’s Office of Cost Assessment and Program  Evaluation cost range estimate was at least $10.5 billion with the potential of higher cost  depending on constrained or unconstrained funding profile.
NNSA and DOE senior management a new alternative of building two  new, smaller facilities to include all capabilities approved in the 2012  AOA, and senior management approved this new alternative. NNSA  began to consider alternatives other than building a single facility  because, according to the UPF Federal Project Director, the project was  facing budget constraints, rising costs, and competition from other high- priority projects within NNSA. In January 2014, the NNSA Administrator  asked for a peer review to develop and recommend an alternative  approach to UPF. In April 2014, the peer reviewers recommended an  alternative similar to the one approved by DOE and NNSA senior  management in December 2013. This alternative would require NNSA to  construct two new, smaller facilities and upgrade existing facilities to  cover the same capabilities approved in the 2012 AOA, all within the $6.5  billion approved top end of the cost range. According to NNSA officials,  this peer review was not an AOA. According to the NNSA Deputy  Federal Project Director for this project, the Federal Project Director  provided direction in April 2014 to proceed with the alternative proposed  by the peer review. According to NNSA officials in October 2014, the  alternative was further modified. At this time, NNSA’s plan was to build  three new, smaller facilities and upgrade existing facilities to provide the  same capabilities approved in 2012 and still remain within the approved  cost range of $4.2 billion to $6.5 billion.
section_title: GAO Identified 24 Best Practices for the Analysis of Alternatives Process
paragraphs: Because no one single set of practices existed that was broadly  recognized by government and private-sector entities for the AOA  process, we identified 24 best practices for identifying, analyzing, and  selecting alternatives. We identified these best practices by (1) compiling  and reviewing commonly mentioned AOA policies and guidance used by  different government and private-sector entities and (2) incorporating  experts’ comments on our draft set of practices to develop a final set of  practices. These practices can be applied to a wide range of activities in  which an alternative must be selected from a set of possible options, as  well as to a broad range of capability areas, projects, and programs.  These practices can provide a framework to help ensure that entities  consistently and reliably select the project alternatives that best meet  mission needs.
We grouped these 24 best practices into four categories: (1) general  principles, (2) identifying alternatives, (3) analyzing alternatives, and (4)  selecting a preferred alternative. The four categories of best practices  address the AOA process from defining the mission need and functional  requirements to independently reviewing its results. On the basis of our  reviews and experts’ comments, we believe that these best practices can  be generally applied from the beginning of the AOA process with  practices from the general principles category, through practices in the  identifying and analyzing alternatives categories, and ending with  practices in the selecting a preferred alternative category. We also  believe that these best practices do not necessarily have to be followed in  order and that some of them can be applied concurrently with other best  practices. For example, the best practice of defining the selection criteria  based on the mission need in the selecting a preferred alternative  category could be addressed at the same time as the best practice of  creating a study plan in the general principles category. The following  sections describe the 24 best practices by category.
subsections: section_title: General Principles Category
paragraphs: The general principles category contains best practices that would need  to be applied before starting the process of identifying, analyzing, and  selecting alternatives, such as determining the mission need and  functional requirements, developing the study time frame and creating a  study plan, and determining who conducts the analysis. It also includes  best practices that would need to be applied throughout the AOA process,  such as documenting all steps taken to identify, analyze, and select  alternatives in a single document. Table 1 lists the best practices in the  general principles category.
section_title: Identifying Alternatives Category
paragraphs: The identifying alternatives category contains best practices that help  ensure the alternatives to be analyzed are sufficient, diverse, and viable.  Table 2 lists the best practices in the identifying alternatives category.
section_title: Analyzing Alternatives Category
paragraphs: The analyzing alternatives category contains best practices that are  necessary to compare the alternatives selected for analysis. The best  practices in this category help ensure that the team conducting the  analysis uses a standard, quantitative process to assess the alternatives.  This category includes best practices related to estimating the costs and  benefits of each alternative over its life cycle and understanding the  impacts of risks and key assumptions on these estimates. Table 3 lists  the best practices in the analyzing alternatives category.
section_title: Selecting a Preferred Alternative Category
paragraphs: The category for selecting a preferred alternative contains best practices  that help ensure the team selects a preferred alternative that best meets  the mission needs. This category includes best practices related to  defining selection criteria, differentiating among the selection criteria, and  independently reviewing the AOA process. Table 4 lists the best practices  in this category.
On the basis of our reviews and experts’ comments, we believe that  conforming to these best practices helps ensure that the preferred  alternative that is selected is the one that best meets the agency’s  mission needs. Not conforming to best practices may lead to an  unreliable AOA, and the agency will not have assurance that the  preferred alternative best meets its mission needs. Appendix III lists the  best practices from all four categories.
section_title: DOE’s AOA Requirements and Guidance Do Not Conform to Best Practices
paragraphs: Neither DOE’s AOA requirements nor its guidance conform to best  practices and, therefore, DOE does not have assurance that applying  these requirements and guidance may lead to reliable AOAs. Our  review of DOE’s requirements contained within Order 413.3B found that  they minimally meet best practices overall, and that they do not fully or  substantially meet best practices in any of the four categories. Our  analysis also found that, even when DOE’s requirements are combined  with guidance contained in the guides associated with Order 413.3B, they  only partially meet best practices overall, and they only fully or  substantially meet best practices in the identifying alternatives and  selecting a preferred alternative categories but not in the general  principles or analyzing alternatives categories.
subsections: section_title: DOE’s Order 413.3B Requirements Do Not Conform to Best Practices
paragraphs: DOE’s Order 413.3B requirements for the AOA process minimally meet  best practices. Therefore, DOE does not have assurance that applying  these requirements may lead to reliable AOAs. DOE’s requirements do  not fully or substantially meet best practices in any of the four AOA  categories, and they conform to only 1 of the 24 best practices we  identified—the practice of having the customer define functional  requirements based on the mission need. Our assessment of DOE’s  Order 413.3B requirements found the following:
General principles. DOE’s Order 413.3B requirements minimally  meet best practices for the general principles category by  substantially meeting 1 of the 8 best practices in this category,  partially or minimally meeting 4, and not meeting the other 3. DOE  requires that the program office define a project’s mission need  without a predetermined solution and that the Integrated Project Team  must represent a diverse range of disciplines, develop functional  requirements to satisfy the mission need, and document some of the  steps and assumptions required as part of conducting the AOA.  DOE’s Order 413.3B does not make any reference to appropriate  AOA time frames, an AOA study plan, or conducting the analysis  without a predetermined solution.
Identifying alternatives. DOE’s Order 413.3B requirements  minimally meet best practices for identifying alternatives by partially or  minimally meeting 2 of the 4 best practices in this category and not  meeting the other 2. DOE requires that the project’s customer, not the  AOA team, explore a variety of alternatives and that the alternatives  be defined clearly and concisely. DOE’s Order 413.3B does not make  any reference to including a status quo alternative in the AOA or pre- screening the list of identified alternatives for viability.
Analyzing alternatives. DOE’s Order 413.3B requirements minimally  meet best practices for analyzing alternatives by partially or minimally  meeting 3 of the 8 best practices in this category and not meeting the  other 5. DOE requires that life-cycle cost assumptions be included in  the conceptual design report, which leaves room for interpretation  because it could be interpreted as requiring the inclusion of life-cycle  cost estimates for all alternatives or, as DOE and NNSA officials said  that they interpret this statement, as requiring a life-cycle cost  estimate for only the selected alternative after completion of the AOA  process. DOE also requires that project risks and mitigation strategies  are assessed. DOE’s Order 413.3B does not make any reference to  presenting life-cycle cost estimates in present value terms, quantifying  benefits, explaining how measures of benefit support the mission  need, or testing the sensitivity of cost and benefit estimates to risks  and changes in key assumptions.
Selecting a preferred alternative. DOE’s Order 413.3B requirements  do not meet best practices for selecting a preferred alternative by  minimally meeting 1 of the 4 best practices in this category and not  meeting the other 3. The best practice that DOE minimally meets is  having an entity independent of the AOA process reviewing the extent  to which all best practices have been followed. Order 413.3B contains  four separate reviews that might be required during CD-0 and CD-1  processes depending on the amount of the project’s cost estimate, but  these reviews are limited to two aspects of the AOA process: the  validation of (1) the mission need statement and (2) the cost  estimates. In addition, one of these reviews is not an independent  review because it is conducted by the program office, which is on the  project’s chain of command. Table 5 lists these required reviews.  DOE’s requirements do not meet the other best practices in this  category because DOE’s Order 413.3B does not make any reference  to defining selection criteria based on the mission need; weighting  selection criteria to reflect the relative importance of each; or  comparing the alternatives considered, using net present value.
Table 6 and appendix IV summarize our assessment of DOE’s  requirements.
section_title: DOE’s Requirements Combined with Guidance Do Not Conform to Best Practices
paragraphs: DOE’s Order 413.3B AOA requirements combined with guidance partially  meet best practices. Therefore, DOE does not have assurance that  applying its requirements combined with its guidance may lead to reliable  AOAs. DOE’s requirements combined with guidance fully or substantially  met best practices in the identifying alternatives and selecting a preferred  alternative categories, but not in the general principles or analyzing  alternatives categories. DOE’s requirements combined with guidance  conform to 9 of 24 best practices. Our assessment of DOE’s  requirements combined with guidance found the following:
General principles. DOE’s requirements combined with guidance  partially meet best practices for the general principles category by  fully or substantially meeting 3 of the 8 best practices in this category,  partially or minimally meeting 3, and not meeting the other 2. DOE’s  guidance supplements the requirements discussed in the previous  section by suggesting that the Federal Project Director and the  Integrated Project Team conduct different parts of the AOA, but the  guidance does not suggest specific areas of expertise to be  represented on the team. The guidance also suggests that a study  plan be developed but does not suggest what to include in that plan.  Neither DOE’s requirements nor guidance refer to appropriate AOA  time frames or conducting the analysis without a predetermined  solution.
Identifying alternatives. DOE’s requirements combined with  guidance substantially meet best practices for the identifying  alternatives category by fully meeting 1 of the 4 best practices in this  category and partially or minimally meeting the other 3. DOE’s  guidance supplements the requirements by suggesting that at least  three alternatives be considered, including one representing the  status quo, and by listing the advantages and disadvantages of each  alternative. Additionally, DOE’s guidance suggests pre-screening the  alternatives for viability, but the guidance does not suggest  documenting the reasons for eliminating any alternatives.
Analyzing alternatives. DOE’s requirements combined with  guidance partially meet best practices for the analyzing alternatives  category by fully or substantially meeting 3 of the 8 best practices in  this category, partially or minimally meeting 3, and not meeting the  other 2. DOE’s guidance supplements the requirements by suggesting  that life-cycle cost estimates be developed for each of the  alternatives, the life-cycle cost estimates be presented as ranges  instead of point estimates, life-cycle benefit estimate be developed for  each alternative, risk be taken into consideration when evaluating  alternatives, and life-cycle cost estimates be adjusted for risk. Neither  DOE’s requirements nor guidance refer to presenting life-cycle cost  estimates in present value terms, explaining how each measure of  benefit supports the mission need, or testing the sensitivity of the cost  and benefit estimates to changes in key assumptions.
Selecting a preferred alternative. DOE’s requirements combined  with guidance substantially meet best practices for the selecting a  preferred alternative category by fully or substantially meeting 2 of the  4 best practices in this category and partially or minimally meeting the  other 2. DOE’s guidance supplements the requirements by suggesting  that a variety of discriminators are considered when selecting a  preferred alternative, the selection criteria are weighted to reflect the  relative importance of each, and the alternatives are compared using  net present value. The guidance also suggests another review of  more aspects of the AOA process, such as whether the alternative  selection process evaluates a range of appropriate attributes for each  alternative including cost, maintainability, safety, technology  requirements, risks, and regulatory requirements, but the guidance  suggests that this review be performed by the program office and not  by a party independent of the project. Table 7 below and appendix IV  summarize our assessment of DOE’s requirements combined with  guidance.
DOE and NNSA officials told us that unreliable AOAs are a risk factor for  major cost increases and schedule delays for NNSA projects. DOE APM  and NNSA APM officials acknowledged that DOE does not have definitive  guidance on conducting AOAs. According to these officials, the absence  of adequate guidance has led to some projects pursuing alternatives that  subsequently proved to be unaffordable. For example, each of NNSA’s  ongoing major projects (e.g., the Chemistry and Metallurgy Research  Replacement Nuclear Facility, the Mixed Oxide Fuel Fabrication Facility,  or the Uranium Processing Facility) has undergone reassessments of  alternatives due to cost increases and schedule delays. As we previously  reported, federal standards for internal control related to risk assessment  call for agency management to assess the risks faced entity-wide and at  the activity level, and that once risks have been identified, management  should decide what actions should be taken to mitigate them.developing a reliable AOA process, DOE may not be successful in  mitigating the risk it has identified related to this process.
section_title: NNSA’s AOAs for Three Recent Projects Did Not Conform to Best Practices and Did Not Consistently Follow Certain DOE Guidance
paragraphs: For three recent projects, our analysis found that NNSA did not conform  to most of the best practices in conducting the AOAs,concerns about the reliability of these AOAs. NNSA also did not  consistently follow certain DOE optional guidance for these AOAs.
subsections: section_title: NNSA’s AOAs for the Three Recent Projects Did Not Conform to Best Practices
paragraphs: NNSA did not conform to best practices overall in conducting the AOAs  for the three recent projects we examined—the High Explosive Science,  Technology and Engineering Project, the Radioactive Liquid Waste  Treatment Facility, and the Uranium Processing Facility, therefore, raising  concerns about the reliability of these AOAs. NNSA also conducted each  AOA differently. (See apps. V through VII for more detailed summaries of  our assessment of the AOAs for each project.)
subsections: section_title: High Explosive Science, Technology and Engineering Project AOA
paragraphs: For the High Explosive, Science, Technology and Engineering Project,  NNSA partially met best practices overall in conducting the AOA.  Therefore, this AOA may not be reliable. NNSA fully or substantially met  best practices in the general principles and identifying alternatives  categories but not in the analyzing alternatives or selecting a preferred  alternative categories. NNSA conformed to 11 of 24 best practices when  conducting this AOA. Our assessment of NNSA’s AOA for this project  identified the following:
General principles. NNSA substantially met best practices for the  general principles category by fully or substantially meeting 4 of the 8  best practices in this category and by partially or minimally meeting  the other 4. NNSA convened a team, which included federal and  contractor project officials to conduct the analysis. Before conducting  the analysis, NNSA developed a 17-step plan for conducting the AOA.  When conducting the analysis, the team described in project  documentation many, but not all, of the steps it took, and listed most  of the assumptions made in the analysis but did not justify them.  NNSA also undertook the analysis having an “originally envisioned”  alternative. NNSA ultimately did not select this alternative as the  preferred alternative, but NNSA did not explain how the analysis was  conducted without having a predetermined solution.
Identifying alternatives. NNSA fully met best practices for the  identifying alternatives category by fully or substantially meeting all 4  best practices in the category. NNSA identified nine alternatives that  represented a diverse range of potential solutions to the mission  need, including one that represented the status quo. NNSA defined  these alternatives in detail, including providing descriptions of the  specific characteristics used to create cost estimates. Before  proceeding to the analysis, NNSA eliminated four alternatives it did  not consider viable and described the general reasons for eliminating  them but did not provide specific reasons for the scores it gave each  alternative as part of the screening process.
Analyzing alternatives. NNSA minimally met best practices for the  analyzing alternatives category by fully meeting 1 of the 8 best  practices in this category, partially or minimally meeting 4, and not  meeting the remaining 3. To assess the alternatives, NNSA  developed cost estimates for each alternative, and presented these  cost estimates with a stated accuracy range of 15 percent below to 50  percent above, but they were not life-cycle cost estimates because  NNSA did not include such costs as decontamination and  decommissioning in the estimates because it assumed those costs  would be identical among the alternatives. NNSA did not develop  quantitative estimates of the benefits that would result from each  alternative, and mentioned the benefits of the alternatives only  qualitatively through a discussion of some of the pros and cons of  each alternative. To assess the risks associated with the alternatives,  NNSA developed a list of risks for each alternative, but it did not  develop potential mitigation strategies for all of the risks and did not  test the sensitivity of the cost estimates to these risks or to changes in  key assumptions.
Selecting a preferred alternative. NNSA partially met best practices  for the selecting a preferred alternative category by fully or  substantially meeting 2 of the 4 best practices in this category,  partially meeting 1 and not meeting 1. To select a preferred  alternative, NNSA developed 21 selection criteria such as life-cycle  costs, schedules, risks, flexibility, and ability to meet the mission  need. NNSA then weighted these criteria on a five-point scale to  reflect their relative importance, and selected the alternative that  received the highest overall score. As of June 2014, NNSA planned to  conduct an independent project review to assess, among other things,  whether the selected preferred alternative remained the most viable  and feasible alternative for meeting the approved mission need, but it  did not plan on assessing other parts of the AOA process, including  the identification of alternatives, the criteria used for selecting the  alternative, or the benefit estimates for each alternative. Table 8  below and appendix V summarize our assessment of the AOA  conducted for this project.
For the Radioactive Liquid Waste Treatment Facility project, NNSA  partially met best practices overall in conducting the most recent AOA in  2013. Therefore, this AOA also may not be reliable. NNSA fully or  substantially met best practices in the general principles and identifying  alternatives categories but not in the analyzing alternatives or selecting a  preferred alternative categories. NNSA conformed to 8 of 24 best  practices when conducting this AOA. Our assessment of the AOA  conducted for this project identified the following:
General principles. NNSA substantially met best practices for the  general principles category by fully or substantially meeting 3 of the 8  best practices in this category and by partially or minimally meeting  the other 5. NNSA convened a team consisting of project  management and technical experts. NNSA developed detailed study  documentation for some areas of the AOA process, such as for risk  and cost analysis, but not for others, such as for describing how  NNSA screened alternatives for viability or how NNSA developed the  selection criteria. NNSA also listed but did not justify the assumptions  used in the AOA report, and did not discuss any potential constraints  associated with the analysis. This analysis also may have been  conducted with having a predetermined solution because one of the  two alternatives considered was developed from a previous design  that had already incurred significant cost increases and so was  unlikely to be selected.
Identifying alternatives. NNSA substantially met best practices for  the identifying alternatives category by fully or substantially meeting 2  of the 4 best practices in the category, and by partially or minimally  meeting the other 2. NNSA identified thirty-seven options for various  pieces of this project, and then combined those options into two  complete alternatives to be analyzed: one that was based on the  design selected in the 2006 AOA and another with a design similar to  a previously-considered alternative. This approach compared an  alternative developed from a design that had already incurred cost  increases significant enough to warrant a reassessment of  alternatives to only one other alternative. NNSA did, however, define  the two alternatives in detail, assess the viability of the status quo and  screen the list of thirty-seven options for viability before combining  them into the two final alternatives.
Analyzing alternatives. NNSA partially met best practices for the  analyzing alternatives category by fully or substantially meeting 3 of  the 8 best practices in this category, partially or minimally meeting 2,  and not meeting 3. To assess the alternatives, NNSA developed full  life-cycle cost estimates for both of the alternatives, which it presented  in present-value terms, but which it did not present with a confidence  interval or range. NNSA did not develop quantitative estimates of the  benefits that would result from each alternative, and did not explain  how any measure of benefit used would support the mission need. To  assess the risks associated with the alternatives, NNSA developed a  list of risks and mitigation strategies for each alternative, and NNSA  adjusted the total project cost estimates for each alternative, but not  the life-cycle cost estimates, for these risks. NNSA did not, however,  test the sensitivity of the full life-cycle cost estimates to risks or  changes in key assumptions.
Selecting a preferred alternative. NNSA minimally met best  practices for the selecting a preferred alternative category by partially  or minimally meeting 2 of the 4 best practices in this category and by  not meeting the other 2. To select a preferred alternative, NNSA  compared the life-cycle cost estimates, the total project cost  estimates, estimated schedules, and the scope of the alternatives.  NNSA did not weight these selection criteria and did not compare the  alternatives using net present value. NNSA conducted an  independent project review of only the selected alternative, not of  each alternative considered. NNSA also conducted a cost estimate  review of total project cost estimates, not the life-cycle cost estimates,  of each alternative. Table 9 below and appendix VI summarize our  assessment of the AOA conducted for this project.
For the Uranium Processing Facility project, NNSA partially met best  practices overall in conducting the most recent AOA in 2012. Therefore,  this AOA also may not be reliable. NNSA fully or substantially met best  practices in the identifying alternatives category but not in the general  principles, analyzing alternatives, or selecting a preferred alternative  categories. NNSA conformed to 6 of 24 best practices when conducting  this AOA. Our assessment of NNSA’s 2012 AOA for this project identified  the following observations:
General principles. NNSA partially met best practices for the general  principles category by fully or substantially meeting 2 of the 8 best  practices in this category and by partially or minimally meeting the  other 6. NNSA convened a team composed of federal project  management staff and contractors to conduct the AOA, and they  conducted the bulk of the analysis over 3 days in January 2012.  NNSA did not create a detailed plan for conducting the AOA.  Additionally, NNSA conducted the analysis for this project with,  according to NNSA officials, a fundamental assumption that the  selected alternative would be a single-facility solution such as the  alternative selected in the project’s previous AOA conducted in 2007.
Identifying alternatives. NNSA substantially met best practices for  the identifying alternatives category by fully meeting 1 of the 4 best  practices in the category, and by partially or minimally meeting the  other 3. NNSA identified six alternatives, all of which were variations  of the single-facility solution because, according to project officials,  they were “wedded” to that idea. NNSA also did not consider the  status quo as an alternative. NNSA did determine that one alternative  was not viable before proceeding with the analysis but did not  document the screening of all the alternatives against viability criteria.  NNSA described the alternatives in sufficient detail.
Analyzing alternatives. NNSA minimally met best practices for the  analyzing alternatives category by fully meeting 1 of the 8 best  practices in the category, partially or minimally meeting 5, and by not  meeting the other 2. To assess the alternatives, according to UPF  project officials, NNSA developed relative cost estimates for each  alternative that were updates of a 2010 total project cost estimate and  that were not life-cycle cost estimates because they did not include  costs that were the same across all alternatives. NNSA also did not  develop quantitative estimates of the benefits that would result from  each alternative. To assess the risks associated with the alternatives,  NNSA listed categories of risk, such as cost, schedule, and technical  risks in the description of each alternative but did not discuss specific  risks or mitigation strategies. NNSA adjusted the design and  construction elements of the cost estimates based on the risks  identified in developing the 2010 total project cost estimate, but NNSA  did not document these risks in project documentation and did not test  the sensitivity of the estimates to changes in key assumptions.
Selecting a preferred alternative. NNSA partially met best practices  for the selecting a preferred alternative category by fully or  substantially meeting 2 of the 4 best practices in this category and by  partially or minimally meeting the other 2. To select a preferred  alternative, according to UPF project officials, NNSA used  commercially available software to compare each alternative to every  other alternative. NNSA used six selection criteria, including life-cycle  cost, execution flexibility, and ability to achieve benefits from  modernizing and consolidating enriched uranium processing, but  NNSA did not compare the alternatives using net present value.  NNSA weighted these criteria on a percentage scale and selected the  alternative that scored the highest. NNSA also conducted an  independent review after conducting the AOA to determine if the  project was technically ready, but this was not a review of the AOA  process. Table 10 below and appendix VII summarize our  assessment of the AOA conducted for this project.
NNSA conducted each AOA differently and applied best practices  differently among the projects. For example, as mentioned above in the  description of the AOAs conducted at these projects, NNSA conducted  the risk analysis differently for each project. The different ways that NNSA  conducted the AOAs for the projects we examined may be due to the  nonprescriptive nature of the Order 413.3B requirements. NNSA project  officials stated that the AOA requirements in Order 413.3B are very high  level and do not prescribe specific methods to follow. According to these  officials, Order 413.3B provides a very general set of requirements for  what the AOA process should accomplish. For example, DOE’s Order  413.3B states that life-cycle cost assumptions should be included in the  conceptual design report. The word “assumptions” leaves room for  interpretation because it could be interpreted as requiring the inclusion of  life-cycle cost estimates for all alternatives or as requiring a life-cycle cost  estimate for only the selected alternative after completion of the AOA  process. In the absence of specific requirements, each AOA team  developed its own approach to conducting an AOA. For example, NNSA  officials who worked on the AOA for the High Explosive, Science,  Technology and Engineering Project stated that they had to develop  much of the AOA process themselves and that they based their process  on historical AOAs conducted at the Pantex site in Texas. Officials  working on the AOA for the Radioactive Liquid Waste Treatment Facility  project at Los Alamos Nuclear Laboratory told us that they also used site- specific guidance including the laboratory’s engineering standards  manual as well as their own professional experience. In contrast, officials  working on the Uranium Processing Facility project stated that they  conducted the AOA based on guidance contained within a commercially  available, off-the-shelf software suite.
In addition, there may be some confusion about roles and responsibilities  for leading the AOA process at NNSA. NNSA officials in the Office of  Acquisition and Project Management, who are responsible for managing  the construction process, stated that their office does not have lead  responsibility for the AOA process. However, the NNSA Concept of  Operations for the Office of Acquisition and Project Management states  that, while the program offices lead the development of the mission need  and functional requirements and the selection of the preferred  alternatives, the NNSA Office of Acquisition and Project Management has  the lead for the development of the analyses of alternatives. NNSA  officials stated that they will reexamine the Concept of Operations to  clarify who has responsibility for the AOA process. The officials explained  that NNSA APM has begun looking at, among other things, DOE’s current  directives and guidance and NNSA’s policies and procedures relating to  AOA, the Office of Management and Budget’s guidance relating to AOAs,  and the legislation Congress has enacted in response to other federal  agencies’ AOA challenges.
section_title: NNSA Did Not Consistently Follow Certain DOE AOA Guidance for Its Projects
paragraphs: For the three AOAs we examined, NNSA consistently followed the DOE  AOA requirement that conformed to best practices but did not consistently  follow the elements of optional guidance that conform to best practices.  As noted above, DOE has one requirement that conforms to best  practices, the best practice of defining functional requirements based on  the mission need. NNSA followed this requirement for all three AOAs we  reviewed. Similarly, DOE has eight other elements of guidance that  conform to best practices (see table 11). For the three projects we  reviewed, NNSA followed from one to three of these eight elements of  guidance.
The specific elements of guidance NNSA followed for the AOAs differed  substantially by project. Specifically, NNSA followed three elements of  DOE guidance that conform to best practices for the High Explosive,  Science, Technology and Engineering Facility Project AOA and followed  two such elements of guidance for the Radioactive Liquid Waste  Treatment Facility project and only one such element of guidance for the  Uranium Processing Facility project AOAs. For example, the AOA for  NNSA’s High Explosive, Science, Technology and Engineering Project  followed the DOE guidance that suggests identifying and considering at  least three viable alternatives, including the status quo; the AOA identified  nine alternatives, including one that represented the status quo. In  contrast, the 2012 AOA for the Uranium Processing Facility identified six  alternatives, but none represented the status quo, and the Radioactive  Liquid Waste Treatment Facility project AOA identified two alternatives.
Table 11 below shows the extent to which NNSA’s AOAs for the projects  we reviewed followed DOE’s guidance that conform to best practices.
section_title: Conclusions
paragraphs: The AOA process is a key first step in the project management process at  which DOE and NNSA have the opportunity to put projects on the right  path forward. DOE has some requirements and guidance for this process  in its project management Order 413.3B and associated guides.
However, DOE’s AOA requirements in Order 413.3B conform to only 1 of  the 24 best practices we identified from different government and private- sector entities, and the additional suggestions provided in DOE’s guides  are still not sufficient to create a reliable AOA process. The AOAs for the  three recent NNSA projects we reviewed—the High Explosive Science,  Technology and Engineering Project, the Radioactive Liquid Waste  Treatment Facility, and the Uranium Processing Facility—did not conform  to best practices overall and may therefore not be reliable. DOE and  NNSA officials acknowledge that unreliable AOAs are a risk factor for  major cost increases and schedule delays for NNSA projects. Federal  standards for internal control related to risk assessment call for  management to decide on actions to mitigate identified risks. Developing  requirements that incorporate best practices would help DOE mitigate the  risk for major cost increases and schedule delays by creating a  framework to ensure a reliable AOA process. Developing reliable AOAs is  particularly important at this time because NNSA has recently reassessed  or is in the process of reassessing alternatives for all its major  construction projects, after spending billions of dollars on designing and  partially constructing these projects. Without a process to develop reliable  AOAs, NNSA may continue on this path and continue to have limited  assurance that it is selecting alternatives that best meet its mission needs  and will not result in major cost increases and schedule delays in the  future.
section_title: Recommendation for Executive Action
paragraphs: To minimize the risk of developing unreliable AOAs and incurring major  cost increases and schedule delays on projects, we recommend that the  Secretary of Energy direct DOE’s Office of Acquisition and Project  Management to update its project management order requirements to  incorporate best practices for conducting an AOA.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided DOE with a draft of this report for its review and comment. In  its written comments, reproduced in appendix VIII, DOE agreed with the  report’s recommendation. In its written comments, DOE stated that DOE’s  order for project management (DOE O 413.3B) will be assessed for  revision following the issuance of the revision to DOE-STD1189,  Integration of Safety into the Design Process, currently scheduled for  November 2016. DOE stated that it will consider the AOA best practices  when the policy is updated. In the interim, DOE stated that it has  strengthened guidance in the area of AOA, including issuing a Lifecycle  Cost Handbook, and that in fiscal year 2015 it will develop and issue an  AOA Handbook that will reflect the best practices for AOAs. DOE also  provided technical comments that were incorporated, as appropriate.
We are pleased that DOE agreed with our recommendation and that it will  take action to issue an AOA Handbook. However, while an AOA  Handbook may be a useful interim measure, the unspecified, open-ended  date for updating the project management order that contains  requirements (i.e., sometime after November 2016) and the statement  that the AOA best practices will be considered, not incorporated, may  indicate DOE’s lack of urgency in implementing this recommendation.
We are sending copies of this report to the appropriate congressional  committees, the Secretary of Energy, and other interested parties. In  addition, the report is available at no charge on the GAO website at  http://www.gao.gov.
If you or your staff have any questions about this report, please contact  me at (202) 512-3841 or trimbled@gao.gov. Contact points for our Offices  of Congressional Relations and Public Affairs may be found on the last  page of this report. GAO staff who made major contributions to this report  are listed in appendix IX.
section_title: Appendix I: Scope and Methodology
paragraphs: To identify and describe best practices for the analysis of alternatives  (AOA) process, we first searched for a source of generally accepted best  practices that we could use as criteria for comparison with the  Department of Energy’s (DOE) AOA process, but we could not identify a  single source that was broadly recognized by government and private- sector entities and that could serve as the definitive source of best  practices for the AOA process. In the absence of such a definitive source,  we identified AOA handbooks, guidebooks, requirements, and other AOA- related information from federal and private-sector entities. We also sent  a request to subject-matter experts, including experts from DOE, the  National Nuclear Security Administration (NNSA), and various other  government or private-sector entities, to help us identify further relevant  information. These subject-matter experts are part of GAO’s Projects  Controls Expert Working Group, which contains members with expertise  in program and project management, capital acquisition, cost estimation,  risk and sensitivity analysis, earned value management and scheduling,  and represent a diverse range of government and private-sector entities.
We reviewed the information related to the AOA process, including  information from our Cost Estimating Guide,Advancement of Cost Engineering International, Department of Defense,  Department of Homeland Security, National Aeronautics and Space  Administration, Office of Management and Budget, United States Air  Force, and a private-sector entity that specializes in acquisition  management. We compiled a draft set of best practices commonly  mentioned across these different entities’ AOA policies and guidance, and  we sent this draft set of AOA best practices to the experts for review in  advance of our semiannual meeting of the Projects Controls Expert  Working Group that took place in March 2014. More than 90 experts  participated, including officials from DOE and NNSA. We received  comments from some of these experts both during this meeting and by e- mail after the meeting. We developed a final set of 24 best practices for  the AOA process based on the comments received. We grouped these  best practices into four categories: (1) general principles, (2) identifying  alternatives, (3) analyzing alternatives, and (4) selecting a preferred  alternative.   the Association for the  To determine the extent to which DOE’s requirements and guidance for  conducting an AOA conform to AOA best practices, we first identified the  process NNSA is required to follow when conducting an AOA. To do so,  we reviewed DOE’s project management order and its associated guides,  as well as other documentation received from NNSA’s Office of  Acquisition and Project Management, including a compilation of DOE’s  requirements and guidance related to conducting an AOA. We also  interviewed officials from DOE’s Office of Acquisition and Project  Management, NNSA’s Office of Acquisition and Project Management,  NNSA’s Office of Cost Estimating and Program Evaluation, DOE’s Office  of Science, and a former DOE official familiar with DOE’s and NNSA’s  project management process, as well as NNSA and contractor officials  from the programs and projects we selected for review. We then  confirmed DOE’s requirements and guidance with DOE and NNSA  officials, from whom we received technical comments.
We performed two sets of analyses: we compared the best practices to  (1) DOE’s requirements and (2) DOE’s requirements combined with the  guidance. We used a five-point scoring system to determine the extent to   We first used this  which DOE’s AOA process conforms to best practices.scoring system to determine how well DOE’s requirements and guidance  conform to each best practice. We then used the average of the scores  for the best practices in each of the four categories to determine an  overall score for each category, and we then used the average of the  scores for the four categories as the final score for the overall DOE AOA  process. If the score for each best practice, the average score for each  category, or the final score for the AOA process was “fully met” or  “substantially met,” we concluded that the AOA process conformed to  best practices and therefore could be considered reliable. In contrast, if  the score was “partially met,” “minimally met,” or “not met,” we concluded  that the AOA process did not conform to best practices and therefore  could not be considered reliable. For us to consider the AOA process  reliable, the entire AOA process had to receive an average score of “fully  met” or “substantially met,” and each individual category—(1) general  principles, (2) identifying alternatives, (3) analyzing alternatives, and (4)  selecting a preferred alternative—had to receive an average score of  “fully met” or “substantially met” to ensure that the agency adequately and  consistently performed all parts of the AOA process. GAO does not  weight its cost estimating, scheduling, earned value management and  AOA best practices because it is not possible to quantitatively determine  the relative weights of each of the criteria. Weighting has the potential to  vary across programs, as well as due to where a program may be in its  different stages of its life cycle. Therefore the fairest standardized  methodology to evaluate programs against is to have all criteria weighted  equally.
To determine the extent to which NNSA conformed to best practices in  conducting the AOAs for recent NNSA projects, we used as criteria the  best practices identified as part of this engagement. We chose projects  for which NNSA had completed the AOA process since November 2010,  when DOE’s most recent version of the project management order came  into effect, or for which NNSA was scheduled to complete the AOA  process by the end of fiscal year 2014. We chose projects for which  NNSA had completed, or was nearing completion of, an AOA process  under this version of the order because this version included significant  revisions, including more specific and detailed requirements for  conducting an AOA. We also selected NNSA projects that were  scheduled to complete an AOA by the end of fiscal year 2014 because  most of the AOA documentation would be available for these projects. We  identified these projects by examining DOE’s data from its February 2014  Monthly Project Portfolio Status Report. The three projects listed in this  report that met our criteria were: (1) the High Explosive Science,  Technology and Engineering Project at NNSA’s Pantex site in Texas, for  which NNSA was scheduled to finish the AOA process by the end of  September 2014 at the time of our project selection; (2) the Radioactive  Liquid Waste Treatment Facility at NNSA’s Los Alamos National  Laboratory in New Mexico, for which NNSA completed its most recent  reassessment of alternatives in September 2013; and (3) the Uranium  Processing Facility at NNSA’s Y-12 site in Tennessee, for which NNSA  completed its most recent reassessment of alternatives in June 2012.
We checked with knowledgeable DOE and NNSA officials, reviewed  recent GAO reports, and compared the data in the project status report  with information received from NNSA on the projects to assess the  reliability of the data in this status report, and we determined that the data  were sufficiently reliable to ensure these projects met our criteria. For  each of these projects, we reviewed project documentation, and we  interviewed NNSA and contractor officials in charge of the AOA to  compare the process followed by NNSA with best practices. To score  each AOA process, (1) two GAO analysts separately examined the AOA  documentation received from the agency and then agreed on a score for  each of the 24 best practices, then, (2) an AOA specialist independent of  the engagement team reviewed the AOA documentation and the scores  assigned by the analysts for accuracy and cross-checked the scores in all  the analyses for consistency. After completing this process for each  individual best practice in all the analyses, we calculated the scores for  each category—(1) general principles, (2) identifying alternatives, (3)  analyzing alternatives, and (4) selecting a preferred alternative—and the  overall score for each analysis. We sent our analyses to DOE and NNSA  for review twice, and we used the same scoring process when we revised  the scores based on their technical comments and any additional  evidence received.
In April 2014, as we were conducting our engagement, NNSA released a  peer review report that recommended a new alternative for the Uranium  Processing Facility. We examined whether this peer review reflected any  characteristics of an AOA by interviewing NNSA officials and comparing  the peer review with the AOA best practices. In addition, as part of this  objective, we determined the extent to which NNSA conformed to certain  DOE requirements and guidance—those DOE requirements and  guidance that conformed to best practices—in conducting the AOAs for  these projects. We used the results of the earlier analysis comparing  DOE’s requirements and guidance to best practices to determine which  DOE requirements and guidance conformed to best practices. We then  reviewed project documentation and interviewed NNSA and contractor  officials in charge of the AOA process to assess the extent to which  NNSA projects followed DOE’s requirements and guidance that  conformed to best practices.
We conducted this performance audit from February 2014 to December  2014 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
section_title: Appendix II: The Department of Energy’s Analysis of Alternatives Process
paragraphs: The Department of Energy’s (DOE) analysis of alternatives (AOA)  process includes the following requirements from DOE Order 413.3B and  suggested guidance from seven guides associated with this order. The  source of the requirements or guidance is listed in parentheses at the end  of the item.
The Integrated Project Team—the team supporting the Federal  Project Director to deliver a capital asset project and conduct the  AOA—consists of professionals representing a diverse range of  disciplines and specific knowledge to successfully execute the  project. (O 413.3B p. C-11)
As part of the preconceptual design (CD-0) approval process:
The conceptual design must develop the scope required to  satisfy the program’s mission requirements and identify  requirements and features. A Mission Need Statement  document is developed to identify the mission need—the  gap that needs to be addressed—and functional  requirements—the general parameters that the selected  alternative must have to address the mission need. (O  413.3B pp. A-4, C-5)
The mission need must be independent of a particular  solution. (O 413.3B p. A-4)
The program office responsible for the capital asset project  must identify a credible performance gap between its  current capabilities and capacities and those required to  achieve the goals articulated in its strategic plan. (O  413.3B p. A-4)
The program office is afforded the flexibility to explore a  variety of solutions and not limit potential solutions. (O  413.3B p. A-4)
As part of the conceptual design (CD-1) approval process:
The conceptual design must develop reliable cost and  schedule range estimates for the alternatives considered.  The order also requires that whatever figure or range that  is provided at the CD-0 and CD-1 stages must explicitly  note relevant caveats concerning risks and uncertainties  inherent in early estimated given the immature  requirements definition at these stages. (O 413.3B pp. C-5,  C-16)
The conceptual design also must develop an assessment  of project risks and identification of appropriate risk  handling strategies. (O 413.3B p. C-4)
Develop a conceptual design report that includes, among  other things, a clear and concise description of the  alternatives analyzed, the basis for the selected  alternative, how the selected alternative meets the mission  need, the functions and requirements that define the  alternative and demonstrate the capability that the  alternative can be successful, and life-cycle cost  assumptions. (O 413.3B pp. A-7, C-5)
The order also requires that:
The project’s sponsor must never be the sole cost  estimator at any stage, and that the second cost estimator  must be from outside the chain of command, to avoid  conflict of interest. (O 413.3B pp. A-1, C-16)
Conduct several independent reviews during CD-0 and  CD-1 approval processes, depending on the estimated  cost of the project, related to two aspects of the AOA  process: the validation of the mission need statement and  of the cost estimates. (O 413.3B pp. A-7, C-7, C-17, C-18)
The selected alternative must be the optimum solution that  must provide the essential functions and capabilities at an  optimum life-cycle cost, consistent with required cost,  scope, schedule, performance, and risk considerations. (O  413.3B pp. A-2, A-5)
Additionally, DOE’s order requires that as a project moves toward  approval of CD-2, if the top end of the approved CD-1 cost range for the  selected alternative grows by more than 50 percent, the program must  reassess alternatives. (O 413.3B p. A-6)
The order also requires as part of CD-1 approval that an analysis of  alternatives must be conducted under the National Environmental Policy  Act of 1969 (NEPA). (O 413.3B p. A-7)
Summarize a planned approach to conduct an analysis of  alternatives. (G 413.3-17 p. 4)
The guides provide numerous examples of optional tasks for the  Integrated Project Team and the Federal Project Director to  perform as part of the AOA process, such as identifying functional  requirements, evaluating alternatives for satisfying the  requirements, conducting the appropriate analyses, and  recommending a preferred alternative. (G 413.3-1 pp. 5, 7, 14,  15) (G 413.3-5A p. 8) (G 413.3-13 pp. 2, 3, 17) (G 413.3-18A p.  8) (G-413.3-21 p. 9)
Explore concepts and consider alternatives for meeting the  mission need until a set of viable, affordable, and sustainable  alternatives is reached; consider at least three viable alternatives  for analysis, including one that represents the status quo; and  develop and list the primary advantages and disadvantages for  each alternative. (G-413.3-13 p. 9, G 413.3-21 p. B-2)
Develop cost estimates prior to the CD-2 approval milestone that  are explicit ranges instead of point estimates, including life-cycle  cost estimates of likely alternatives being considered that have a  broad accuracy range from a low of -20 to -50 percent to a high of  +30 to +100 percent. (G 413.3-21 p. 15)
Quantify the benefits of alternatives over their life cycle. (G 413.3- 13 p. 10)
Adjust life-cycle cost and benefit estimates for risk to ensure  consideration of the alternative with the best cost-benefit ratio and  generally the lowest life-cycle cost to the federal government. (G  413.3-21 p. 9)
Consider various selection criteria for the alternatives, including  cost and schedule, funding and budget, and technology and  engineering. (G 413.3-13 p. 9)
Weight the selection criteria used to select a preferred alternative  for relative importance and compare alternatives using net present  value or annuities. (G 413.3-13 p. 10; G 413.3-21 p. 69)
At the end of the analysis, the AOA team presents the  recommended alternative based on the preceding analysis in an  integrated form, summarizing why an alternative is preferred, and  supporting the recommendation of the preferred alternative with  facts from the analysis. (G 413.3-13 p. 13)
The program office performs an alternative selection and cost- range review to assess whether the AOA process (1) evaluates a  range of appropriate attributes for each alternative, including cost,  risks, safety, technology, and regulatory requirements and (2) is  reasonable and provides best value to the federal government.  (G 413.3-9 pp. 16-19)
section_title: Appendix III: Best Practices for the Analysis of Alternatives Process
paragraphs: We identified 24 best practices for identifying, analyzing, and selecting  alternatives. These practices can be applied to a wide range of activities  in which an alternative must be selected from a set of possible options,  and to a broad range of capability areas, projects, and programs. These  practices provide a framework to ensure that entities consistently and  reliably select the alternatives that best meet mission needs.
We grouped these 24 best practices into four categories: (1) general  principles, (2) identifying alternatives, (3) analyzing alternatives, and (4)  selecting a preferred alternative. The four categories of best practices  address the entire analysis of alternatives (AOA) process from defining  the mission need and functional requirements to independently reviewing  its results. On the basis of our reviews and experts’ comments, we  believe that these best practices can be generally applied from the  beginning of the AOA process with practices from the general principles  category, through practices in the identifying and analyzing alternatives  categories, and ending with practices in the selecting a preferred  alternative category. We also believe that these best practices do not  necessarily have to be followed in order and that some of them can be  applied concurrently with other best practices. For example, the best  practice of defining the selection criteria based on the mission need in the  selecting a preferred alternative category could be addressed at the same  time as the best practice of creating a study plan in the general principles  category.
Table 12 below provides a complete list of best practices.
section_title: Appendix IV: DOE’s AOA Requirements and Guidance Compared with Best Practices
paragraphs: Overall, the Department of Energy’s (DOE) requirements for analysis of  alternatives (AOA) in Order 413.3B minimally meet the best practices we  identified. DOE’s requirements combined with its guidance in the guides  associated with the order partially meet the best practices. Table 13  below describes our analysis of DOE’s AOA requirements and guidance  compared with best practices.
section_title: Appendix V: NNSA’s Analysis of Alternatives Conducted at the High Explosive, Science, Technology and Engineering Project Compared with Best Practices
paragraphs: Overall, the National Nuclear Security Administration’s (NNSA) analysis  of alternatives (AOA) conducted for the High Explosive Science,  Technology and Engineering Project partially met best practices. The  mission need for this project—to replace aging high explosive facilities at  NNSA’s Pantex site in Texas—was approved in November 2011. NNSA  expects to complete the AOA process by obtaining CD-1 in December  2014. As of September 2014, NNSA estimated that the total project cost  would range from $100 to $155 million at an 85 percent confidence level.  NNSA expects to finalize the AOA process by the end of 2014. Table 14  below compares the AOA conducted at the High Explosive, Science,  Technology and Engineering Project with AOA best practices.
section_title: Appendix VI: NNSA’s 2013 Analysis of Alternatives Conducted at the Radioactive Liquid Waste Treatment Facility Compared with Best Practices
paragraphs: Overall, the National Nuclear Security Administration’s (NNSA) analysis  of alternatives (AOA) conducted for the Radioactive Liquid Waste  Treatment Facility project partially met best practices. The mission need  for this project—to replace the current, aging facility—was approved in  October 2004. NNSA approved an initial AOA for this project in 2006, and  after substantial cost increases, conducted a second AOA (analyzed  here) in 2013. NNSA currently estimates the project will cost between  $168 million and $220 million. Table 15 below compares the AOA  conducted at the Radioactive Liquid Waste Treatment Facility with AOA  best practices.
section_title: Appendix VII: NNSA’s 2012 Analysis of Alternatives Conducted at the Uranium Processing Facility Compared with Best Practices
paragraphs: Overall, the National Nuclear Security Administration’s (NNSA) analysis  of alternatives (AOA) conducted for the Uranium Processing Facility  (UPF) project partially met best practices. The mission need for this  project—to consolidate and replace existing enriched uranium processing  capabilities at NNSA’s Y-12 site—was approved in 2004. NNSA  conducted a first AOA for this project in 2007, and after substantial cost  increases, conducted a second AOA (analyzed here) in 2012. NNSA  estimated the project will cost between $4.2 billion and $6.5 billion. Table  16 below compares the AOA conducted at the Uranium Processing  Facility with AOA best practices.
section_title: Appendix VIII: Comments from the Department of Energy
section_title: Appendix IX: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the individual named above, Daniel Feehan, Assistant  Director; Bethany Benitez; Antoinette Capaccio; Jennifer Echard; Alex  Fedell; Cristian Ion; Kirsten Lauber; Jennifer Leotta; Cynthia Norris; Karen  Richey; Dan C. Royer; and Kiki Theodoropoulos made key contributions  to this report.