DOE and NNSA Project Management: Analysis of Alternatives Could Be Improved by Incorporating Best Practices
Released: 2014-12-11T00:00:00

Letter

Background

This section describes DOE’s requirements and guidance for AOA; DOE and NNSA offices responsible for conducting AOAs for capital asset projects; and NNSA’s recent projects that have completed, or are nearing completion of, an AOA.

DOE’s Requirements and Guidance for Analysis of Alternatives

DOE’s mandatory requirements and optional guidance for identifying, analyzing, and selecting alternatives when conducting AOAs also apply to NNSA. DOE’s requirements for the AOA process are outlined in Order 413.3B governing the Program and Project Management for the Acquisition of Capital Assets (hereafter referred to as Order 413.3B). This order includes requirements for the acquisition of capital assets, with the stated goal of delivering fully capable projects within the planned cost, schedule, and performance baseline. In addition to the order’s requirements, DOE has guidance for identifying, analyzing, and selecting alternatives (hereafter referred to as “guidance”) that is found throughout seven guides associated with the order. DOE states at the beginning of each of these guides that they include nonmandatory approaches for meeting requirements, that guides are not requirements documents, and that they are not to be construed as requirements in any audit for appraisal of compliance with the parent policy or order.

DOE’s Order 413.3B establishes five critical decision processes of project development that each end with a major approval milestone—or “critical decision point”; these decision processes cover the life of a project. Under Order 413.3B, an AOA occurs during the span of the first two critical decision processes—the preconceptual design process (CD-0) and the conceptual design process (CD-1)—with the majority of the AOA being conducted during the conceptual design process and ending with CD-1 approval. Figure 1 illustrates when DOE conducts the analysis of alternatives as part of its project management process.

DOE’s Order 413.3B and its associated guides include the AOA-related requirements and guidance listed below, among others. Appendix II includes a full description of DOE’s requirements and guidance for conducting an AOA.

As part of the preconceptual design (CD-0) approval process, the mission need—which DOE defines in Order 413.3B as a credible gap between current capabilities and those required to meet the goals articulated in the strategic plan—and functional requirements—the general parameters that the selected alternative must have to address the mission need—must be identified. The mission need must be independent of a particular alternative, and the program office responsible for the capital asset project must explore a variety of alternatives.

As part of the conceptual design (CD-1) approval process, reliable cost and schedule range estimates for the alternatives considered must be developed, and whatever figure or range is provided at the CD-0 and CD-1 stages must explicitly note relevant caveats concerning risks and uncertainties inherent in early estimates. In addition, a conceptual design report must be developed that includes, among other things, a clear and concise description of the alternatives analyzed, the basis for the selected alternative, how the selected alternative meets the mission need, the functional requirements that define the alternative and demonstrate that the alternative can be successful, and life-cycle cost assumptions.

During the CD-0 and CD-1 approval process, several independent reviews must be conducted, depending on the estimated cost of the project, related to two aspects of the AOA process: validation of (1) the mission need statement and (2) the cost estimates.

As a project moves toward approval of CD-2, if the top end of the approved CD-1 cost range for the selected alternative grows by more than 50 percent, the program office must reassess alternatives by conducting another AOA and obtaining another CD-1 approval.

DOE’s guides suggest summarizing a planned approach to conduct an analysis of alternatives; considering at least three viable alternatives for analysis, including one that represents the status quo; developing cost estimates that are explicit ranges instead of point estimates; including life-cycle cost estimates of the alternatives being considered; quantifying the benefits of alternatives over their life cycle; adjusting life-cycle cost and benefit estimates for risk; considering various selection criteria for the alternatives; weighting the selection criteria for relative importance; and comparing alternatives using net present value.The guidance suggests that, at the end of the analysis, the AOA team present the recommended alternative based on the preceding analysis in an integrated form, summarizing why an alternative is preferred and supporting the recommendation of the preferred alternative with facts from the analysis. In addition, DOE’s guidance suggests that the program office review the alternative selection and cost range to assess whether the AOA process (1) evaluates a range of appropriate attributes for each alternative, including cost, risks, safety, technology and regulatory requirements, and (2) is reasonable and provides best value to the government.

Order 413.3B also requires prior to CD-1 approval that DOE conduct a separate analysis of alternatives under the National Environmental Policy Act of 1969 (NEPA). Under NEPA, federal agencies must identify and assess the likely environmental effects of proposed projects, and reasonable alternatives, using an environmental assessment or, if the projects likely would significantly affect the quality of the human environment, a more detailed environmental impact statement. Recognizing the need for better integration between the AOA conducted under Order 413.3B and the AOA conducted under NEPA, in early 2012, DOE established a multidisciplinary team of NEPA-compliance and program- and project-management specialists, and in June 2012, the Secretary of Energy issued a memo highlighting the importance of integrating these two AOAs. DOE officials who were part of this team stated that DOE has not yet decided how to connect the NEPA process with the AOA process under Order 413.3B.

DOE and NNSA Offices Responsible for Conducting AOAs for Capital Asset Projects

Within DOE and NNSA, several groups are responsible for various aspects of conducting an AOA:

DOE Office of Acquisition and Project Management (DOE APM). This office is responsible for, according to DOE APM officials, writing DOE’s project management Order 413.3B and its associated guides, including requirements and guidance for conducting an AOA, and for reviewing some DOE projects.

NNSA Office of Acquisition and Project Management (NNSA APM). This office is responsible for managing construction of capital asset projects within approved cost and schedule estimates, and for conducting at least yearly reviews of the construction projects to evaluate technical, cost, scope, and other aspects of the projects. According to NNSA APM’s Concept of Operations, this office leads the development of the analysis of alternatives for NNSA projects.

NNSA program offices. Program offices are responsible for the projects that fall within their portfolios. For projects, according to NNSA APM’s Concept of Operations, the program offices lead the development of the mission needs and functional requirements, support the analysis of alternatives, and lead the selection of the preferred alternative.

NNSA Office of Cost Estimating and Program Evaluation. The National Defense Authorization Act for Fiscal Year 2014 mandated the permanent establishment of this office to provide an independent review of cost estimates and to advise the NNSA Administrator on AOA policies and procedures, among other responsibilities. NNSA officials stated that they expected this office to work with other NNSA offices to define its roles and responsibilities as they relate to conducting AOAs during fiscal year 2015.

NNSA’s Recent Projects That Have Completed, or Are Nearing Completion of, an AOA

The following describes the three recent NNSA projects that we reviewed that had completed, or are nearing completion of an AOA.

High Explosive Science, Technology and Engineering Project

This project is intended to replace and upgrade existing facilities used for NNSA’s high explosive manufacturing support, surveillance, testing, and technology development program. NNSA began conducting the AOA in November 2011 and expects to complete the AOA process by obtaining CD-1 approval in December 2014. As of September 2014, NNSA estimated that the total project cost would range from $100 million to $155 million at an 85 percent confidence level.

Radioactive Liquid Waste Treatment Facility

This project is intended to replace an existing facility at the Los Alamos National Laboratory to treat and dispose of two types of radioactive waste. According to this project’s mission need statement document, the existing facility was nearing its designed life span and had suffered numerous failures. NNSA completed an initial AOA in 2006, selecting the alternative to place the treatment capabilities for both types of waste in the same structure. NNSA estimated in 2006 that the total project cost would likely range between $82 million and $104 million. After significant cost increases were incurred during the preliminary design process of the project, NNSA completed a reassessment of alternatives in September 2013, which we examine in this report. NNSA indicated in this reassessment that the existing facility was beyond its designed life span. The selected alternative for this reassessment separates treatment capabilities into two separate structures, and NNSA estimated that the revised total project cost would likely range between $168 million and $220 million.

Uranium Processing Facility

This project is intended to replace existing facilities for enriched uranium capabilities that, according to this project’s mission need statement document, require intensive maintenance and are experiencing escalating operating costs. NNSA completed an initial AOA in 2007, selecting the alternative to construct a new building with full capability to process enriched uranium. NNSA estimated in 2007 that the total project cost would likely range between $1.4 billion and $3.5 billion. During the preliminary design phase, the estimated cost of the project increased significantly, and NNSA completed a reassessment of alternatives in June 2012. This second AOA, which we examine in this report, selected a new alternative to construct the same building, but included fewer capabilities, while deferring significant portions of the project’s original scope to be added in the facility at a later time. The approved estimated cost increased to a range of $4.2 billion to $6.5 billion.

This independent cost estimate was requested under the National Defense Authorization Act of 2013. The $9.8 billion is a point estimate and not the full cost range. The top end of the Department of Defense’s Office of Cost Assessment and Program Evaluation cost range estimate was at least $10.5 billion with the potential of higher cost depending on constrained or unconstrained funding profile.

NNSA and DOE senior management a new alternative of building two new, smaller facilities to include all capabilities approved in the 2012 AOA, and senior management approved this new alternative. NNSA began to consider alternatives other than building a single facility because, according to the UPF Federal Project Director, the project was facing budget constraints, rising costs, and competition from other high- priority projects within NNSA. In January 2014, the NNSA Administrator asked for a peer review to develop and recommend an alternative approach to UPF. In April 2014, the peer reviewers recommended an alternative similar to the one approved by DOE and NNSA senior management in December 2013. This alternative would require NNSA to construct two new, smaller facilities and upgrade existing facilities to cover the same capabilities approved in the 2012 AOA, all within the $6.5 billion approved top end of the cost range. According to NNSA officials, this peer review was not an AOA. According to the NNSA Deputy Federal Project Director for this project, the Federal Project Director provided direction in April 2014 to proceed with the alternative proposed by the peer review. According to NNSA officials in October 2014, the alternative was further modified. At this time, NNSA’s plan was to build three new, smaller facilities and upgrade existing facilities to provide the same capabilities approved in 2012 and still remain within the approved cost range of $4.2 billion to $6.5 billion.

GAO Identified 24 Best Practices for the Analysis of Alternatives Process

Because no one single set of practices existed that was broadly recognized by government and private-sector entities for the AOA process, we identified 24 best practices for identifying, analyzing, and selecting alternatives. We identified these best practices by (1) compiling and reviewing commonly mentioned AOA policies and guidance used by different government and private-sector entities and (2) incorporating experts’ comments on our draft set of practices to develop a final set of practices. These practices can be applied to a wide range of activities in which an alternative must be selected from a set of possible options, as well as to a broad range of capability areas, projects, and programs. These practices can provide a framework to help ensure that entities consistently and reliably select the project alternatives that best meet mission needs.

We grouped these 24 best practices into four categories: (1) general principles, (2) identifying alternatives, (3) analyzing alternatives, and (4) selecting a preferred alternative. The four categories of best practices address the AOA process from defining the mission need and functional requirements to independently reviewing its results. On the basis of our reviews and experts’ comments, we believe that these best practices can be generally applied from the beginning of the AOA process with practices from the general principles category, through practices in the identifying and analyzing alternatives categories, and ending with practices in the selecting a preferred alternative category. We also believe that these best practices do not necessarily have to be followed in order and that some of them can be applied concurrently with other best practices. For example, the best practice of defining the selection criteria based on the mission need in the selecting a preferred alternative category could be addressed at the same time as the best practice of creating a study plan in the general principles category. The following sections describe the 24 best practices by category.

General Principles Category

The general principles category contains best practices that would need to be applied before starting the process of identifying, analyzing, and selecting alternatives, such as determining the mission need and functional requirements, developing the study time frame and creating a study plan, and determining who conducts the analysis. It also includes best practices that would need to be applied throughout the AOA process, such as documenting all steps taken to identify, analyze, and select alternatives in a single document. Table 1 lists the best practices in the general principles category.

Identifying Alternatives Category

The identifying alternatives category contains best practices that help ensure the alternatives to be analyzed are sufficient, diverse, and viable. Table 2 lists the best practices in the identifying alternatives category.

Analyzing Alternatives Category

The analyzing alternatives category contains best practices that are necessary to compare the alternatives selected for analysis. The best practices in this category help ensure that the team conducting the analysis uses a standard, quantitative process to assess the alternatives. This category includes best practices related to estimating the costs and benefits of each alternative over its life cycle and understanding the impacts of risks and key assumptions on these estimates. Table 3 lists the best practices in the analyzing alternatives category.

Selecting a Preferred Alternative Category

The category for selecting a preferred alternative contains best practices that help ensure the team selects a preferred alternative that best meets the mission needs. This category includes best practices related to defining selection criteria, differentiating among the selection criteria, and independently reviewing the AOA process. Table 4 lists the best practices in this category.

On the basis of our reviews and experts’ comments, we believe that conforming to these best practices helps ensure that the preferred alternative that is selected is the one that best meets the agency’s mission needs. Not conforming to best practices may lead to an unreliable AOA, and the agency will not have assurance that the preferred alternative best meets its mission needs. Appendix III lists the best practices from all four categories.

DOE’s AOA Requirements and Guidance Do Not Conform to Best Practices

Neither DOE’s AOA requirements nor its guidance conform to best practices and, therefore, DOE does not have assurance that applying these requirements and guidance may lead to reliable AOAs. Our review of DOE’s requirements contained within Order 413.3B found that they minimally meet best practices overall, and that they do not fully or substantially meet best practices in any of the four categories. Our analysis also found that, even when DOE’s requirements are combined with guidance contained in the guides associated with Order 413.3B, they only partially meet best practices overall, and they only fully or substantially meet best practices in the identifying alternatives and selecting a preferred alternative categories but not in the general principles or analyzing alternatives categories.

DOE’s Order 413.3B Requirements Do Not Conform to Best Practices

DOE’s Order 413.3B requirements for the AOA process minimally meet best practices. Therefore, DOE does not have assurance that applying these requirements may lead to reliable AOAs. DOE’s requirements do not fully or substantially meet best practices in any of the four AOA categories, and they conform to only 1 of the 24 best practices we identified—the practice of having the customer define functional requirements based on the mission need. Our assessment of DOE’s Order 413.3B requirements found the following:

General principles. DOE’s Order 413.3B requirements minimally meet best practices for the general principles category by substantially meeting 1 of the 8 best practices in this category, partially or minimally meeting 4, and not meeting the other 3. DOE requires that the program office define a project’s mission need without a predetermined solution and that the Integrated Project Team must represent a diverse range of disciplines, develop functional requirements to satisfy the mission need, and document some of the steps and assumptions required as part of conducting the AOA. DOE’s Order 413.3B does not make any reference to appropriate AOA time frames, an AOA study plan, or conducting the analysis without a predetermined solution.

Identifying alternatives. DOE’s Order 413.3B requirements minimally meet best practices for identifying alternatives by partially or minimally meeting 2 of the 4 best practices in this category and not meeting the other 2. DOE requires that the project’s customer, not the AOA team, explore a variety of alternatives and that the alternatives be defined clearly and concisely. DOE’s Order 413.3B does not make any reference to including a status quo alternative in the AOA or pre- screening the list of identified alternatives for viability.

Analyzing alternatives. DOE’s Order 413.3B requirements minimally meet best practices for analyzing alternatives by partially or minimally meeting 3 of the 8 best practices in this category and not meeting the other 5. DOE requires that life-cycle cost assumptions be included in the conceptual design report, which leaves room for interpretation because it could be interpreted as requiring the inclusion of life-cycle cost estimates for all alternatives or, as DOE and NNSA officials said that they interpret this statement, as requiring a life-cycle cost estimate for only the selected alternative after completion of the AOA process. DOE also requires that project risks and mitigation strategies are assessed. DOE’s Order 413.3B does not make any reference to presenting life-cycle cost estimates in present value terms, quantifying benefits, explaining how measures of benefit support the mission need, or testing the sensitivity of cost and benefit estimates to risks and changes in key assumptions.

Selecting a preferred alternative. DOE’s Order 413.3B requirements do not meet best practices for selecting a preferred alternative by minimally meeting 1 of the 4 best practices in this category and not meeting the other 3. The best practice that DOE minimally meets is having an entity independent of the AOA process reviewing the extent to which all best practices have been followed. Order 413.3B contains four separate reviews that might be required during CD-0 and CD-1 processes depending on the amount of the project’s cost estimate, but these reviews are limited to two aspects of the AOA process: the validation of (1) the mission need statement and (2) the cost estimates. In addition, one of these reviews is not an independent review because it is conducted by the program office, which is on the project’s chain of command. Table 5 lists these required reviews. DOE’s requirements do not meet the other best practices in this category because DOE’s Order 413.3B does not make any reference to defining selection criteria based on the mission need; weighting selection criteria to reflect the relative importance of each; or comparing the alternatives considered, using net present value.

Table 6 and appendix IV summarize our assessment of DOE’s requirements.

DOE’s Requirements Combined with Guidance Do Not Conform to Best Practices

DOE’s Order 413.3B AOA requirements combined with guidance partially meet best practices. Therefore, DOE does not have assurance that applying its requirements combined with its guidance may lead to reliable AOAs. DOE’s requirements combined with guidance fully or substantially met best practices in the identifying alternatives and selecting a preferred alternative categories, but not in the general principles or analyzing alternatives categories. DOE’s requirements combined with guidance conform to 9 of 24 best practices. Our assessment of DOE’s requirements combined with guidance found the following:

General principles. DOE’s requirements combined with guidance partially meet best practices for the general principles category by fully or substantially meeting 3 of the 8 best practices in this category, partially or minimally meeting 3, and not meeting the other 2. DOE’s guidance supplements the requirements discussed in the previous section by suggesting that the Federal Project Director and the Integrated Project Team conduct different parts of the AOA, but the guidance does not suggest specific areas of expertise to be represented on the team. The guidance also suggests that a study plan be developed but does not suggest what to include in that plan. Neither DOE’s requirements nor guidance refer to appropriate AOA time frames or conducting the analysis without a predetermined solution.

Identifying alternatives. DOE’s requirements combined with guidance substantially meet best practices for the identifying alternatives category by fully meeting 1 of the 4 best practices in this category and partially or minimally meeting the other 3. DOE’s guidance supplements the requirements by suggesting that at least three alternatives be considered, including one representing the status quo, and by listing the advantages and disadvantages of each alternative. Additionally, DOE’s guidance suggests pre-screening the alternatives for viability, but the guidance does not suggest documenting the reasons for eliminating any alternatives.

Analyzing alternatives. DOE’s requirements combined with guidance partially meet best practices for the analyzing alternatives category by fully or substantially meeting 3 of the 8 best practices in this category, partially or minimally meeting 3, and not meeting the other 2. DOE’s guidance supplements the requirements by suggesting that life-cycle cost estimates be developed for each of the alternatives, the life-cycle cost estimates be presented as ranges instead of point estimates, life-cycle benefit estimate be developed for each alternative, risk be taken into consideration when evaluating alternatives, and life-cycle cost estimates be adjusted for risk. Neither DOE’s requirements nor guidance refer to presenting life-cycle cost estimates in present value terms, explaining how each measure of benefit supports the mission need, or testing the sensitivity of the cost and benefit estimates to changes in key assumptions.

Selecting a preferred alternative. DOE’s requirements combined with guidance substantially meet best practices for the selecting a preferred alternative category by fully or substantially meeting 2 of the 4 best practices in this category and partially or minimally meeting the other 2. DOE’s guidance supplements the requirements by suggesting that a variety of discriminators are considered when selecting a preferred alternative, the selection criteria are weighted to reflect the relative importance of each, and the alternatives are compared using net present value. The guidance also suggests another review of more aspects of the AOA process, such as whether the alternative selection process evaluates a range of appropriate attributes for each alternative including cost, maintainability, safety, technology requirements, risks, and regulatory requirements, but the guidance suggests that this review be performed by the program office and not by a party independent of the project. Table 7 below and appendix IV summarize our assessment of DOE’s requirements combined with guidance.

DOE and NNSA officials told us that unreliable AOAs are a risk factor for major cost increases and schedule delays for NNSA projects. DOE APM and NNSA APM officials acknowledged that DOE does not have definitive guidance on conducting AOAs. According to these officials, the absence of adequate guidance has led to some projects pursuing alternatives that subsequently proved to be unaffordable. For example, each of NNSA’s ongoing major projects (e.g., the Chemistry and Metallurgy Research Replacement Nuclear Facility, the Mixed Oxide Fuel Fabrication Facility, or the Uranium Processing Facility) has undergone reassessments of alternatives due to cost increases and schedule delays. As we previously reported, federal standards for internal control related to risk assessment call for agency management to assess the risks faced entity-wide and at the activity level, and that once risks have been identified, management should decide what actions should be taken to mitigate them.developing a reliable AOA process, DOE may not be successful in mitigating the risk it has identified related to this process.

NNSA’s AOAs for Three Recent Projects Did Not Conform to Best Practices and Did Not Consistently Follow Certain DOE Guidance

For three recent projects, our analysis found that NNSA did not conform to most of the best practices in conducting the AOAs,concerns about the reliability of these AOAs. NNSA also did not consistently follow certain DOE optional guidance for these AOAs.

NNSA’s AOAs for the Three Recent Projects Did Not Conform to Best Practices

NNSA did not conform to best practices overall in conducting the AOAs for the three recent projects we examined—the High Explosive Science, Technology and Engineering Project, the Radioactive Liquid Waste Treatment Facility, and the Uranium Processing Facility, therefore, raising concerns about the reliability of these AOAs. NNSA also conducted each AOA differently. (See apps. V through VII for more detailed summaries of our assessment of the AOAs for each project.)

High Explosive Science, Technology and Engineering Project AOA

For the High Explosive, Science, Technology and Engineering Project, NNSA partially met best practices overall in conducting the AOA. Therefore, this AOA may not be reliable. NNSA fully or substantially met best practices in the general principles and identifying alternatives categories but not in the analyzing alternatives or selecting a preferred alternative categories. NNSA conformed to 11 of 24 best practices when conducting this AOA. Our assessment of NNSA’s AOA for this project identified the following:

General principles. NNSA substantially met best practices for the general principles category by fully or substantially meeting 4 of the 8 best practices in this category and by partially or minimally meeting the other 4. NNSA convened a team, which included federal and contractor project officials to conduct the analysis. Before conducting the analysis, NNSA developed a 17-step plan for conducting the AOA. When conducting the analysis, the team described in project documentation many, but not all, of the steps it took, and listed most of the assumptions made in the analysis but did not justify them. NNSA also undertook the analysis having an “originally envisioned” alternative. NNSA ultimately did not select this alternative as the preferred alternative, but NNSA did not explain how the analysis was conducted without having a predetermined solution.

Identifying alternatives. NNSA fully met best practices for the identifying alternatives category by fully or substantially meeting all 4 best practices in the category. NNSA identified nine alternatives that represented a diverse range of potential solutions to the mission need, including one that represented the status quo. NNSA defined these alternatives in detail, including providing descriptions of the specific characteristics used to create cost estimates. Before proceeding to the analysis, NNSA eliminated four alternatives it did not consider viable and described the general reasons for eliminating them but did not provide specific reasons for the scores it gave each alternative as part of the screening process.

Analyzing alternatives. NNSA minimally met best practices for the analyzing alternatives category by fully meeting 1 of the 8 best practices in this category, partially or minimally meeting 4, and not meeting the remaining 3. To assess the alternatives, NNSA developed cost estimates for each alternative, and presented these cost estimates with a stated accuracy range of 15 percent below to 50 percent above, but they were not life-cycle cost estimates because NNSA did not include such costs as decontamination and decommissioning in the estimates because it assumed those costs would be identical among the alternatives. NNSA did not develop quantitative estimates of the benefits that would result from each alternative, and mentioned the benefits of the alternatives only qualitatively through a discussion of some of the pros and cons of each alternative. To assess the risks associated with the alternatives, NNSA developed a list of risks for each alternative, but it did not develop potential mitigation strategies for all of the risks and did not test the sensitivity of the cost estimates to these risks or to changes in key assumptions.

Selecting a preferred alternative. NNSA partially met best practices for the selecting a preferred alternative category by fully or substantially meeting 2 of the 4 best practices in this category, partially meeting 1 and not meeting 1. To select a preferred alternative, NNSA developed 21 selection criteria such as life-cycle costs, schedules, risks, flexibility, and ability to meet the mission need. NNSA then weighted these criteria on a five-point scale to reflect their relative importance, and selected the alternative that received the highest overall score. As of June 2014, NNSA planned to conduct an independent project review to assess, among other things, whether the selected preferred alternative remained the most viable and feasible alternative for meeting the approved mission need, but it did not plan on assessing other parts of the AOA process, including the identification of alternatives, the criteria used for selecting the alternative, or the benefit estimates for each alternative. Table 8 below and appendix V summarize our assessment of the AOA conducted for this project.

For the Radioactive Liquid Waste Treatment Facility project, NNSA partially met best practices overall in conducting the most recent AOA in 2013. Therefore, this AOA also may not be reliable. NNSA fully or substantially met best practices in the general principles and identifying alternatives categories but not in the analyzing alternatives or selecting a preferred alternative categories. NNSA conformed to 8 of 24 best practices when conducting this AOA. Our assessment of the AOA conducted for this project identified the following:

General principles. NNSA substantially met best practices for the general principles category by fully or substantially meeting 3 of the 8 best practices in this category and by partially or minimally meeting the other 5. NNSA convened a team consisting of project management and technical experts. NNSA developed detailed study documentation for some areas of the AOA process, such as for risk and cost analysis, but not for others, such as for describing how NNSA screened alternatives for viability or how NNSA developed the selection criteria. NNSA also listed but did not justify the assumptions used in the AOA report, and did not discuss any potential constraints associated with the analysis. This analysis also may have been conducted with having a predetermined solution because one of the two alternatives considered was developed from a previous design that had already incurred significant cost increases and so was unlikely to be selected.

Identifying alternatives. NNSA substantially met best practices for the identifying alternatives category by fully or substantially meeting 2 of the 4 best practices in the category, and by partially or minimally meeting the other 2. NNSA identified thirty-seven options for various pieces of this project, and then combined those options into two complete alternatives to be analyzed: one that was based on the design selected in the 2006 AOA and another with a design similar to a previously-considered alternative. This approach compared an alternative developed from a design that had already incurred cost increases significant enough to warrant a reassessment of alternatives to only one other alternative. NNSA did, however, define the two alternatives in detail, assess the viability of the status quo and screen the list of thirty-seven options for viability before combining them into the two final alternatives.

Analyzing alternatives. NNSA partially met best practices for the analyzing alternatives category by fully or substantially meeting 3 of the 8 best practices in this category, partially or minimally meeting 2, and not meeting 3. To assess the alternatives, NNSA developed full life-cycle cost estimates for both of the alternatives, which it presented in present-value terms, but which it did not present with a confidence interval or range. NNSA did not develop quantitative estimates of the benefits that would result from each alternative, and did not explain how any measure of benefit used would support the mission need. To assess the risks associated with the alternatives, NNSA developed a list of risks and mitigation strategies for each alternative, and NNSA adjusted the total project cost estimates for each alternative, but not the life-cycle cost estimates, for these risks. NNSA did not, however, test the sensitivity of the full life-cycle cost estimates to risks or changes in key assumptions.

Selecting a preferred alternative. NNSA minimally met best practices for the selecting a preferred alternative category by partially or minimally meeting 2 of the 4 best practices in this category and by not meeting the other 2. To select a preferred alternative, NNSA compared the life-cycle cost estimates, the total project cost estimates, estimated schedules, and the scope of the alternatives. NNSA did not weight these selection criteria and did not compare the alternatives using net present value. NNSA conducted an independent project review of only the selected alternative, not of each alternative considered. NNSA also conducted a cost estimate review of total project cost estimates, not the life-cycle cost estimates, of each alternative. Table 9 below and appendix VI summarize our assessment of the AOA conducted for this project.

For the Uranium Processing Facility project, NNSA partially met best practices overall in conducting the most recent AOA in 2012. Therefore, this AOA also may not be reliable. NNSA fully or substantially met best practices in the identifying alternatives category but not in the general principles, analyzing alternatives, or selecting a preferred alternative categories. NNSA conformed to 6 of 24 best practices when conducting this AOA. Our assessment of NNSA’s 2012 AOA for this project identified the following observations:

General principles. NNSA partially met best practices for the general principles category by fully or substantially meeting 2 of the 8 best practices in this category and by partially or minimally meeting the other 6. NNSA convened a team composed of federal project management staff and contractors to conduct the AOA, and they conducted the bulk of the analysis over 3 days in January 2012. NNSA did not create a detailed plan for conducting the AOA. Additionally, NNSA conducted the analysis for this project with, according to NNSA officials, a fundamental assumption that the selected alternative would be a single-facility solution such as the alternative selected in the project’s previous AOA conducted in 2007.

Identifying alternatives. NNSA substantially met best practices for the identifying alternatives category by fully meeting 1 of the 4 best practices in the category, and by partially or minimally meeting the other 3. NNSA identified six alternatives, all of which were variations of the single-facility solution because, according to project officials, they were “wedded” to that idea. NNSA also did not consider the status quo as an alternative. NNSA did determine that one alternative was not viable before proceeding with the analysis but did not document the screening of all the alternatives against viability criteria. NNSA described the alternatives in sufficient detail.

Analyzing alternatives. NNSA minimally met best practices for the analyzing alternatives category by fully meeting 1 of the 8 best practices in the category, partially or minimally meeting 5, and by not meeting the other 2. To assess the alternatives, according to UPF project officials, NNSA developed relative cost estimates for each alternative that were updates of a 2010 total project cost estimate and that were not life-cycle cost estimates because they did not include costs that were the same across all alternatives. NNSA also did not develop quantitative estimates of the benefits that would result from each alternative. To assess the risks associated with the alternatives, NNSA listed categories of risk, such as cost, schedule, and technical risks in the description of each alternative but did not discuss specific risks or mitigation strategies. NNSA adjusted the design and construction elements of the cost estimates based on the risks identified in developing the 2010 total project cost estimate, but NNSA did not document these risks in project documentation and did not test the sensitivity of the estimates to changes in key assumptions.

Selecting a preferred alternative. NNSA partially met best practices for the selecting a preferred alternative category by fully or substantially meeting 2 of the 4 best practices in this category and by partially or minimally meeting the other 2. To select a preferred alternative, according to UPF project officials, NNSA used commercially available software to compare each alternative to every other alternative. NNSA used six selection criteria, including life-cycle cost, execution flexibility, and ability to achieve benefits from modernizing and consolidating enriched uranium processing, but NNSA did not compare the alternatives using net present value. NNSA weighted these criteria on a percentage scale and selected the alternative that scored the highest. NNSA also conducted an independent review after conducting the AOA to determine if the project was technically ready, but this was not a review of the AOA process. Table 10 below and appendix VII summarize our assessment of the AOA conducted for this project.

NNSA conducted each AOA differently and applied best practices differently among the projects. For example, as mentioned above in the description of the AOAs conducted at these projects, NNSA conducted the risk analysis differently for each project. The different ways that NNSA conducted the AOAs for the projects we examined may be due to the nonprescriptive nature of the Order 413.3B requirements. NNSA project officials stated that the AOA requirements in Order 413.3B are very high level and do not prescribe specific methods to follow. According to these officials, Order 413.3B provides a very general set of requirements for what the AOA process should accomplish. For example, DOE’s Order 413.3B states that life-cycle cost assumptions should be included in the conceptual design report. The word “assumptions” leaves room for interpretation because it could be interpreted as requiring the inclusion of life-cycle cost estimates for all alternatives or as requiring a life-cycle cost estimate for only the selected alternative after completion of the AOA process. In the absence of specific requirements, each AOA team developed its own approach to conducting an AOA. For example, NNSA officials who worked on the AOA for the High Explosive, Science, Technology and Engineering Project stated that they had to develop much of the AOA process themselves and that they based their process on historical AOAs conducted at the Pantex site in Texas. Officials working on the AOA for the Radioactive Liquid Waste Treatment Facility project at Los Alamos Nuclear Laboratory told us that they also used site- specific guidance including the laboratory’s engineering standards manual as well as their own professional experience. In contrast, officials working on the Uranium Processing Facility project stated that they conducted the AOA based on guidance contained within a commercially available, off-the-shelf software suite.

In addition, there may be some confusion about roles and responsibilities for leading the AOA process at NNSA. NNSA officials in the Office of Acquisition and Project Management, who are responsible for managing the construction process, stated that their office does not have lead responsibility for the AOA process. However, the NNSA Concept of Operations for the Office of Acquisition and Project Management states that, while the program offices lead the development of the mission need and functional requirements and the selection of the preferred alternatives, the NNSA Office of Acquisition and Project Management has the lead for the development of the analyses of alternatives. NNSA officials stated that they will reexamine the Concept of Operations to clarify who has responsibility for the AOA process. The officials explained that NNSA APM has begun looking at, among other things, DOE’s current directives and guidance and NNSA’s policies and procedures relating to AOA, the Office of Management and Budget’s guidance relating to AOAs, and the legislation Congress has enacted in response to other federal agencies’ AOA challenges.

NNSA Did Not Consistently Follow Certain DOE AOA Guidance for Its Projects

For the three AOAs we examined, NNSA consistently followed the DOE AOA requirement that conformed to best practices but did not consistently follow the elements of optional guidance that conform to best practices. As noted above, DOE has one requirement that conforms to best practices, the best practice of defining functional requirements based on the mission need. NNSA followed this requirement for all three AOAs we reviewed. Similarly, DOE has eight other elements of guidance that conform to best practices (see table 11). For the three projects we reviewed, NNSA followed from one to three of these eight elements of guidance.

The specific elements of guidance NNSA followed for the AOAs differed substantially by project. Specifically, NNSA followed three elements of DOE guidance that conform to best practices for the High Explosive, Science, Technology and Engineering Facility Project AOA and followed two such elements of guidance for the Radioactive Liquid Waste Treatment Facility project and only one such element of guidance for the Uranium Processing Facility project AOAs. For example, the AOA for NNSA’s High Explosive, Science, Technology and Engineering Project followed the DOE guidance that suggests identifying and considering at least three viable alternatives, including the status quo; the AOA identified nine alternatives, including one that represented the status quo. In contrast, the 2012 AOA for the Uranium Processing Facility identified six alternatives, but none represented the status quo, and the Radioactive Liquid Waste Treatment Facility project AOA identified two alternatives.

Table 11 below shows the extent to which NNSA’s AOAs for the projects we reviewed followed DOE’s guidance that conform to best practices.

Conclusions

The AOA process is a key first step in the project management process at which DOE and NNSA have the opportunity to put projects on the right path forward. DOE has some requirements and guidance for this process in its project management Order 413.3B and associated guides.

However, DOE’s AOA requirements in Order 413.3B conform to only 1 of the 24 best practices we identified from different government and private- sector entities, and the additional suggestions provided in DOE’s guides are still not sufficient to create a reliable AOA process. The AOAs for the three recent NNSA projects we reviewed—the High Explosive Science, Technology and Engineering Project, the Radioactive Liquid Waste Treatment Facility, and the Uranium Processing Facility—did not conform to best practices overall and may therefore not be reliable. DOE and NNSA officials acknowledge that unreliable AOAs are a risk factor for major cost increases and schedule delays for NNSA projects. Federal standards for internal control related to risk assessment call for management to decide on actions to mitigate identified risks. Developing requirements that incorporate best practices would help DOE mitigate the risk for major cost increases and schedule delays by creating a framework to ensure a reliable AOA process. Developing reliable AOAs is particularly important at this time because NNSA has recently reassessed or is in the process of reassessing alternatives for all its major construction projects, after spending billions of dollars on designing and partially constructing these projects. Without a process to develop reliable AOAs, NNSA may continue on this path and continue to have limited assurance that it is selecting alternatives that best meet its mission needs and will not result in major cost increases and schedule delays in the future.

Recommendation for Executive Action

To minimize the risk of developing unreliable AOAs and incurring major cost increases and schedule delays on projects, we recommend that the Secretary of Energy direct DOE’s Office of Acquisition and Project Management to update its project management order requirements to incorporate best practices for conducting an AOA.

Agency Comments and Our Evaluation

We provided DOE with a draft of this report for its review and comment. In its written comments, reproduced in appendix VIII, DOE agreed with the report’s recommendation. In its written comments, DOE stated that DOE’s order for project management (DOE O 413.3B) will be assessed for revision following the issuance of the revision to DOE-STD1189, Integration of Safety into the Design Process, currently scheduled for November 2016. DOE stated that it will consider the AOA best practices when the policy is updated. In the interim, DOE stated that it has strengthened guidance in the area of AOA, including issuing a Lifecycle Cost Handbook, and that in fiscal year 2015 it will develop and issue an AOA Handbook that will reflect the best practices for AOAs. DOE also provided technical comments that were incorporated, as appropriate.

We are pleased that DOE agreed with our recommendation and that it will take action to issue an AOA Handbook. However, while an AOA Handbook may be a useful interim measure, the unspecified, open-ended date for updating the project management order that contains requirements (i.e., sometime after November 2016) and the statement that the AOA best practices will be considered, not incorporated, may indicate DOE’s lack of urgency in implementing this recommendation.

We are sending copies of this report to the appropriate congressional committees, the Secretary of Energy, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-3841 or trimbled@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix IX.

Appendix I: Scope and Methodology

To identify and describe best practices for the analysis of alternatives (AOA) process, we first searched for a source of generally accepted best practices that we could use as criteria for comparison with the Department of Energy’s (DOE) AOA process, but we could not identify a single source that was broadly recognized by government and private- sector entities and that could serve as the definitive source of best practices for the AOA process. In the absence of such a definitive source, we identified AOA handbooks, guidebooks, requirements, and other AOA- related information from federal and private-sector entities. We also sent a request to subject-matter experts, including experts from DOE, the National Nuclear Security Administration (NNSA), and various other government or private-sector entities, to help us identify further relevant information. These subject-matter experts are part of GAO’s Projects Controls Expert Working Group, which contains members with expertise in program and project management, capital acquisition, cost estimation, risk and sensitivity analysis, earned value management and scheduling, and represent a diverse range of government and private-sector entities.

We reviewed the information related to the AOA process, including information from our Cost Estimating Guide,Advancement of Cost Engineering International, Department of Defense, Department of Homeland Security, National Aeronautics and Space Administration, Office of Management and Budget, United States Air Force, and a private-sector entity that specializes in acquisition management. We compiled a draft set of best practices commonly mentioned across these different entities’ AOA policies and guidance, and we sent this draft set of AOA best practices to the experts for review in advance of our semiannual meeting of the Projects Controls Expert Working Group that took place in March 2014. More than 90 experts participated, including officials from DOE and NNSA. We received comments from some of these experts both during this meeting and by e- mail after the meeting. We developed a final set of 24 best practices for the AOA process based on the comments received. We grouped these best practices into four categories: (1) general principles, (2) identifying alternatives, (3) analyzing alternatives, and (4) selecting a preferred alternative. the Association for the To determine the extent to which DOE’s requirements and guidance for conducting an AOA conform to AOA best practices, we first identified the process NNSA is required to follow when conducting an AOA. To do so, we reviewed DOE’s project management order and its associated guides, as well as other documentation received from NNSA’s Office of Acquisition and Project Management, including a compilation of DOE’s requirements and guidance related to conducting an AOA. We also interviewed officials from DOE’s Office of Acquisition and Project Management, NNSA’s Office of Acquisition and Project Management, NNSA’s Office of Cost Estimating and Program Evaluation, DOE’s Office of Science, and a former DOE official familiar with DOE’s and NNSA’s project management process, as well as NNSA and contractor officials from the programs and projects we selected for review. We then confirmed DOE’s requirements and guidance with DOE and NNSA officials, from whom we received technical comments.

We performed two sets of analyses: we compared the best practices to (1) DOE’s requirements and (2) DOE’s requirements combined with the guidance. We used a five-point scoring system to determine the extent to We first used this which DOE’s AOA process conforms to best practices.scoring system to determine how well DOE’s requirements and guidance conform to each best practice. We then used the average of the scores for the best practices in each of the four categories to determine an overall score for each category, and we then used the average of the scores for the four categories as the final score for the overall DOE AOA process. If the score for each best practice, the average score for each category, or the final score for the AOA process was “fully met” or “substantially met,” we concluded that the AOA process conformed to best practices and therefore could be considered reliable. In contrast, if the score was “partially met,” “minimally met,” or “not met,” we concluded that the AOA process did not conform to best practices and therefore could not be considered reliable. For us to consider the AOA process reliable, the entire AOA process had to receive an average score of “fully met” or “substantially met,” and each individual category—(1) general principles, (2) identifying alternatives, (3) analyzing alternatives, and (4) selecting a preferred alternative—had to receive an average score of “fully met” or “substantially met” to ensure that the agency adequately and consistently performed all parts of the AOA process. GAO does not weight its cost estimating, scheduling, earned value management and AOA best practices because it is not possible to quantitatively determine the relative weights of each of the criteria. Weighting has the potential to vary across programs, as well as due to where a program may be in its different stages of its life cycle. Therefore the fairest standardized methodology to evaluate programs against is to have all criteria weighted equally.

To determine the extent to which NNSA conformed to best practices in conducting the AOAs for recent NNSA projects, we used as criteria the best practices identified as part of this engagement. We chose projects for which NNSA had completed the AOA process since November 2010, when DOE’s most recent version of the project management order came into effect, or for which NNSA was scheduled to complete the AOA process by the end of fiscal year 2014. We chose projects for which NNSA had completed, or was nearing completion of, an AOA process under this version of the order because this version included significant revisions, including more specific and detailed requirements for conducting an AOA. We also selected NNSA projects that were scheduled to complete an AOA by the end of fiscal year 2014 because most of the AOA documentation would be available for these projects. We identified these projects by examining DOE’s data from its February 2014 Monthly Project Portfolio Status Report. The three projects listed in this report that met our criteria were: (1) the High Explosive Science, Technology and Engineering Project at NNSA’s Pantex site in Texas, for which NNSA was scheduled to finish the AOA process by the end of September 2014 at the time of our project selection; (2) the Radioactive Liquid Waste Treatment Facility at NNSA’s Los Alamos National Laboratory in New Mexico, for which NNSA completed its most recent reassessment of alternatives in September 2013; and (3) the Uranium Processing Facility at NNSA’s Y-12 site in Tennessee, for which NNSA completed its most recent reassessment of alternatives in June 2012.

We checked with knowledgeable DOE and NNSA officials, reviewed recent GAO reports, and compared the data in the project status report with information received from NNSA on the projects to assess the reliability of the data in this status report, and we determined that the data were sufficiently reliable to ensure these projects met our criteria. For each of these projects, we reviewed project documentation, and we interviewed NNSA and contractor officials in charge of the AOA to compare the process followed by NNSA with best practices. To score each AOA process, (1) two GAO analysts separately examined the AOA documentation received from the agency and then agreed on a score for each of the 24 best practices, then, (2) an AOA specialist independent of the engagement team reviewed the AOA documentation and the scores assigned by the analysts for accuracy and cross-checked the scores in all the analyses for consistency. After completing this process for each individual best practice in all the analyses, we calculated the scores for each category—(1) general principles, (2) identifying alternatives, (3) analyzing alternatives, and (4) selecting a preferred alternative—and the overall score for each analysis. We sent our analyses to DOE and NNSA for review twice, and we used the same scoring process when we revised the scores based on their technical comments and any additional evidence received.

In April 2014, as we were conducting our engagement, NNSA released a peer review report that recommended a new alternative for the Uranium Processing Facility. We examined whether this peer review reflected any characteristics of an AOA by interviewing NNSA officials and comparing the peer review with the AOA best practices. In addition, as part of this objective, we determined the extent to which NNSA conformed to certain DOE requirements and guidance—those DOE requirements and guidance that conformed to best practices—in conducting the AOAs for these projects. We used the results of the earlier analysis comparing DOE’s requirements and guidance to best practices to determine which DOE requirements and guidance conformed to best practices. We then reviewed project documentation and interviewed NNSA and contractor officials in charge of the AOA process to assess the extent to which NNSA projects followed DOE’s requirements and guidance that conformed to best practices.

We conducted this performance audit from February 2014 to December 2014 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: The Department of Energy’s Analysis of Alternatives Process

The Department of Energy’s (DOE) analysis of alternatives (AOA) process includes the following requirements from DOE Order 413.3B and suggested guidance from seven guides associated with this order. The source of the requirements or guidance is listed in parentheses at the end of the item.

The Integrated Project Team—the team supporting the Federal Project Director to deliver a capital asset project and conduct the AOA—consists of professionals representing a diverse range of disciplines and specific knowledge to successfully execute the project. (O 413.3B p. C-11)

As part of the preconceptual design (CD-0) approval process:

The conceptual design must develop the scope required to satisfy the program’s mission requirements and identify requirements and features. A Mission Need Statement document is developed to identify the mission need—the gap that needs to be addressed—and functional requirements—the general parameters that the selected alternative must have to address the mission need. (O 413.3B pp. A-4, C-5)

The mission need must be independent of a particular solution. (O 413.3B p. A-4)

The program office responsible for the capital asset project must identify a credible performance gap between its current capabilities and capacities and those required to achieve the goals articulated in its strategic plan. (O 413.3B p. A-4)

The program office is afforded the flexibility to explore a variety of solutions and not limit potential solutions. (O 413.3B p. A-4)

As part of the conceptual design (CD-1) approval process:

The conceptual design must develop reliable cost and schedule range estimates for the alternatives considered. The order also requires that whatever figure or range that is provided at the CD-0 and CD-1 stages must explicitly note relevant caveats concerning risks and uncertainties inherent in early estimated given the immature requirements definition at these stages. (O 413.3B pp. C-5, C-16)

The conceptual design also must develop an assessment of project risks and identification of appropriate risk handling strategies. (O 413.3B p. C-4)

Develop a conceptual design report that includes, among other things, a clear and concise description of the alternatives analyzed, the basis for the selected alternative, how the selected alternative meets the mission need, the functions and requirements that define the alternative and demonstrate the capability that the alternative can be successful, and life-cycle cost assumptions. (O 413.3B pp. A-7, C-5)

The order also requires that:

The project’s sponsor must never be the sole cost estimator at any stage, and that the second cost estimator must be from outside the chain of command, to avoid conflict of interest. (O 413.3B pp. A-1, C-16)

Conduct several independent reviews during CD-0 and CD-1 approval processes, depending on the estimated cost of the project, related to two aspects of the AOA process: the validation of the mission need statement and of the cost estimates. (O 413.3B pp. A-7, C-7, C-17, C-18)

The selected alternative must be the optimum solution that must provide the essential functions and capabilities at an optimum life-cycle cost, consistent with required cost, scope, schedule, performance, and risk considerations. (O 413.3B pp. A-2, A-5)

Additionally, DOE’s order requires that as a project moves toward approval of CD-2, if the top end of the approved CD-1 cost range for the selected alternative grows by more than 50 percent, the program must reassess alternatives. (O 413.3B p. A-6)

The order also requires as part of CD-1 approval that an analysis of alternatives must be conducted under the National Environmental Policy Act of 1969 (NEPA). (O 413.3B p. A-7)

Summarize a planned approach to conduct an analysis of alternatives. (G 413.3-17 p. 4)

The guides provide numerous examples of optional tasks for the Integrated Project Team and the Federal Project Director to perform as part of the AOA process, such as identifying functional requirements, evaluating alternatives for satisfying the requirements, conducting the appropriate analyses, and recommending a preferred alternative. (G 413.3-1 pp. 5, 7, 14, 15) (G 413.3-5A p. 8) (G 413.3-13 pp. 2, 3, 17) (G 413.3-18A p. 8) (G-413.3-21 p. 9)

Explore concepts and consider alternatives for meeting the mission need until a set of viable, affordable, and sustainable alternatives is reached; consider at least three viable alternatives for analysis, including one that represents the status quo; and develop and list the primary advantages and disadvantages for each alternative. (G-413.3-13 p. 9, G 413.3-21 p. B-2)

Develop cost estimates prior to the CD-2 approval milestone that are explicit ranges instead of point estimates, including life-cycle cost estimates of likely alternatives being considered that have a broad accuracy range from a low of -20 to -50 percent to a high of +30 to +100 percent. (G 413.3-21 p. 15)

Quantify the benefits of alternatives over their life cycle. (G 413.3- 13 p. 10)

Adjust life-cycle cost and benefit estimates for risk to ensure consideration of the alternative with the best cost-benefit ratio and generally the lowest life-cycle cost to the federal government. (G 413.3-21 p. 9)

Consider various selection criteria for the alternatives, including cost and schedule, funding and budget, and technology and engineering. (G 413.3-13 p. 9)

Weight the selection criteria used to select a preferred alternative for relative importance and compare alternatives using net present value or annuities. (G 413.3-13 p. 10; G 413.3-21 p. 69)

At the end of the analysis, the AOA team presents the recommended alternative based on the preceding analysis in an integrated form, summarizing why an alternative is preferred, and supporting the recommendation of the preferred alternative with facts from the analysis. (G 413.3-13 p. 13)

The program office performs an alternative selection and cost- range review to assess whether the AOA process (1) evaluates a range of appropriate attributes for each alternative, including cost, risks, safety, technology, and regulatory requirements and (2) is reasonable and provides best value to the federal government. (G 413.3-9 pp. 16-19)

Appendix III: Best Practices for the Analysis of Alternatives Process

We identified 24 best practices for identifying, analyzing, and selecting alternatives. These practices can be applied to a wide range of activities in which an alternative must be selected from a set of possible options, and to a broad range of capability areas, projects, and programs. These practices provide a framework to ensure that entities consistently and reliably select the alternatives that best meet mission needs.

We grouped these 24 best practices into four categories: (1) general principles, (2) identifying alternatives, (3) analyzing alternatives, and (4) selecting a preferred alternative. The four categories of best practices address the entire analysis of alternatives (AOA) process from defining the mission need and functional requirements to independently reviewing its results. On the basis of our reviews and experts’ comments, we believe that these best practices can be generally applied from the beginning of the AOA process with practices from the general principles category, through practices in the identifying and analyzing alternatives categories, and ending with practices in the selecting a preferred alternative category. We also believe that these best practices do not necessarily have to be followed in order and that some of them can be applied concurrently with other best practices. For example, the best practice of defining the selection criteria based on the mission need in the selecting a preferred alternative category could be addressed at the same time as the best practice of creating a study plan in the general principles category.

Table 12 below provides a complete list of best practices.

Appendix IV: DOE’s AOA Requirements and Guidance Compared with Best Practices

Overall, the Department of Energy’s (DOE) requirements for analysis of alternatives (AOA) in Order 413.3B minimally meet the best practices we identified. DOE’s requirements combined with its guidance in the guides associated with the order partially meet the best practices. Table 13 below describes our analysis of DOE’s AOA requirements and guidance compared with best practices.

Appendix V: NNSA’s Analysis of Alternatives Conducted at the High Explosive, Science, Technology and Engineering Project Compared with Best Practices

Overall, the National Nuclear Security Administration’s (NNSA) analysis of alternatives (AOA) conducted for the High Explosive Science, Technology and Engineering Project partially met best practices. The mission need for this project—to replace aging high explosive facilities at NNSA’s Pantex site in Texas—was approved in November 2011. NNSA expects to complete the AOA process by obtaining CD-1 in December 2014. As of September 2014, NNSA estimated that the total project cost would range from $100 to $155 million at an 85 percent confidence level. NNSA expects to finalize the AOA process by the end of 2014. Table 14 below compares the AOA conducted at the High Explosive, Science, Technology and Engineering Project with AOA best practices.

Appendix VI: NNSA’s 2013 Analysis of Alternatives Conducted at the Radioactive Liquid Waste Treatment Facility Compared with Best Practices

Overall, the National Nuclear Security Administration’s (NNSA) analysis of alternatives (AOA) conducted for the Radioactive Liquid Waste Treatment Facility project partially met best practices. The mission need for this project—to replace the current, aging facility—was approved in October 2004. NNSA approved an initial AOA for this project in 2006, and after substantial cost increases, conducted a second AOA (analyzed here) in 2013. NNSA currently estimates the project will cost between $168 million and $220 million. Table 15 below compares the AOA conducted at the Radioactive Liquid Waste Treatment Facility with AOA best practices.

Appendix VII: NNSA’s 2012 Analysis of Alternatives Conducted at the Uranium Processing Facility Compared with Best Practices

Overall, the National Nuclear Security Administration’s (NNSA) analysis of alternatives (AOA) conducted for the Uranium Processing Facility (UPF) project partially met best practices. The mission need for this project—to consolidate and replace existing enriched uranium processing capabilities at NNSA’s Y-12 site—was approved in 2004. NNSA conducted a first AOA for this project in 2007, and after substantial cost increases, conducted a second AOA (analyzed here) in 2012. NNSA estimated the project will cost between $4.2 billion and $6.5 billion. Table 16 below compares the AOA conducted at the Uranium Processing Facility with AOA best practices.

Appendix VIII: Comments from the Department of Energy

Appendix IX: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the individual named above, Daniel Feehan, Assistant Director; Bethany Benitez; Antoinette Capaccio; Jennifer Echard; Alex Fedell; Cristian Ion; Kirsten Lauber; Jennifer Leotta; Cynthia Norris; Karen Richey; Dan C. Royer; and Kiki Theodoropoulos made key contributions to this report.