text: Background This section describes (1) the purpose of LEPs and the process that NNSA and DOD use to manage them, known as the phase 6.X process; (2) the management of the ongoing LEP for the W76 warhead—an important historical reference for the B61-12 LEP—and the status of the two other ongoing LEPs; (3) future nuclear modernization plans and our past conclusions and recommendations on the affordability of these plans; and (4) the objectives of the B61-12 LEP and the roles and responsibilities of NNSA and the Air Force in conducting the program. LEPs’ Purpose and the Phase 6.X Process for Managing Them NNSA and DOD undertake LEPs to refurbish or replace nuclear weapons’ components to extend their lives, enhance their safety and security characteristics, and consolidate the stockpile into fewer weapon types to minimize maintenance and testing costs while preserving needed military capabilities. NNSA and DOD jointly manage LEPs under a multi-step process known as the phase 6.X process. The B61-12 LEP is currently in phase 6.4 (production engineering) of this process. Figure 1 illustrates the phase 6.X process. The phase 6.X process and the roles and functions of DOD, DOE, and NNSA in nuclear weapon refurbishment activities are described in a guidance document known as the Procedural Guideline for the Phase 6.X Process. The document also calls for NNSA to formally update its program cost estimate and reissue it as the baseline cost report prior to entering phase 6.4. In January 2017, NNSA issued a supplemental directive that also directs the Office of Cost Estimating and Program Evaluation to prepare an independent cost estimate for each nuclear weapon system undergoing life extension before an LEP enters phase 6.4. The Procedural Guideline for the Phase 6.X Process also describes the roles and functions of two joint bodies that provide oversight and approval functions to LEPs and other nuclear weapons-related activities: the Nuclear Weapons Council and its Standing and Safety Committee. The Nuclear Weapons Council is the joint DOD and DOE activity that serves as the focal point for interagency activities to maintain the nuclear weapons stockpile. Its membership includes the Under Secretary of Defense for Acquisition, Technology and Logistics (generally the Chair); the Under Secretary of Defense for Policy; the Vice Chairman of the Joint Chiefs of Staff; the Commander of U.S. Strategic Command; and the Department of Energy’s Under Secretary for Nuclear Security, who also serves as the Administrator of the National Nuclear Security Administration. In addition, the Nuclear Weapons Council charters a Project Officers Group for each weapon system to provide a technical forum for weapon development and management activities. Each Project Officers Group is led by a project officer from either the Navy or Air Force, the two military services that maintain and operate nuclear weapons. Management of the W76-1 LEP and Other Ongoing LEPs According to B61-12 program officials, the W76-1 LEP—which NNSA expects to complete in fiscal year 2019—has served as an important historical reference as NNSA prepared its plans and cost estimates for the B61-12 LEP. In August 2017, NNSA issued a study documenting lessons learned from difficulties it encountered in managing the W76-1 LEP. According to the study, prior to the W76-1 LEP, NNSA had not undertaken full-scale weapon system design activities since the 1982 design of the W88 warhead. Among other findings, the lessons learned study stressed the importance of using modern tools to validate and manage an LEP’s system and technical requirements to maintain cost, schedule, and performance during all phases of the program. This finding is consistent with our March 2009 findings that NNSA and DOD established an unrealistic schedule for the W76-1 LEP, did not establish a consistent cost baseline, and did not effectively manage technical risks in the program. These problems resulted in delays, additional expenditures, and difficulties tracking the cost of the program. Notably, the program had to delay first production of the W76-1 from September 2007 to September 2008 when it encountered problems with the final test batch of a key material, known as Fogbank. We recommended that NNSA develop realistic schedules for the W76-1 and future LEPs that build in additional time for unexpected technical challenges that may delay the programs. NNSA agreed with our recommendation and has taken steps toward improvement in this area, which we continue to monitor. In addition to the B61-12 and W76-1 LEPs, NNSA and DOD are managing two other LEPs: the W88 Alteration 370 program and the W80- 4 LEP. Table 1 provides basic information on all four ongoing LEPs. Future Nuclear Modernization Plans and Their Affordability In addition to the four ongoing LEPs, NNSA and DOD have outlined plans for several future nuclear weapon modernization programs: Under NNSA’s current program of record, which does not yet reflect new requirements that will be generated based on the 2018 Nuclear Posture Review, NNSA has plans for life extension efforts to transition the nuclear stockpile to three interoperable ballistic missile warheads and two air-delivered weapons. NNSA has described this plan as the 3+2 strategy. To undertake this strategy, NNSA has proposed initiating a series of interoperable warhead programs between about 2020 and 2060. NNSA’s plans for the first ballistic missile warhead in the 3+2 strategy—the Interoperable Warhead 1—indicate that, if authorized by Congress, the warhead would cost an estimated $12.4 billion from 2020 to 2041. As we reported in August 2015, NNSA paused the Interoperable Warhead 1 program in fiscal year 2014 to provide more time to study the concept of interoperability and to reduce uncertainty about the agency’s ability to achieve necessary plutonium and uranium capabilities to support the LEP. Under its current program of record, NNSA plans to resume the Interoperable Warhead 1 program in fiscal year 2019. Under its current program of record, NNSA has also begun preliminary planning for Interoperable Warhead 2, Interoperable Warhead 3, and B61-12 follow-on programs that, if authorized, would start in the 2020s and 2030s. In the 2018 Nuclear Posture Review, DOD stated a near-term intention to modify a small number of existing submarine-launched ballistic missile warheads to provide a low-yield option, and a long- term intention to pursue a modern nuclear-armed sea-launched cruise missile. The NNSA Administrator stated in March 2018 that NNSA would continue to work with DOD to determine the resources, time, and funding required to address these and other policies specified in the Nuclear Posture Review. As we concluded in an April 2017 report, these plans come during a particularly challenging decade for NNSA’s nuclear modernization efforts, as the agency plans to simultaneously execute at least four LEPs along with major construction projects, such as efforts to modernize NNSA’s uranium and plutonium capabilities. We further concluded that NNSA’s modernization budget estimates for fiscal years 2022 through 2026 may exceed the funding levels programmed for modernization in future budgets, raising affordability concerns. Moreover, we concluded that NNSA had not addressed a projected “bow wave” of future funding needs—that is, an impending and significant increase in requirements for additional funds—or the mismatch between potential funding needs and potential funding available. We recommended that NNSA include an assessment of the affordability of NNSA’s portfolio of modernization programs in future versions of the Stockpile Stewardship and Management Plan—for example, by presenting options NNSA could consider to bring its estimates of modernization funding needs into alignment with potential future budgets. NNSA did not explicitly agree or disagree with our recommendation, but we will continue to monitor any actions NNSA takes in response to the recommendation. Objectives of the B61-12 LEP and the Roles and Responsibilities of NNSA and the Air Force The B61-12 LEP has several objectives: consolidating the nuclear bomb stockpile, improving the accuracy of the resulting weapon through a new guidance assembly, and addressing other age-related issues. Consolidating the stockpile. Under the B61-12 LEP, NNSA and the Air Force plan to consolidate and replace four of the five variants of the B61 that were in the active stockpile at the time the B61 LEP began. Improving accuracy. The B61-12 is to be equipped with a new tail kit guidance assembly that enables it to be delivered with greater accuracy than the B61 bombs it replaces, which are equipped with parachutes. More specifically, according to Air Force officials and documents, the assembly will provide the B61-12 with a guided freefall capability while retaining a ballistic (unguided) delivery capability. The greater accuracy of the B61-12 is to enable the B61-12 to meet all the military requirements for which past versions of the B61 were designed. Addressing other age-related issues. The B61-12 LEP is to extend the service life of the B61 by at least 20 years, make field maintenance of the weapon easier for Air Force technicians, and provide modern security features. NNSA manages its B61-12 LEP activities through a federal program office on Kirtland Air Force Base in Albuquerque, New Mexico, under the direction of the federal program manager. It manages the work of six government-owned, contractor-operated NNSA laboratories and sites that serve as design and production agencies for the LEP. Sandia National Laboratories, also located on Kirtland Air Force Base, serves as the systems-level integrator for the overall weapon design. Figure 2 shows the six sites participating in the B61-12 LEP and their respective roles. The Air Force’s responsibilities, in addition to managing the acquisition of the tail kit guidance assembly, include integrating the B61-12 with its delivery aircraft and the operational flight program software of these aircraft. The Air Force Nuclear Weapons Center, also at Kirtland Air Force Base and under the direction of the Air Force lead project officer, manages technical integration and other LEP-related tasks required to qualify, certify, and field the weapon. The delivery aircraft that carry the B61-12 are being designed to deliver the weapon in two different modes with two different systems, the second of which provides the enhanced capabilities offered by the new tail kit guidance assembly. System 1 aircraft will have an analog interface with the B61-12 that is designed to deliver the weapon in a ballistic mode, with the tail kit in a fixed position. System 2 aircraft will have a digital interface with the B61-12, enabling the guided delivery capability afforded by the tail kit assembly. Figure 3 illustrates the delivery aircraft for the B61-12. NNSA Substantially Incorporated Best Practices in Developing the B61- 12 Program Cost Estimate NNSA substantially incorporated most of the cost estimating best practices identified by our past work when it developed the $7.6 billion program cost estimate for the B61-12 LEP. Our cost estimating guide identifies best practices for developing a high-quality, reliable cost estimate and identifies four characteristics of such an estimate: it is comprehensive, well-documented, accurate, and credible. These four characteristics and some of the best practices that underlie them are illustrated in figure 4. We assessed the B61-12 program cost estimate by comparing it with the best practices identified in our cost estimating guide and found that it substantially met the criteria for all four characteristics of a high-quality, reliable cost estimate (see fig. 5). A summary of our assessment is presented below, including reasons that the program cost estimate substantially met the criteria under each of the four characteristics as well as some examples of the best practices that the cost estimate could have more fully incorporated. Appendix I provides additional information on our assessment. Comprehensive: Substantially Met. The program established a consistent and clearly defined work breakdown structure—a hierarchical structure that subdivides the work necessary to accomplish the program’s objectives into smaller elements—to ensure that costs were not double-counted or omitted. The clearly defined work breakdown structure also helped the B61-12 program office manage the process of integrating each site contractor’s estimate for the cost of its activities into the overall program estimate. To more fully incorporate the criteria for a comprehensive cost estimate, the program would have had to take additional steps, such as including the full life-cycle costs associated with the B61-12 weapon in the estimate. Specifically, the estimate would have had to include costs such as program costs incurred prior to phase 6.3, the cost of NNSA federal program office personnel, components that are being shared by different nuclear weapon programs (such as the weapon’s radar), and costs associated with maintenance of the B61-12 after the LEP ends and the weapon enters the stockpile. In addition, the estimate would have had to include an assessment of how the program would be affected if key assumptions, such as the timing of the delivery of the tail kit guidance assembly, did not hold true. Nevertheless, the program incorporated practices that substantially met the criteria for a comprehensive cost estimate, which we believe contributed to the program’s estimate being reliable. Well-documented: Substantially met. In our visits to NNSA sites and our associated review of site contractors’ documents, we found that site contractors provided detailed documentation of their contributions to the cost estimate to the B61-12 program office. At all of the sites we visited, experienced cost estimating teams captured specific information on the data and data sources used to inform their estimates. To more fully incorporate the criteria for a well-documented cost estimate, the documentation that the site contractors provided to the NNSA program office would have had to capture the reliability of the underlying data and discuss how the data were normalized. Nevertheless, the program incorporated practices that substantially met the criteria for a well-documented cost estimate, which we believe contributed to the program’s estimate being reliable. Accurate: Substantially met. Technical personnel at both the NNSA sites and the Albuquerque federal program office discussed program risks to ensure that the program estimate represented a most likely, unbiased cost. Furthermore, all of the site-level cost estimates we examined—which the federal program office integrates into the overall program cost estimate—drew on historical data from primary sources, including internal financial systems from either past B61 costs or previous LEPs. Use of such sources is consistent with the best practice of grounding the estimates in a historical record of cost estimating and actual experiences on other comparable programs. In addition, the federal program office routinely reviews contract performance reports from each of the B61-12 sites to track variances between estimated and actual costs on a monthly basis. To more fully incorporate the criteria for an accurate cost estimate, the program would have had to use site estimates that were calculated in base- year dollars and then uniformly adjusted for inflation at the program level, and clearly defined the method it used to determine inflation indexes. Instead, all of the site contractors developed their cost estimates in then-year dollars and applied varied inflation indexes. Nevertheless, the program incorporated practices that substantially met the criteria for an accurate cost estimate, which we believe contributed to the program’s estimate being reliable. Credible: Substantially met. The B61-12 LEP became the first LEP to undergo a statutorily required independent cost estimate, conducted by the Office of Cost Estimating and Program Evaluation. Additionally, a different NNSA office developed a third cost estimate for the program to aid in the preparation of NNSA’s budget materials. Each of these three estimates used a different methodology. NNSA used this third estimate to cross-check overall program costs. Moreover, to assess risk and uncertainty in the program, most of the site estimates we reviewed included a detailed, quantifiable risk assessment for their portion of the overall program estimate. To more fully incorporate the criteria for a credible cost estimate, the program’s sensitivity analysis would have had to more fully examine and document cost impacts for the overall estimate and the individual site estimates. Instead, according to NNSA officials, it focused primarily on schedule and critical path analysis. Moreover, to more fully incorporate the criteria for a credible cost estimate, the program would have had to address risk correlation and the calculation of confidence levels differently. In the program’s analysis of risks and uncertainties in the program, we found the program inconsistently examined correlation among program risks. Specifically, according to NNSA officials, to arrive at the 70 percent confidence level for the overall program cost estimate, the program office added site-level cost estimates together at the 50 percent and 70 percent confidence levels. As noted in our cost guide, adding risk results for the underlying estimates in this way results in an incorrect confidence level for the overall estimate. Nevertheless, the program incorporated practices that substantially met the criteria for a credible cost estimate, which we believe contributed to the program’s estimate being reliable. We consider a cost estimate to be reliable if the overall assessment ratings for each of the four characteristics are substantially or fully met—as was the case with the B61-12 program cost estimate, which substantially met these criteria. For that reason, we are not making recommendations related to the program’s use of cost estimating best practices. However, by fully incorporating all of the best practices for the four characteristics, NNSA can better ensure that its future cost estimates are of high quality and reliable. Varying Methods and Assumptions about Future Program Performance Led to Differences between the Estimates, but NNSA Did Not Document the Rationale for Adopting the Program’s Estimate Unchanged The $7.6 billion program cost estimate for the B61-12 LEP differs from the $10 billion independent cost estimate primarily because the program office used different methods and assumptions than the Office of Cost Estimating and Program Evaluation, which prepared the independent cost estimate. The B61-12 program developed its estimate by compiling site- specific cost and schedule estimates for activities at each of the NNSA sites participating in the LEP; in contrast, the independent cost estimate projected a cost and completion date by evaluating program activities completed to date and applying a historical model to estimate costs and durations for remaining activities. As noted in our cost guide, both of these methods are commonly applied. To reconcile the differences between the two estimates, high-ranking NNSA officials met with officials from the B61-12 program office and the Office of Cost Estimating and Program Evaluation to discuss the estimates in 2016. However, NNSA did not document the rationale for its decision to use the program office’s lower estimate unchanged or a plan for how it would take the independent cost estimate into consideration. We previously recommended that NNSA should establish a requirement for its management to document and justify key decisions based on a reconciliation of LEP cost estimates with the Office of Cost Estimating and Program Evaluation’s independent cost estimates. NNSA agreed with this recommendation. The B61-12 Program Office and the Office That Developed the Independent Cost Estimate Used Different Estimating Methods and Assumptions Cost estimating best practices specify that programs should develop a point estimate—the best guess at the program’s cost estimate, given the underlying data—by collecting, analyzing, and validating program data and then using one of several commonly used methods for estimating the program’s cost. Once a program has developed a point estimate, the program should compare it to an independent cost estimate, which gives an objective measure of whether the program’s point estimate is reasonable. In January 2017, NNSA issued two directives implementing statutory requirements for the Office of Cost Estimating and Program Evaluation to develop independent cost estimates for NNSA programs, including LEPs. The differences between the respective cost estimating methods, both of which are valid, used by the B61-12 program office and the Office of Cost Estimating and Program Evaluation are the primary reason for the differences between the program estimate and the independent cost estimate. According to B61-12 program officials, the program generally developed its point estimate by using a “bottom-up” method formally known as the “engineering build-up” cost estimating method. In using this method, a program subdivides the work necessary to accomplish its objectives into a work breakdown structure. The program then develops estimates of costs at the lowest level of the work breakdown structure, one piece at a time, and uses the sum of the pieces to form the overall estimate. To develop its cost estimate, the B61-12 program office required all participating NNSA site contractors to prepare and submit their own cost estimates for the work to be performed on the LEP and provided instructions on what data to provide to the program office. For example, these instructions specified that all sites must apply a bottom-up estimating approach that includes detailed quantities and integrated resource-loaded schedules for all work breakdown structure elements under their management. The program office then compiled the site- provided information in a database to arrive at a total program cost. The program office also aggregated schedule information from the sites, which maintain detailed resource-loaded integrated site schedules, to develop an NNSA Integrated Master Schedule. As we previously noted, the program office estimated in October 2016, based on this process, that NNSA’s portion of work on the B61-12 LEP would cost $7.6 billion and that the LEP would be completed in fiscal year 2025, with a first production unit date of March 2020. In contrast, to develop the independent cost estimate, the Office of Cost Estimating and Program Evaluation used an estimating method that employed data on the B61-12 LEP’s actual performance, coupled with historical information from the W76-1 LEP for stages of the phase 6.X process that the B61-12 LEP had not yet reached. Specifically, the office gathered data on 1,600 activities in the NNSA Integrated Master Schedule for the LEP. The office tracked these 1,600 activities from August 2014 through March 2016 by evaluating data from successive versions of the NNSA Integrated Master Schedule, which the B61-12 program office updates monthly based on actual program performance to date. In a memo summarizing the office’s independent cost estimate, the office stated that the program’s task completion rate lagged the baseline plan. The office concluded that the LEP’s first production unit date would occur 2 years after the March 2020 target date unless the program took measures to reduce the LEP’s scope by removing tasks, delaying activities until after the first production unit date, or relaxing requirements to accommodate less mature components than originally planned. The office also concluded, based on the program’s spending rate of approximately $45 million per month, that pre-first production unit costs would increase by about $1 billion over the program’s estimate. To estimate the cost and schedule of the program after the first production unit date, the office used both B61-12 LEP actuals and historical information from the W76-1 LEP, comparing W76-1 funding levels to B61- 12 spending levels. On the basis of its analysis, the office concluded that full-scale production of the B61-12 would cost approximately $1 billion more than the program office estimated. All told, the independent cost estimate projected that the B61-12 LEP would cost approximately $10.0 billion and take about 2 years longer—with a projected completion date in fiscal year 2027—barring changes to the program’s scope. The B61-12 program office and the Office of Cost Estimating and Program Evaluation also have differences of opinion regarding the continued validity of the August 2014 schedule performance data and its relevance to the independent cost estimate. According to B61-12 program officials, the information in the NNSA Integrated Master Schedule improved and changed after the Office of Cost Estimating and Program Evaluation gathered initial schedule performance data in August 2014 and used this information as a starting point to evaluate the program’s performance. These issues include the following: The officials described the data available to the Office of Cost Estimating and Program Evaluation in August 2014 as tentative, saying that the program can now use the NNSA Integrated Master Schedule to track performance at a more detailed level. According to B61-12 program officials, the program made important decisions that affected components on the program schedule’s critical path at the time the August 2014 schedule performance data were gathered. Subsequent to establishing the baseline, for example, the program office restructured the path to first production unit for high- explosives components, correcting errors that had been captured in the August 2014 data and changing to a more streamlined approach to qualify high-explosives components from legacy material. This decision affected the program’s critical path to first production unit, moving the completion date earlier. The program undertook schedule recovery efforts that eased schedule constraints affecting other program elements that were on the critical path at the time of the August 2014 data. As a result of these factors, B61-12 program officials said that the entire baseline schedule that the Office of Cost Estimating and Program Evaluation analyzed appeared more problematic than the updated schedule and that the entirety of the independent cost estimate was thrown off by the obsolete August 2014 data. Officials from the Office of Cost Estimating and Program Evaluation told us they disagree with the B61-12 program office’s assessment of the independent cost estimate schedule analysis. These officials said that they understand that the schedule baseline is continuously changing but that the independent cost estimate schedule analysis is not dependent on a particular baseline. Rather, they said that the analysis is based on actual schedule performance for the 1,600 activities that represent the scope required to achieve the design maturity and that the program office specified in August 2014 as needed to reach the first production unit (phase 6.5) milestone. Officials from the Office of Cost Estimating and Program Evaluation said that although they will not formally assess the B61-12 LEP’s schedule again until the end of phase 6.4 of the program, their informal analysis of NNSA Integrated Master Schedule data as of February 2017 still showed the same rate of activity completion that underpinned the office’s independent cost estimate. At some point, according to these officials, the program will have to double or even triple its rate of activity completion to finish the LEP on schedule, which will increase cost. In contrast, B61-12 program officials stated that, given the improved quality of the program’s integrated master schedule data, they expect that the independent cost estimate that the Office of Cost Estimating and Program Evaluation prepares at the end of phase 6.4 of the LEP will be closer to the program’s estimate than to the October 2016 independent cost estimate. Program officials also said that the program’s performance to date supports their position that the program cost estimate is accurate. The positions of the two offices also differ regarding the B61-12 program’s ability to accelerate work in the production stages of the LEP to ensure that the LEP meets its completion date. B61-12 program officials stated that they have options other than to complete tasks sequentially and at a steady rate, so they do not expect the “straight-line” level of productivity assumed in the independent cost estimate analysis to occur. For example, some parts do not have to be built in a particular sequence. Instead, program officials said, the production agencies can build different lots of components when they are ready, so technologies that are ready earlier than others can be moved to production in the war reserve lot while other components remain in earlier stages. They also said that the program would not maintain an even spending rate of $45 million per month, as suggested in the independent cost estimate. Rather, they noted, the program’s spending rate is currently $55 million a month, and the program plans for it to rise to $65 million per month as the current production engineering phase of the LEP draws to a close and the production agencies accelerate their activities. These factors notwithstanding, one Office of Cost Estimating and Program Evaluation official observed that activities in the later stages of an LEP remain complex and carry risks. The official cited the history of the challenges that led to the delay of first production of the W76-1, cited earlier in this report, and said that the B61-12 program faces the added challenges of having to integrate with several delivery aircraft and of having more electronic components than the W76-1. As noted in our cost estimating guide, studies have shown limited opportunity for getting a delayed program back on track after it is more than 15 percent to 20 percent complete. NNSA Officials Met to Reconcile the Program’s Cost Estimate and the Independent Cost Estimate but Did Not Document the Rationale for Adopting the Program’s Estimate Unchanged Cost estimating best practices specify that a program cost estimate and an independent cost estimate should be reconciled and that differences between them should be examined and discussed to achieve understanding of overall program risk. Officials from NNSA, including from the Office of Cost Estimating and Program Evaluation and the B61- 12 program office, told us that they held several discussions in 2016 regarding the differences between the program estimate and the independent cost estimate. These included a meeting with the second- highest ranking official in NNSA—the principal deputy administrator— during which the respective offices presented their estimates and explained the methods used to produce them. After these meetings, the principal deputy administrator and the NNSA Administrator agreed to approve the program estimate unchanged. According to B61-12 program officials, the program adapted some of its practices as a result of their interactions with the Office of Cost Estimating and Program Evaluation. For example, officials said that they changed the program’s procedure for baseline changes to ensure consistency across the participating sites. The program also began to conduct baseline execution index analyses, as the Office of Cost Estimating and Program Evaluation recommended. Baseline execution index analyses track a program’s execution of tasks to date by monitoring the percentage of activities that a program has completed early or on time and that have a baseline for completion within the month the analysis is conducted. According to program officials, similar to a schedule performance index in an earned value management system, the baseline execution index gives an alternate cumulative measure that gives a program an opportunity to improve as it proceeds. However, B61-12 program officials said that they did not document the rationale for adopting the program cost estimate without making changes informed by the independent cost estimate. They told us that any attempt to combine the results of the two estimates would have been difficult, considering the significant differences between the program’s cost estimating model and the Office of Cost Estimating and Program Evaluation’s model. We recommended in a January 2018 report that NNSA should establish a requirement for its management to document and justify key decisions based on a reconciliation of LEP cost estimates with the Office of Cost Estimating and Program Evaluation’s independent cost estimates. We concluded in the report that without a requirement for its management to document and justify key decisions based on a reconciliation of program cost estimates with the Office of Cost Estimating and Program Evaluation’s independent cost estimates, NNSA may not have assurance that the independent cost estimates are being appropriately incorporated into the LEP decision-making process, potentially decreasing the reliability of program cost estimates. Our prior work has shown that, in general, because the independent cost estimate team is outside the acquisition chain, is not associated with the program, and has nothing at stake with regard to program outcome or funding decisions, its estimate is usually considered more accurate than the program’s internal estimate. In addition, our prior work has shown that independent cost estimates are historically higher than program office cost estimates because the team conducting the independent cost estimate is more objective and less prone to accept optimistic assumptions. However, we have also found that because independent cost estimates are typically higher than program office cost estimates, in some cases management may choose to ignore them because the estimates are too high. NNSA agreed with our January 2018 recommendation, stating that by March 2018, it would establish a protocol to document management decisions regarding significant variances between LEP cost estimates and the independent cost estimates produced by the Office of Cost Estimating and Program Evaluation. However, NNSA has not provided evidence that it has done so. We continue to believe that documenting key decisions regarding cost estimates is particularly important in the context of LEPs, where decisions could increase a program’s costs by billions of dollars. NNSA and DOD Have Identified Risks and Are Taking Steps to Manage Them NNSA and DOD have identified and are managing various risks that could complicate efforts to meet the fiscal year 2025 completion date for the B61-12 LEP. Some of these risks that the agencies are managing are within the program’s areas of responsibility, such as an aggressive flight test schedule, and additional risks could be identified within these areas. To manage risks, the program uses a formal risk management process and has taken steps such as consolidating flight tests and holding more regular meetings between NNSA’s design and production agencies. The program also faces risks that program officials told us lie outside the program’s direct control—such as risks related to the F-35 delivery aircraft, nuclear certification, and NATO coordination issues—and officials said they have provided information to the responsible DOD organizations to help address these risks. NNSA and DOD Have Taken Steps to Identify and Help Manage Risks within the Program’s Responsibility, and They May Identify Additional Risks NNSA and DOD have taken steps to identify and help manage risks within the B61-12 LEP’s responsibility, and program officials said that they may identify additional risks as the program progresses. More specifically, the program has a formal risk management process through which it has identified risks and could identify additional risks as the program proceeds, according to agency officials. Risks already identified and being managed include risks related to the program’s aggressive schedule of flight tests and to finalizing design and coordinating procurement and delivery of components. Risk Identification Process The B61-12 LEP has a formal risk management process that has identified joint NNSA and Air Force risks within the program’s areas of responsibility that could significantly impact the overall program’s schedule, its cost, or the technical performance of the weapon. According to program officials and the Program Joint Risk Management Plan, this process calls for each program element in NNSA or the Air Force to be responsible for identifying and managing risks at the lowest level possible. After the program element reviews and documents a risk, it then reviews the risk to determine its applicability to be considered a joint risk—that is, a risk that has the potential to affect any of the top-level program milestones or the program’s ability to successfully meet system performance requirements. Program officials told us that the Air Force lead project officer decides whether to accept the risk into the joint risk list. Senior management oversees those risks through a formal management plan. The process includes continual reviews to identify new risks that may emerge. The Joint Risk Review Board meets as new potential risks are identified to review their likelihood and consequence. Officials from both the Office of Cost Estimating and Program Evaluation and DOD’s Office of Nuclear Matters told us that during phase 6.4 and thereafter, the program may still discover new risks—”unknown unknowns”—during technical tests to qualify components and the development of production processes. The process also has steps to manage risks and remove them from the joint high-risk list, if the Joint Risk Management Board judges them to have been resolved to closure or a low-risk status, according to program officials. NNSA and DOD program officials said that the program’s risk management process has resulted in the resolution of about three- quarters of the identified high risks on the joint risk list. They also observed that the program’s Selected Acquisition Reports, through which NNSA and the Air Force report to the congressional defense committees on the program’s cost and schedule, have been unchanged since 2013 regarding major program milestones. Program officials said that to provide a 90-day schedule buffer and add flexibility to the program’s schedule in the event of unexpected difficulties, the program has planned to reach phase 6.5 in December 2019, ahead of the phase 6.5 date of March 2020 that is reported in the Selected Acquisition Reports. However, other officials told us that it is too soon to say whether the program can manage the identified risks, or other unidentified risks, to prevent delays in a program that has relatively little schedule margin. Problems can emerge even during the first production stage of an LEP, as happened in the W76-1 LEP due to the Fogbank production challenges we discuss earlier in this report and in our March 2009 review of W76 and B61 modernization efforts. Managing Risks Related to the Program’s Aggressive Schedule of Flight Tests DOD and NNSA officials we interviewed generally agreed that the program faces risks in completing an aggressive flight testing schedule to support the first production unit deadline. According to the officials, the B61-12 program needs to complete more than 60 flight tests over a 3- year period to meet this deadline. Completing the tests entails actively coordinating with the Air Force organizations that manage the various aircraft that will carry the B61-12 weapon: the B-2 bomber and the F-15, F-16, F-35, and PA-200 fighters. According to B61-12 program officials, aircraft may not be available when needed for the planned flight testing. This risk is of particular concern for B-2 bombers, they said, because only one B-2 test unit is available and it is in heavy demand for other Air Force purposes. Program officials characterized the flight test schedule as aggressive and ambitious, but feasible, and told us the program has managed the risks caused by the tight testing timeframes by coordinating with the responsible organizations and consolidating tests to minimize the amount of time required on each type of aircraft. Further, when aircraft are not available as planned, program officials said they can revise the sequence of tests. For instance, they accommodated the unavailability of a B-2 test asset on a planned test date by moving up a test date on the F- 16. This schedule adjustment avoided a ripple effect of delays on the overall testing schedule, according to Air Force officials. The video in figure 6 shows an F-16 dropping an inert B61-12 bomb during a flight test on March 14, 2017. NNSA and DOD have identified and taken steps to manage risks related to finalizing the weapon’s design and coordinating the procurement and delivery of components. These risks include: Technical risks associated with the design and production of various components. Officials told us some components of the bomb and tail kit assembly are on the program’s list of joint risks. They said that their use of the joint risk management process calls management attention to potentially serious risks and helps the program to manage these risks as early and as continually as possible. For example, NNSA officials said that when technical risks arose in designing one classified component on the program’s critical path—potentially affecting the design schedule—they augmented the design team with additional scientists in an effort to ensure that the component would be completed in time to support the production schedule. Similarly, to manage design risks related to the exacting specifications for certain components, Kansas City National Security Campus is working to develop sufficiently precise gages to measure the required specifications during production. Late design changes from design agencies provided to the production agencies. NNSA’s Fiscal Year 2018 Stockpile Stewardship and Management Plan identified late changes to component design as a risk facing the B61-12 program and other LEPs. Contractor officials we interviewed from the Kansas City National Security Campus and the Pantex Plant said that late changes to weapon design requirements from the Sandia and Los Alamos design agencies could create schedule problems for establishing production processes at the production sites. Kansas City National Security Campus officials expressed concerns that some component requirements continue to change—some arising from testing results—which creates a tension between improving the design and stabilizing production requirements and processes. Pantex officials also told us about a potentially significant production delay if late design changes require Pantex to get new production tools or testers. Late design changes could occur as scientists at the design agencies analyze test results. Flight tests, for example, produce a volume of information. Officials at the federal program office in Albuquerque said that 4 test flights on the B-2, conducted in July 2017, produced 4 to 6 hours’ worth of data per flight. Officials at both the Pantex and Kansas City sites said they have developed management strategies to provide some flexibility in their production schedules, such as speeding production by having staff work longer shifts. Moreover, because of lessons learned from prior LEPs, officials at both sites told us that coordination between production sites and design agencies has significantly improved over past practices— specifically, by having ongoing engagement that started earlier in the weapon development process. For instance, Pantex officials told us that they hold monthly meetings with design agencies to discuss design changes. A Pantex official told us that, as a result of addressing production concerns early, design requirements developed at the design agencies are less likely to result in unanticipated production problems. Vendor risks associated with procuring various bomb components. According to NNSA officials we interviewed, some bomb components are procured through single commercial vendors, in small lots, or are unusual. Kansas City National Security Campus officials told us that they had to replace one vendor that could no longer provide a certain material and that they generally risk losing potential or existing vendors because vendors prefer contracts for larger volumes of components than NNSA needs for the B61-12 bomb. In addition, unique materials for the bomb include certain components with specific compositions of rubber and plastics. Officials at the Kansas City National Security Campus said that they have encountered difficulties with getting rubber and plastic components from vendors that consistently meet composition specifications or with sustaining vendors’ interest in producing small batches of precision-manufactured components. In one such instance, they said they improved incentives and communication with a vendor to avoid losing a source for a key component. The officials said they also contract with smaller vendors when larger vendors may not be interested in the size of the contract NNSA offers. Delays in delivery of components from other production sites to Pantex for full bomb assembly. NNSA and Pantex officials told us that Pantex, which will assemble the full B61-12 bomb, depends on the other production sites delivering the components in a timely manner. NNSA production sites are scheduled to provide components to Pantex 120 days before the first production unit date. Pantex and NNSA officials have identified some schedule flexibility for assembling the first production unit at Pantex, depending on which components have delayed deliveries. Specifically, if the delayed components are those needed later in the assembly process, such as the bomb’s nose assembly, Pantex could stay on schedule by assembling other delivered components until the delayed components are needed. Delivery of other components, such as detonators, is more time- sensitive, and it is essential that these be delivered on time for assembly to proceed as planned, according to Pantex officials. NNSA and DOD Have Taken Steps to Identify and Help Manage Risks outside the Program’s Direct Control According to program officials, certain risks that may have a bearing on the B61-12 LEP or that may affect the fielding of the weapon lie in areas outside the program’s direct control. Nevertheless, program managers have taken steps to coordinate with other responsible parties to help address these risks. For example, two of the three delivery aircraft designated as system 2 aircraft—the F-35 and the B-2—have not yet completed development and procurement of operational flight program software that will enable the aircraft to deliver the B61-12 with the enhanced accuracy offered by the tail kit assembly, a key feature of the LEP. B61-12 program officials told us that the program offices responsible for each of these aircraft must manage the development and procurement of the operational flight program software. To help inform the software development process, the B61-12 program provided the F-35 and B-2 program offices with information about the weapon’s interface with the airplane, including information from flight tests performed on an earlier version of the F-35, according to program officials. NNSA and DOD officials characterized B-2 development related to the B61-12 as significantly more advanced than F-35 development. Specifically, Air Force officials said that a developmental version of the B-2 operational flight program software was fielded and certified in 2017 and would undergo final weapon system demonstration flight tests in October 2019 and nuclear design certification in June 2020. By contrast, they said that the F-35 software will not be ready for nuclear design certification until January 2023, after the B61-12 program’s first production unit date. The F-35 program office will be responsible for funding tests and aircraft- weapon integration activities, according to the Air Force officials. Because of the need to defer some flight tests until the software for the B-2 and F- 35 aircraft is ready, only one of the three system 2 delivery aircraft has undergone testing of the B61-12 bomb’s capabilities in its system 2 setting: the F-15E, on which NNSA and DOD conducted the first system 2 tests of the B61-12 in August 2017. Other risks outside the program’s direct control concern nuclear certification and the NATO mission. Nuclear certification—ensuring that people and objects that come into contact with the weapon will not adversely affect its performance characteristics—is a prerequisite to fielding the B61-12 and other nuclear weapons, but it is the responsibility of the Air Force organizations that manage the delivery aircraft. In a classified report issued in January 2018, we discuss risks related to nuclear certification of dual capable aircraft, which are able to deliver conventional munitions or nuclear bombs. B61-12 program officials told us that they are working to address these risks by providing information on the weapon to all of the organizations that manage the delivery aircraft. Similarly, in another classified report issued in February 2018, we discuss a risk related to the NATO mission that may affect the B61-12 LEP; program officials told us that they are working to address this risk, as well. We made recommendations in the two classified reports related to these risks; the responsible agencies agreed with our recommendations and stated their intention to take action in response to them. Agency Comments We provided a draft of this product to NNSA and DOD for comment. NNSA provided technical comments, which we incorporated as appropriate. DOD indicated that it did not have any comments. We are sending copies of this report to the appropriate congressional committees, the Secretaries of Defense and Energy, the Administrator of NNSA, and other interested parties. In addition, this report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff members have any questions about this report, please contact me at (202) 512-3841 or bawdena@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made significant contributions to the report are listed in appendix II. Appendix I: Results of GAO’s Assessment of the B61-12 Life Extension Program Cost Estimate Compared with Best Practices Characteristic Comprehensive Best practice The cost estimate includes all life cycle costs. The cost estimate completely defines the program, reflects the current schedule, and is technically reasonable. The cost estimate work breakdown structure—a hierarchical structure that subdivides the work necessary to accomplish the program’s objectives into smaller elements—is product-oriented, traceable to the statement of work/objective, and at an appropriate level of detail to ensure that cost elements are neither omitted nor double-counted. The estimate documents all cost-influencing ground rules and assumptions. The documentation should capture the source data used, the reliability of the data, and how the data were normalized. The documentation describes in sufficient detail the calculations performed and the estimating methodology used to derive each element’s cost. The documentation describes step by step how the estimate was developed so that a cost analyst unfamiliar with the program could understand what was done and replicate it. The documentation discusses the technical baseline description, and the data in the baseline is consistent with the estimate. The documentation provides evidence that the cost estimate was reviewed and accepted by management. The cost estimate results are unbiased, not overly conservative or optimistic, and based on an assessment of most likely costs. The estimate has been adjusted properly for inflation. The estimate contains few, if any, minor mistakes. The cost estimate is regularly updated to reflect significant changes in the program so that it always reflects current status. Variances between planned and actual costs are documented, explained, and reviewed. The estimate is based on a historical record of cost estimating and actual experiences from other comparable programs. Characteristic Credible Best practice The cost estimate includes a sensitivity analysis that identifies a range of possible costs based on varying major assumptions, parameters, and data inputs. A risk and uncertainty analysis was conducted that quantified the imperfectly understood risks and identified the effects of changing key cost driver assumptions and factors. Major cost elements were cross checked to see whether results were similar. An independent cost estimate was conducted by a group outside the acquiring organization to determine whether other estimating methods produce similar results. Appendix II: GAO Contact and Staff Acknowledgments GAO Contact: Staff Acknowledgments: In addition to the contact named above, Jonathan Gill (Assistant Director), Rob Grace (Analyst in Charge), Terry Hanford (Senior Analyst), and Jennifer Leotta (Senior Operations Research Analyst) made key contributions to this report. Also contributing to this report were Antoinette C. Capaccio, Scott Fletcher, Penney Harwell Caramia, Cynthia Norris, Karen Richey, and Sara Sullivan.
target: Weapons in the U.S. nuclear stockpile are aging. To refurbish or replace nuclear weapons' aging components, NNSA and DOD undertake LEPs. The B61-12 LEP is the most complex and expensive LEP to date. In October 2016, NNSA formalized a program cost estimate of about $7.6 billion, which is lower than an independent cost estimate of about $10 billion. Senate Report 113-44 included a provision for GAO to periodically assess the status of the B61-12 LEP. This report assesses (1) the extent to which NNSA followed best practices for cost estimation in producing the program cost estimate for the B61-12 LEP; (2) the reasons for differences between the program cost estimate and the independent cost estimate and how the differences were reconciled; and (3) the extent to which NNSA and DOD have identified and managed program risks. GAO assessed the program cost estimate against best practices, reviewed NNSA and DOD documents, conducted site visits to four NNSA and Air Force sites responsible for design, production, and management activities, and interviewed NNSA and DOD officials. The National Nuclear Security Administration (NNSA) incorporated most cost estimating best practices to develop the program cost estimate for the B61-12 Life Extension Program (LEP), which seeks to consolidate four versions of a nuclear weapon—the B61 bomb—into a bomb called the B61-12. As shown in the figure below, the program substantially met best practices for ensuring the estimate was comprehensive, well-documented, accurate, and credible. The B61-12 LEP's program cost estimate differs from an estimate prepared by another NNSA office independent of the program primarily because the program used different methods and assumptions than the independent office. The program developed its estimate by compiling cost and schedule estimates for activities at each of the NNSA contractor sites participating in the LEP. In contrast, the independent office evaluated program activities completed to date and applied a historical model to estimate costs and durations for remaining activities. NNSA management met with officials from both offices to reconcile the estimates but did not document the rationale for adopting the program estimate unchanged. GAO recommended in a January 2018 report that NNSA document and justify such decisions, in part because GAO's prior work has shown that independent cost estimates historically are higher than programs' cost estimates because the team conducting the independent estimate is more objective and less prone to accept optimistic assumptions. In response to the January 2018 report, NNSA agreed to establish a protocol to document management decisions on significant variances between program and independent cost estimates, but it has not yet provided evidence that it has done so. NNSA and the Department of Defense (DOD) have identified and are managing risks that could complicate efforts to meet the LEP's fiscal year 2025 completion date. Risks within the program's areas of responsibility include an aggressive flight test schedule for bomb delivery aircraft. The program is managing these and other risks with a formal risk management process. The program has also taken steps to address risks outside its direct control, such as risks related to the readiness and certification of the weapon's F-35 delivery aircraft, by providing information to the responsible DOD organizations.