Nuclear Nonproliferation: NNSA Needs to Improve Its Program Management Policy and Practices
Released: 2017-09-28T00:00:00

Letter

Background

DNN Selected Subprograms

Within DNN, the work of the four selected subprograms—Nuclear Material Removal, HEU Reactor Conversion, Radiological Security, and International Nuclear Security—focuses on efforts to remove and dispose of excess nuclear material from civilian sites worldwide, convert civilian research reactors to the use of non-weapons-useable nuclear fuel, secure radiological materials at their source in the United States and abroad, and improve the security of weapons-useable nuclear material in key countries. The selected subprograms organize their work in programmatic areas which we refer to as components, and under each component the subprograms manage projects. Table 1 below describes the work of each subprogram and the components in which the subprogram organizes its work scope.

Program Management Leading Practices Related to Schedule and Cost Management

PMI’s The Standard for Program Management and GAO’s schedule and cost guides identify program management leading practices related to schedule and cost estimating and measuring performance against baselines, as follows:

PMI guidelines. According to PMI’s guidelines, programs practice life-cycle management, which involves schedule and financial management throughout the course of the program’s life-cycle phases—program definition, benefits delivery, and closure. In particular, PMI states that in conducting program schedule management, programs use a master schedule that integrates the schedules of program components necessary to achieve the program’s goal. In program financial management, program cost estimates should be clearly defined and should consider the full life- cycle costs of the program. According to PMI, programs should also establish and measure performance against baselines for both schedule and cost.

GAO schedule and cost guides. GAO’s schedule and cost guides, which draw from federal organizations and industry, define best practices about the processes needed for the development and management of high-quality and reliable schedule and cost estimates. Similar to PMI’s guidelines, according to the GAO guides, programs should establish and use an integrated master schedule, establish cost estimates that cover the full life cycle of the program, document and define assumptions tailored to the program, incorporate analysis of program risk and uncertainty in schedule and cost estimates, and manage a program’s schedule and cost by measuring against a baseline.

Selected DNN Subprograms Generally Do Not Use Selected Leading Practices to Manage Schedule and Cost

The four DNN subprograms we chose for review generally do not use selected leading program management practices to manage schedule and cost. Specifically, at the time of our review, none of the subprograms had schedule and cost estimates that encompassed its entire life cycle, although one subprogram planned to develop such estimates for its recently-extended life cycle. In addition, none of the selected subprograms measure their overall schedule and cost performance against baseline estimates. NNSA officials said that the subprograms had not developed schedule and cost estimates that cover their life cycles and did not measure the subprograms against baselines due, in part, to uncertainty in planning scope and schedules that rely on the cooperation of other countries. DNN also does not require subprograms to have such estimates or to measure performance against schedule and cost baselines. Following these practices, however, would provide NNSA managers and other stakeholders more complete information to evaluate how much the subprograms may cost to achieve their goals, the amount of time they may need to achieve these goals, and their actual versus planned performance.

According to leading practices, programs should (1) establish a master schedule that integrates the schedules of program components necessary to achieve the program’s goal, such as specified performance to be achieved over a defined life cycle, (2) determine costs that consider the full life-cycle costs of the program, and (3) measure performance against baselines for both schedule and cost. Figure 2 illustrates the extent to which the selected subprograms have established schedule and cost estimates compared to their planned life-cycle completion dates, if any.

The Nuclear Material Removal subprogram had schedule and cost estimates that encompassed all three of its subprogram components through the subprogram’s previously planned completion date of fiscal year 2022. However, the subprogram had yet to update its schedule and cost estimate through its new planned completion date of fiscal year 2027, which was established in May 2017. The subprogram did not have readily available information on performance against its former schedule and cost estimates. Specifically:

Schedule. As of April 2017, the subprogram’s schedule, which encompassed all three subprogram components, included 52 ongoing and planned projects with estimated completion dates by the end of fiscal year 2022 for most of these projects to reach a goal to remove or disposition a total of 8,466 kilograms of nuclear material. In May 2017, the subprogram extended its life cycle from fiscal year 2022 to fiscal year 2027 but at the time of our review had yet to update its schedule of planned projects to be completed during fiscal years 2023 through 2027. According to NNSA officials, they extended the subprogram’s life cycle in part because certain projects planned to be completed by fiscal year 2022 were delayed and the subprogram’s work was expanded.

Cost. The subprogram had a cost estimate for its planned work through fiscal year 2022 but at the time of our review had yet to update its cost estimate for the overall subprogram through its new planned completion date of fiscal year 2027. Specifically, as of June 2017, the subprogram had a cost estimate of about $595 million, according to our analysis of information provided by the subprogram. This estimate covered the planned work scope of all three subprogram components to be completed during fiscal year 2017 through 2022. The subprogram, however, did not have estimated costs for completing work scope planned during fiscal years 2023 through 2027. According to NNSA officials, as of June 2017, they were developing a cost estimate for the remaining years, although the officials did not specify when the cost estimate would be completed.

Measuring performance against baselines. The subprogram did not measure its overall performance against schedule and cost baselines. NNSA reported to Congress in July 2014 that the subprogram planned to remove or disposition approximately 3,000 kilograms of nuclear material by fiscal year 2022 at an estimated cost of about $600 million. However, the subprogram did not track information on its performance against the cost estimate. According to NNSA officials, removal projects have too many uncertain costs. Instead, NNSA officials said that they update the subprogram’s life- cycle cost each year as part of the annual planning for the next fiscal year’s budget request. Until the subprogram develops schedule and cost estimates to support the recently revised life-cycle completion date of fiscal year 2027, it does not have the baselines it needs to measure its overall schedule and cost performance. Although the subprogram did not measure its overall performance against established schedule and cost baselines, according to monthly performance reports, the subprogram baselined and measured the schedule performance of individual removal projects by tracking the difference in number of days between forecasted project completion dates and baseline completion dates. However, the subprogram did not have information that integrated project performance information to provide an overall picture of schedule performance for the entire subprogram.

HEU Reactor Conversion

The HEU Reactor Conversion subprogram had schedule and cost estimates that covered the remaining work scope to complete two of three subprogram components by fiscal year 2033 but not for a third component estimated to be completed in fiscal year 2035. The subprogram also did not measure its overall performance against schedule and cost baselines. Specifically:

Schedule. The HEU Reactor Conversion subprogram did not have a schedule for the overall subprogram through completion of its life cycle. Instead, the subprogram had a schedule for all work scope planned for the 5-year FYNSP, which included the schedule for the remaining work to complete one of the three subprogram components–Molybdenum 99 (Mo99) efforts. Beyond the FYNSP planning period, the subprogram has an estimated completion date of fiscal year 2033 for a second component—U.S reactor conversions— and has developed a schedule for completion of the component. For the third subprogram component—international reactor conversions— the subprogram estimates a fiscal year 2035 completion date for its remaining work scope to convert or verify the shutdown of 44 international reactors, but it had not developed a complete schedule to meet that date. Specifically, the subprogram’s schedule was not up- to-date for 22 of the 44 international reactors in the subprogram’s planned work scope to support the estimated fiscal year 2035 completion date for these reactors. Instead, in the subprogram’s schedule, these reactors had estimated completion dates by fiscal year 2030. NNSA officials explained that the schedule was not up-to- date for these reactors because the reactors are in countries where the subprogram cannot currently plan or implement the conversions due to limitations in cooperation with these countries. For example, DNN cannot plan the schedule for conversion of reactors in Russia that are in the subprogram’s scope until the United States and Russia resume joint nuclear security activities that the United States discontinued following Russia’s invasion of Ukraine in 2014. NNSA officials said that the 2035 date is their best judgment of the earliest date when the subprogram could complete the conversions or verify certain reactors’ shutdowns based on the assumption that the United States and Russia may resume nuclear security cooperation in the 2020s. Because of the high degree of uncertainty with this date, the subprogram did not update the schedule to reflect the 2035 date, according to the officials. Appendix II provides tables that list the planned reactor and facility projects in the HEU Reactor Conversion subprogram, their locations, and estimated conversion or shutdown completion dates.

Cost. The HEU Reactor Conversion subprogram did not have a life- cycle cost estimate for the overall subprogram, but had overall life- cycle cost estimates for two of the three subprogram components. The subprogram had cost estimates that totaled approximately $1.1 billion through fiscal year 2033 and that included the remaining estimated life-cycle costs for the subprogram’s U.S. reactor conversions component and its Mo99 efforts. For the third component—international reactor conversions—the subprogram only estimated costs for the 5-year FYNSP, not through the estimated completion date for the component of fiscal year 2035. According to NNSA officials, developing a cost estimate that includes all remaining international reactor conversions through 2035 would be challenging because the costs for these projects are highly uncertain and vary depending on the willingness of each country to cooperate as well as the unique technical, regulatory, and other factors that vary for each reactor in each country. The subprogram, however, had established estimated life-cycle budgets for completing the conversion or verifying the shutdown of each reactor in its work scope, which could be used, along with other information, to develop a cost estimate for the subprogram component.

Measuring performance against baselines. The subprogram did not measure overall subprogram performance against schedule and cost baselines. Specifically, as mentioned above, the subprogram did not have schedule and cost estimates for the overall subprogram that it could use to establish baselines to measure the performance of the overall subprogram. Although the subprogram had life-cycle estimates for its U.S. reactors and Mo99 components, the subprogram did not use these estimates as baselines to measure the overall subprogram components’ performance. The subprogram measured schedule performance of individual projects under its three components against baselines by tracking the difference in number of days and months between forecasted project completion dates and baseline completion dates. However, it did not integrate and roll up the project information to provide an assessment of its overall schedule performance. In addition, the subprogram baselined and measured cost performance of the U.S. High Performance Research Reactor project—which constitutes six of the seven reactors under its U.S. reactor conversions component—by tracking changes in the project’s estimated life-cycle cost. However, the subprogram did not have similar information that tracked changes in cost estimates of other projects under its three components.

Radiological Security

The Radiological Security subprogram did not have schedule and cost estimates for three components through the subprogram’s planned completion date in fiscal year 2033. The subprogram also did not measure overall subprogram performance against schedule and cost baselines. Specifically:

Schedule. The subprogram has an estimated completion date of fiscal year 2033 but did not have an overall schedule that covered its three components for meeting the 2033 date. Instead, the subprogram had a schedule that covered work to be completed under its three components during the 5-year FYNSP (fiscal years 2017 through 2021). Specifically, for two of the three subprogram components— radiological source removal and nonradioisotopic technologies—the subprogram has not established specific work scope and schedules beyond fiscal year 2021 because of uncertainty about the future. For example, according to the subprogram’s director, planning the adoption of nonradioisotopic technologies is uncertain because the timing of when such technologies can be adopted depends, in part, on regulations and international laws, making it challenging for the subprogram to define the scope of work. For the third subprogram component—radiological source protection—the subprogram has an estimated completion date of fiscal year 2033 to reach a total target to secure 4,394 buildings in its inventory of sites worldwide with high- priority radiological sources. However, the subprogram had not developed a schedule of specific projects to be completed beyond the 5-year FYNSP to meet that date and target. NNSA officials said that they are often uncertain when a project will be able to start because it depends greatly on circumstances in each country. Appendix III provides the Radiological Security subprogram’s planned work scope for the radiological source protection component from fiscal years 2017 through 2033.

Cost. The Radiological Security subprogram did not have a life-cycle cost estimate for the overall subprogram through its estimated completion date of fiscal year 2033. Specifically, the subprogram had a cost estimate of about $849 million for all three components covering the 5-year FYNSP. However, for two of the three subprogram components—radiological source removal and nonradioisotopic technologies—the subprogram had not developed cost estimates beyond the 5-year FYNSP because, as mentioned above, it had not developed work scope for these components in the out-years. For example, according to the subprogram’s director, the subprogram’s radiological source removal component depends on the voluntary participation of users of radiological sources that register their sources with the subprogram. Therefore, the subprogram cannot estimate the number of sources to be removed in out-years. For the third subprogram component—radiological protection—the subprogram had assumed a stable budget to complete its target to secure 4,394 buildings by fiscal year 2033. However, according to the director of the subprogram, this budget assumption was not intended to be a reliable life-cycle cost estimate.

Measuring performance against baselines. As mentioned above, the subprogram did not have schedule and cost estimates for the overall subprogram needed to establish baselines to measure their overall performance. The subprogram, however, baselined and measured the schedule performance of individual projects under its three components by tracking the difference in number of days between forecasted project completion dates and baseline completion dates. The subprogram, however, did not integrate and roll up the project schedule performance information to provide performance information for the overall subprogram.

International Nuclear Security

The International Nuclear Security subprogram maintained schedule and cost estimates for the 5-year FYNSP (fiscal years 2017 through 2021) but did not have schedule and cost estimates for work scope in the years beyond the FYNSP. In addition, the subprogram did not measure overall performance against baselines. Specifically:

Schedule. The International Nuclear Security subprogram had not established a life-cycle schedule for the overall subprogram or its two component efforts, as it had not identified specific work scope or end- point targets beyond fiscal year 2021 and considers its mission to be enduring (i.e. without an end-date). Instead, the subprogram had only estimated a schedule for work scope in individual countries during the 5-year FYNSP. According to the subprogram director, the subprogram is expected to operate indefinitely and continue as long as nuclear materials exist to improve security in countries possessing such materials. However, the subprogram had not planned project-specific work scope in years beyond the FYNSP because, according to the subprogram director, it is difficult to estimate the subprogram’s likely level of foreign counterpart engagement in individual countries beyond 5 years.

Cost. Because it has not identified out-year work scope, the International Nuclear Security subprogram did not have an overall life- cycle cost estimate and only had an estimate of about $530 million for the work to be completed during the 5-year FYNSP period. According to NNSA officials, they have not developed a cost estimate for work scope in the years beyond the FYNSP because assumptions about future work will likely change due to the uncertainty in relationships with partner countries.

Measuring performance against baselines. The International Nuclear Security subprogram did not measure performance of the subprogram against schedule and cost baselines. Specifically, as mentioned above, the subprogram did not have the schedule and cost estimates for the subprogram’s life cycle beyond fiscal year 2021 needed to establish baselines to measure its overall performance. In addition, the subprogram did not use its 5-year FYNSP estimates as baselines to measure performance. Instead, the subprogram updates the FYNSP estimates each year in planning the next fiscal year’s budget request. Moreover, unlike the other three subprograms, the International Nuclear Security subprogram did not have project schedule baseline information that could be integrated and rolled up to provide information on the performance of the overall subprogram.

In general, NNSA officials explained that uncertainty in planning the selected subprograms’ work scope or schedules, particularly for components with projects that rely on the cooperation of foreign countries, was among the reasons they did not have schedule and cost estimates that covered the subprograms’ life cycles or that went beyond the 5-year required planning period. In addition, according to these officials, DNN senior management does not require subprograms to establish schedule and cost estimates that cover the entire subprogram life cycle and to use these estimates as baselines to measure subprogram performance.

However, uncertainty should not prevent these subprograms from establishing more complete or longer-term estimates to account for the time and resources they need to achieve their goals. As mentioned above, without such estimates, the subprograms do not have the baseline information they need to track their performance. According to leading practices, developing reliable schedule and cost estimates can be achieved by following steps that address data limitations and risks and uncertainties for a program. For example, according to the GAO schedule guide, a reliable schedule should reflect all of a program’s activities and recognize that uncertainties and unknown factors in schedule estimates can stem from, among other things, data limitations. In addition, according to the GAO cost guide, the cost-estimating process involves defining and documenting assumptions that are tailored to the specific program, such as about the program’s life-cycle phases, political issues, or technology development. Assumptions should be based on historical data to minimize uncertainty and risk. These same assumptions should also be used to develop the program schedule. For management to make good decisions, the program estimate must reflect the degree of uncertainty so that a level of confidence can be given about the estimate. Accordingly, because assumptions defined for a particular program’s schedule and cost estimate can vary, they should always be inputs to the program’s risk analyses of cost and schedule.

Programs use different methods to quantify uncertainty and risk in developing a schedule or cost estimate. DOE’s cost estimating guide describes approaches for programs to incorporate risk and uncertainty in cost estimates such as the use of lower- and upper-bound cost ranges that are developed based on risk analysis. Other NNSA programs use these approaches in developing schedule and cost estimates for highly uncertain, long-term program plans. In particular, NNSA’s Office of Defense Programs develops and reports high- and low-range cost estimates for elements of NNSA’s nuclear weapons modernization programs in part to account for the uncertainty in these long-term program estimates. As mentioned above, such estimates would provide NNSA managers and other stakeholders information to help evaluate resources and compare the costs and benefits of different programs and priorities. Because the selected subprograms do not measure their overall schedule and cost performance against baselines, NNSA managers, stakeholders, and Congress have incomplete information about these subprograms’ actual-versus-planned schedule and cost performance over their duration and are, therefore, at risk of being unable to assess when a subprogram is likely to be completed or whether it will cost more or less than planned.

DNN’s Program Management Policy Includes Some Leading Practices, but Does Not Address Life-Cycle Schedule and Cost Management

DNN’s Revised Policy Includes Leading Practices on Risk and Quality Management

DNN’s 2017 revised policy includes new sections that address leading practices on risk and quality management that all DNN programs and subprograms should follow. NNSA officials said they added these sections based on their review of leading practices in PMI’s The Standard for Program Management and GAO’s Standards for Internal Control in the Federal Government to ensure these leading practices were incorporated and required for DNN programs.

Risk management. According to leading practices on risk management, programs should have processes to manage risks, including processes to identify, assess, and respond to risks. In the revised DNN policy, under a new section on risk management, all DNN programs and subprograms are required to prepare risk management plans to help identify, analyze, handle, and monitor risk. For example, a DNN subprogram may identify the risk of schedule slippage due to political constraints in working with foreign countries and could incorporate and monitor that risk in planning.

Quality management. According to program management leading practices on quality management, program quality should be continuously monitored. A new DNN policy section on continual improvement requires DNN programs and subprograms to plan and implement methods, such as program evaluations and management assessments, in order to monitor and improve processes. For example, a DNN subprogram may use an independent review by the NNSA Office of Management and Budget to help improve its program management processes, such as how it tracks cost, scope, and schedule. The revised policy also outlines steps for corrective actions to be taken when noncompliance is detected. These steps range from determining the cause of noncompliance to reviewing the effectiveness of corrective actions taken.

These new sections added requirements for DNN program management that were not previously documented. For example, in the prior policy, risk management was not a requirement for DNN programs and subprograms. In addition, NNSA officials said that they added the continual improvement section to the revised policy after reviewing PMI’s practices on quality assurance, which they believed would clarify responsibilities regarding management assessments and independent reviews.

The Revised DNN Policy Does Not Include Leading Practices on Life-Cycle Schedule and Cost Management

The revised DNN policy does not address or require leading practices on life-cycle schedule and cost management for DNN programs or subprograms. Specifically, the revised policy does not outline requirements for programs or subprograms to establish life-cycle cost estimates or measure performance against schedule or cost baselines. Instead, the revised policy provides requirements on schedule and cost management limited to the NNSA budgeting process covering the 5-year FYNSP. For example, according to the revised DNN policy, programs and subprograms must conduct program management activities, such as budget formulation, in alignment with anticipated resources in the FYNSP. Additionally, the policy requires programs and subprograms to establish performance measurement data and track cost or schedule performance, but only within the FYNSP.

According to leading practices, life-cycle management is important to program management and includes schedule and cost management activities that span the duration of the program. According to PMI, all programs, regardless of length, have life cycles; furthermore, leading practices indicate that activities related to managing the schedule, cost, and scope of a program should be conducted for the life of the program. For example, leading practices call for calculating cost estimates as close to the beginning of a work effort as possible that consider the full program life cycle, and then documenting this baseline to measure performance.

According to NNSA officials, the revised DNN policy does not include requirements to practice life-cycle management, including life-cycle schedule and cost management, because officials determined that life- cycle management did not apply to some DNN programs that NNSA officials believe are enduring or continuous. For example, as mentioned above, the director of the International Nuclear Security subprogram said that the subprogram will phase out of certain areas or reduce engagement with certain countries in the future but that it is expected to continue as long as nuclear materials exist and will work to improve security in countries possessing such materials.

We disagree that life-cycle program management does not apply to programs or subprograms that may have an enduring mission. Managers need to make informed decisions about whether a program is affordable within the agency’s portfolio. NNSA and DNN should be able to compare DNN’s various programs’ requirements several years beyond its 5-year planning period. According to the GAO cost guide, in developing estimates, programs should define assumptions tailored to the program, such as assumptions about the program’s life-cycle phases. For example, the International Nuclear Security subprogram could take steps to define end-point targets for when it may phase out work in certain areas or countries in the future. In addition, according to the GAO schedule guide, a comprehensive schedule should reflect all of a program’s activities and recognize that uncertainties and unknown factors in schedule estimates can stem from, among other things, data limitations. Moreover, because assumptions themselves can vary, they should always be inputs to program risk analyses of cost and schedule.

According to NNSA officials, although the revised policy does not include requirements for life-cycle cost estimating, DNN programs could address this in their individual program management plans. NNSA officials stated that these program management plans for programs and subprograms should be detailed enough to also provide information on how the program will track progress, including by identifying changes to the planned schedule.

However, the revised DNN policy does not clearly require DNN programs or subprograms to have program management plans, nor does it specify elements of such plans. Specifically, the revised DNN policy requires each program to develop “program management documentation” that identifies program scope, schedule, and cost during the fiscal year and operating procedures for the fiscal year, but it does not outline similar requirements for the program’s life cycle. In addition, the revised policy does not specify requirements or guidance, such as on cost estimation, for what programs or subprograms are to include in the program management documentation. In contrast, PMI indicates that programs should develop a program management plan that includes plans for program financial management, schedule management, and scope management for all phases of the program’s life cycle. According to NNSA officials, the revised DNN policy is the only directive or documentation that spells out what is needed or required to be included in a program management plan.

Although the revised DNN policy does not clearly require DNN programs or subprograms to have program management plans, some DNN programs have developed or are developing such plans. For example, the Global Material Security program, which oversees the Radiological Security and International Nuclear Security subprograms, issued a new program management plan in April 2017. The Global Material Security program management plan requires that each subprogram maintain a 5- year budget for the FYNSP with cost estimates, but it does not require or provide guidance on developing life-cycle schedule or cost estimates. NNSA officials said that DNN underwent a major reorganization of its programs in January 2015, and some of the new program offices are still preparing their program management plans. For example, the Material Management and Minimization program that oversees the Nuclear Material Removal and HEU Reactor Conversion subprograms is still developing its program management plan, according to NNSA officials.

In addition, the four selected subprograms had various documented plans, but none fully addressed life-cycle schedule and cost management.

Nuclear Material Removal. The subprogram did not have a current program management plan that had been updated since the 2015 reorganization of DNN but instead relied on an older plan that covered a different scope than the scope of the current subprogram.

HEU Reactor Conversion. The subprogram did not have a program management plan for the overall subprogram. Instead, the subprogram had project execution plans for its U.S. reactor conversion projects and its Mo99 projects and relied on an outdated document for its international reactor conversion projects.

Radiological Security. The subprogram had a program management plan that included requirements for the use of project life-cycle baselines and for conducting cost estimation for the 5-year FYNSP. However, the plan had no requirement for developing a cost estimate for the life cycle of the subprogram and for using such an estimate to measure performance of the overall subprogram.

International Nuclear Security. The subprogram had a program management plan that required cost estimating for 1 fiscal year. However, the plan did not include requirements for life-cycle estimates and for using initial or updated baselines to measure performance.

NNSA subprogram officials said that they do not have readily available life-cycle cost estimates and baseline measurement data in part because they are not asked to provide it. For example, NNSA officials from the HEU Reactor Conversion subprogram said that they did not have sufficient staff to track performance against initial baselines because it was not a priority for management, although it would be possible to do so if required. One of the stated goals of the revised DNN policy is to facilitate DNN-wide implementation of methods for programs and subprograms to monitor, measure, and improve management processes. However, because the policy does not require more complete information from DNN programs and subprograms on their cost, schedule, and performance against baselines—consistent with leading practices—it is not clear that this policy goal can be achieved.

Conclusions

When organizations apply leading program management practices—such as establishing schedules and cost estimates covering their planned life cycles and measuring performance against such baselines—they may be able to enhance their chances of achieving success across a range of programs. However, the four selected DNN subprograms are generally not applying these selected leading practices for life-cycle program schedule and cost management, due in part to the uncertainty and risks in working with international partners. However, methods and approaches exist that allow programs to account for uncertainty and risk in developing schedule and cost estimates for their planned scope of work. Furthermore, while the revised DNN program management policy has incorporated some leading practices, it does not include requirements and guidance for DNN programs and subprograms to practice life-cycle schedule and cost estimating and does not require program management plans that could be the vehicle for DNN programs and subprograms to specify the use of such estimates. Updating the DNN program management policy to include requirements for DNN programs and subprograms to follow leading practices for life-cycle program management would help NNSA ensure that managers, stakeholders, and Congress have better information on how much DNN programs and subprograms may cost to achieve their goals, the amount of time they may need to achieve these goals, and how efficiently and effectively they are actually being executed compared to plans.

Recommendation for Executive Action

The NNSA Deputy Administrator for DNN should revise the DNN program management policy to require DNN programs and subprograms to follow life-cycle program management. These requirements should include development of schedule and cost estimates that cover the life cycle of DNN programs and subprograms, use of methods to account for uncertainty and risk in such estimates, use of cost and schedule baselines to measure performance over program and subprogram life cycles, and development of program management plans. (Recommendation 1)

Agency Comments and Our Evaluation

We provided NNSA with a draft of this report for its review and comment. In written comments, which are summarized below and reproduced in appendix IV, NNSA neither agreed nor disagreed with our recommendation to revise the DNN program management policy to require DNN programs and subprograms to follow life-cycle program management. However, NNSA stated that it plans to take action in response to the recommendation.

In general, NNSA stated that DNN will update its program management policy to formally document current practice and clarify expectations for addressing uncertainty. Specifically, NNSA said it will update the policy to: (1) reflect that life-cycle cost and schedule management should be applied at the project or subprogram level where appropriate, considering the extent of uncertainty impacting scope, potential timelines, and executability; (2) define the methodologies to (a) account for uncertainties where applying these techniques would result in a reasonable range of estimates that would be useful for planning and scheduling purposes or (b) document risk and track actions to reduce uncertainty where applicable; (3) address expectations for assessing cost and schedule performance, commensurate with the level of certainty present at baselining; and (4) address requirements for documenting program management plans.

Although we acknowledge NNSA’s plan to update its policy, we have concerns regarding whether its proposed actions will ensure that DNN programs and subprograms effectively follow leading practices for life- cycle schedule and cost management in the future. First, we do not believe that updating the DNN program management policy to formally document current program management practice addresses our recommendation. NNSA’s response suggests that its update to the policy is intended to reflect current DNN program management practices rather than signal a need for corrective action to address the DNN program management limitations we identified. Specifically, as we stated in our report, none of the four subprograms we reviewed had schedule and cost estimates that encompassed the entire life cycle, although one subprogram planned to develop such estimates for its recently-extended life cycle. In addition, NNSA’s proposed update to the DNN program management policy to reflect life-cycle schedule and cost management “where appropriate” is vague, and may give programs and subprograms too much discretion to avoid the requirement. To have an effective requirement on life-cycle program management and to be responsive to our recommendation, NNSA will need to clearly define the criteria for when a program should be exempt from a requirement to follow life-cycle program management.

Finally, the meaning of NNSA’s proposed update to the policy to address expectations for assessing cost and schedule performance, commensurate with the level of certainty present at baselining is unclear. Specifically, it is unclear whether NNSA plans to require that DNN subprograms use cost and schedule baselines to measure performance, or whether it plans to exempt programs or subprograms from such practices based on unstated expectations. As we stated in our report, none of the subprograms we reviewed measured their overall schedule and cost performance against baseline estimates. To ensure that DNN subprograms take steps to measure schedule and cost performance against baselines and to be responsive to our recommendation, NNSA will need to define clear expectations for DNN programs and subprograms to follow.

NNSA also provided general comments in its written comments regarding DNN program management.

First, NNSA commented that DNN currently implements elements of life- cycle program management where appropriate and reasonable. However, according to NNSA, the majority of its international activities operate with an unusually high level of uncertainty regarding potential international cooperation and with limited information on international operations to understand the scope of work required to support useful planning and estimating. In NNSA’s view, the high uncertainty would result in range estimates so broad as to serve no useful purpose, and there is no appreciable cost-benefit to expending resources on such calculations.

We recognize that organizations need flexibility to determine when it is appropriate and useful to apply leading practices on life-cycle program management. However, as noted in our report, managers need to make informed decisions about whether a program is affordable within the agency’s portfolio. Without more complete schedule and cost information on DNN subprograms, NNSA managers and other stakeholders have degraded information on the elements of DNN’s portfolio, which may limit their ability to assess and justify the affordability of long-term plans. If NNSA believes that some of DNN’s planned international work scope is too uncertain for subprograms to develop estimates of schedule and cost that cover their life cycles, then NNSA should evaluate whether it is appropriate to identify such work scope in DNN’s long-term plans at all.

Second, NNSA commented that no specific requirement exists for DNN programs and subprograms to implement life-cycle cost estimates, and that DNN complies with current requirements. NNSA also commented that the proper application of leading practices recognizes that cost- benefits, as well as the potential usefulness and reliability of estimates, are important considerations. In instances in which uncertainty is extremely high, NNSA stated that focus shifts to disclosure of risks, and the establishment and tracking of actions to reduce the level of uncertainty. According to NNSA’s comments, DNN discloses risks and tracks actions to reduce the level of uncertainty extensively, and this was reflected in the most recent update to the DNN program management policy with the addition of a new section on risk management. NNSA also stated that as uncertainty is reduced, then other principles can be applied where appropriate.

We stated in our report that no specific requirement exists for DNN programs and subprograms to implement life-cycle cost estimates. Specifically, we noted that the DNN policy required that program management functions be conducted over the 5-year FYNSP. Therefore, we agree that the DNN subprograms we chose to review complied with current requirements. However, our review was not focused on compliance with requirements but rather on the use of leading or good program management practices. We also noted that NNSA’s stated objectives for the DNN policy include establishing a DNN-wide policy that incorporates leading practices for program management and that facilitates the implementation of methods for programs and subprograms to monitor, measure, analyze, and improve management processes. Leading practices on life-cycle program management are important for an organization to successfully plan the resources it needs to achieve its goals and assess its performance in doing so. DNN’s revised policy did not acknowledge management of the program life-cycle as an essential program management function and did not include any requirements on leading practices on life-cycle schedule and cost management.

We agree that risk management processes should be used to monitor risks and track actions to reduce uncertainty. As we stated in our report, the revised DNN policy included a new section on risk management under which all DNN programs and subprograms will be required to prepare risk management plans to help identify, analyze, handle, and monitor risk. However, the new section did not include criteria for DNN subprograms to follow when uncertainty related to risks being monitored is low enough to allow a subprogram to develop life-cycle schedule and cost estimates.

We are sending copies of this report to the appropriate congressional committees, the NNSA Administrator, the NNSA Deputy Administrator for Defense Nuclear Nonproliferation, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-3841 or oakleys@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix V.

Appendix I: Objectives, Scope, and Methodology

This report examines the extent to which (1) selected subprograms within the National Nuclear Security Administration’s (NNSA) Office of Defense Nuclear Nonproliferation (DNN) use program management leading practices to manage schedule and cost, and (2) DNN has incorporated program management leading practices in its revised program management policy.

To conduct this work, we reviewed 4 selected DNN subprograms. DNN has 4 major programs that manage a total of 13 subprograms (a subprogram is a program managed as part of another program). Specifically, we selected the Nuclear Material Removal and Highly Enriched Uranium (HEU) Reactor Conversion subprograms, which DNN manages under its Material Management and Minimization program. In addition, we selected the Radiological Security and International Nuclear Security subprograms, which DNN manages under its Global Material Security program. We selected these subprograms for review because they had defined start dates, end dates, and/or work scope indicating that they had project-like aspects. These subprograms organize their work in programmatic areas which we refer to as components and under each component the subprograms manage various types of projects, such as projects to remove nuclear material from civilian sites worldwide. We also selected the 4 subprograms because they were not the subject of other ongoing or recently completed GAO reviews. The information we obtained from these subprograms is not generalizable, but we believe that we obtained important insights into DNN’s cost and schedule management of these subprograms.

To examine the extent to which the selected DNN subprograms use program management leading practices to manage cost and schedule, we identified selected leading practices by the Project Management Institute (PMI) in The Standard for Program Management and by GAO in its schedule and cost guides. The selected leading practices we identified were the use of a master schedule necessary to achieve a program’s goals, cost estimates that cover the full life-cycle of a program, and schedule and cost baselines to measure performance. We collected and reviewed subprogram planning documents, monthly performance reports, and spreadsheet data on work scope, historical costs, schedules and cost estimates established by the subprograms, and their use of project baselines to measure performance. We also reviewed information the subprograms reported in NNSA’s fiscal year 2017 and 2018 congressional budget justifications. We also interviewed NNSA officials and their contractors who manage the program management information system used by 3 of the 4 subprograms to manage schedule and cost information to understand its capabilities. We interviewed NNSA officials who manage the selected DNN subprograms about the use of these practices and their views on challenges or limitations in using them. We also interviewed representatives at Argonne National Laboratory and Pacific Northwest National Laboratory, which operate projects for the subprograms, to identify how projects develop schedule and cost estimates and pass information on to the subprograms.

To assess the reliability of the schedule and cost estimates on the selected subprograms, we interviewed NNSA officials and national laboratory contractors who were knowledgeable about the process followed to develop and update the estimates and the program management information systems used to manage the schedule and cost information and generate reports. We determined that the data were sufficiently reliable for our purposes, which were to report the subprograms’ estimated schedule completion dates and cost estimates, as well as report the fiscal years and subprogram components and projects covered by the subprogram schedule and cost estimates.

To examine the extent to which DNN has incorporated leading practices into its revised program management policy, we reviewed DNN’s revised program management policy approved in February 2017. We compared the revised policy to the 2005 version to identify the changes included in the revised policy. We reviewed program management leading practices by PMI in The Standard for Program Management and by GAO in its schedule and cost guides and federal internal control standards. For example, we considered the applicable leading practices on schedule and cost management identified above as well as other practices such as those on risk management, quality management, and development of program management plans. We compared these practices to DNN’s requirements and guidance contained in the revised DNN policy. We interviewed NNSA officials about the development of the new policy and their views on the reasons specific leading practices were included in the revised policy and others were not, as well as challenges DNN’s programs and subprograms face in managing program schedule and cost. We also reviewed program management plans for the 4 selected subprograms and the major programs under which these subprograms operate. We then interviewed NNSA officials from the selected subprograms to determine their involvement in developing the revised DNN program management policy and the status of individual program management plans that were under development at the time of our review.

We conducted this performance audit from June 2016 to September 2017 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Scope and Completion Dates for the Highly Enriched Uranium Reactor Conversion Subprogram

The Office of Defense Nuclear Nonproliferation’s Highly Enriched Uranium (HEU) Reactor Conversion subprogram consists of three components: (1) U.S. research reactor conversions, (2) international research reactor conversions, and (3) Molybdenum 99 (Mo99) efforts, which include international Mo99 isotope production reactor conversions and projects to establish new U.S. non-HEU Mo99 production facilities. The subprogram’s current goal is to convert or verify shutdown of 156 HEU reactors and isotope production facilities and to support the establishment of a domestic, non-HEU-based Mo99 production capability. Tables 2 through 4 below list the U.S. reactor conversions, international reactor conversions or shutdowns, and Mo99 projects in the HEU Reactor Conversion subprogram’s planned scope of work, for each of the subprogram’s three components, as of July 2017.

Appendix III: Scope and Completion Dates for the Radiological Security Subprogram’s Source Protection Component

The Office of Defense Nuclear Nonproliferation’s Radiological Security subprogram’s current goal for the radiological source protection component is to upgrade security in 4,394 buildings worldwide by fiscal year 2033. Table 5 shows the estimated number of buildings to be completed each year as of June 2017.

Appendix IV: Comments from the National Nuclear Security Administration

Appendix V: GAO Contact and Staff Acknowledgments

GAO Contact

Shelby S. Oakley, (202) 512-3841 or oakleys@gao.gov.

Staff Acknowledgments

In addition to the individual named above, William E. Hoehn (Assistant Director), Natalie M. Block, R. Scott Fletcher, Brian M. Friedman, Cindy Gilbert, Jason T. Lee, TyAnn Lee, Duc Ngo, Jeanette Soares, Sheryl Stein, and Sara Sullivan made key contributions to this report.