Nuclear Weapons: NNSA Needs to Refine and More Effectively Manage Its New Approach for Assessing and Certifying Nuclear Weapons
Released: 2006-02-03T00:00:00

Letter

Background

Most modern nuclear warheads contain a nuclear explosive package, which contains the primary and the secondary, and a set of nonnuclear components. The nuclear detonation of the primary produces energy that drives the secondary, which produces further nuclear energy of a militarily significant yield. The nonnuclear components control the use, arming, and firing of the warhead. All nuclear weapons developed to date rely on nuclear fission to initiate their explosive release of energy. Most also rely on nuclear fusion to increase their total energy yield. Nuclear fission occurs when the nucleus of a heavy, unstable atom (such as uranium-235) is split into two lighter parts, which releases neutrons and produces large amounts of energy. Nuclear fusion occurs when the nuclei of two light atoms (such as deuterium and tritium) are joined, or fused, to form a heavier atom, with an accompanying release of neutrons and larger amounts of energy.

The U.S. nuclear stockpile consists of nine weapon types. (See table 1.) The lifetimes of the weapons currently in the stockpile have been extended well beyond the minimum life for which they were originally designed—generally about 20 years—increasing the average age of the stockpile and, for the first time, leaving NNSA with large numbers of weapons that are close to 30 years old.

Established in 1993, the Stockpile Stewardship Program faces two main technical challenges: provide (1) a better scientific understanding of the basic phenomena associated with nuclear weapons and (2) an improved capability to predict the impact of aging and remanufactured components on the safety and reliability of nuclear weapons. Specifically, An exploding nuclear weapon creates the highest pressures, greatest temperatures, and most extreme densities ever made by man on earth, within some of the shortest times ever measured. When combined, these variables exist nowhere else in nature. While the United States conducted about 1,000 nuclear weapons tests prior to the moratorium, these tests were conducted mainly to look at broad indicators of weapon performance (such as the yield of a weapon) and were often not designed to collect data on specific properties of nuclear weapons physics. After more than 60 years of developing nuclear weapons, while many of the physical processes are well understood and accurately modeled, the United States still does not possess a set of completely known and expressed laws and equations of nuclear weapons physics that link the physical event to first principles.

As nuclear weapons age, a number of physical changes can take place.

The effects of aging are not always gradual, and the potential for unexpected changes in materials causes significant concerns as to whether weapons will continue to function properly. Replacing aging components is, therefore, essential to ensure that the weapon will function as designed. However, it may be difficult or impossible to ensure that all specifications for the manufacturing of new components are precisely met, especially since each weapon was essentially handmade. In addition, some of the manufacturing process lines used for the original production have been disassembled.

In 1995, the President established an annual assessment and reporting requirement designed to help ensure that nuclear weapons remain safe and reliable without underground testing. As part of this requirement, the three weapons laboratories are required to issue a series of reports and letters that address the safety, reliability, performance, and military effectiveness of each weapon type in the stockpile. The letters, submitted to the Secretary of Energy individually by the laboratory directors, summarize the results of the assessment reports and, among other things, express the directors’ conclusions regarding whether an underground nuclear test is needed and the adequacy of various tools and methods currently in use to evaluate the stockpile.

To address these challenges, in 1999 DOE developed a new three-part program structure for the Stockpile Stewardship Program that included a series of campaigns, which DOE defined as technically challenging, multiyear, multifunctional efforts to develop and maintain the critical capabilities needed to continue assessing the safety and reliability of the nuclear stockpile into the foreseeable future without underground testing. DOE originally created 18 campaigns that were designed to focus its efforts in science and computing, applied science and engineering, and production readiness. Six of these campaigns currently focus on the development and improvement of the scientific knowledge, tools, and methods required to provide confidence in the assessment and certification of the safety and reliability of the nuclear stockpile in the absence of nuclear testing. These six campaigns are as follows: The Primary and Secondary campaigns were established to analyze and understand the different scientific phenomena that occur in the primary and secondary stages of a nuclear weapon during detonation. As such, the Primary and Secondary campaigns are intended to support the development and implementation of the QMU methodology and to set the requirements for the computers, computer models, and experimental data needed to assess and certify the performance of nuclear weapons.

The ASC campaign provides the leading-edge supercomputers and models that are used to simulate the detonation and performance of nuclear weapons.

Two campaigns—Advanced Radiography and Dynamic Materials Properties—provide data from laboratory experiments to support nuclear weapons theory and computational modeling. For example, the Advanced Radiography campaign conducts experiments that measure how stockpile materials behave when exposed to explosively driven shocks. One of the major facilities being built to support this campaign is the Dual Axis Radiographic Hydrodynamic Test Facility at LANL.

The ICF campaign develops experimental capabilities and conducts experiments to examine phenomena at high temperature and pressure regimes that approach but do not equal those occurring in a nuclear weapon. As a result, scientists currently have to extrapolate from the results of these experiments to understand similar phenomena in a nuclear weapon. One of the major facilities being built as part of this campaign is the National Ignition Facility at LLNL.

The other two program activities associated with the Stockpile Stewardship Program are “Directed Stockpile Work” and “Readiness in Technical Base and Facilities.” Directed Stockpile Work includes the activities that directly support specific weapons in the stockpile, such as the Stockpile Life Extension Program, which employs a standardized approach for planning and carrying out nuclear weapons refurbishment activities to extend the operational lives of the weapons in the stockpile well beyond their original design lives. The life extension for the W87 was completed in 2004, and three other weapon systems—the B61, W76, and W80—are currently undergoing life extensions. Each life extension program is specific to that weapon type, with different parts being replaced or refurbished for each weapon type. Readiness in Technical Base and Facilities includes the physical infrastructure and operational readiness required to conduct campaign and Directed Stockpile Work activities across the nuclear weapons complex. The complex includes the three nuclear weapons design laboratories (LANL, LLNL, and SNL), the Nevada Test Site, and four production plants—the Pantex Plant in Texas, the Y-12 Plant in Tennessee, a portion of the Savannah River Site in South Carolina, and the Kansas City Plant in Missouri.

From fiscal year 2001 through fiscal year 2005, NNSA spent over $7 billion on the six scientific campaigns (in inflation-adjusted dollars). (See table 2.) NNSA has requested almost $7 billion in funding for these campaigns over the next 5 years. (See table 3.)

Within NNSA, the Office of Defense Programs is responsible for managing the campaigns and the Stockpile Stewardship Program in general. Within this office, two organizations share responsibility for overall management of the scientific campaigns: the Office of the Assistant Deputy Administrator for Research, Development, and Simulation and the Office of the Assistant Deputy Administrator for Inertial Confinement Fusion and the National Ignition Facility Project. The first office oversees campaign activities associated with the Primary and Secondary campaigns—as well as the ASC, Advanced Radiography, and Dynamic Materials Properties campaigns—with a staff of about 13 people. The second office oversees activities associated with the ICF campaign with a single staff person. Actual campaign activities are conducted by scientists and other staff at the three weapons laboratories. LANL and LLNL conduct activities associated with the nuclear explosive package, while SNL performs activities associated with the nonnuclear components that control the use, arming, and firing of the nuclear warhead.

The QMU Methodology Is Highly Promising but Still in the Early Stages of Development

NNSA has endorsed the use of a new common methodology, known as the quantification of margins and uncertainties, or QMU, for assessing and certifying the safety and reliability of the nuclear stockpile. NNSA and laboratory officials told us that they have made progress in applying the principles of QMU to the certification and assessment of nuclear warheads in the stockpile. However, QMU is still in its early stages of development, and important differences exist among the three laboratories in their application of QMU. To date, NNSA has commissioned two technical reviews of the implementation of QMU at the weapons laboratories. While strongly supporting QMU, the reviews found that the development and implementation of QMU was still in its early stages. The reviews recommended that NNSA take steps to further define the technical details supporting the implementation of QMU and integrate the activities of the three weapons laboratories in implementing QMU. However, NNSA and the weapons laboratories have not fully implemented these recommendations. Beyond the issues raised in the two reports, we also found differences in the understanding and application of QMU among the three laboratories.

NNSA Has Endorsed QMU as a New, Common Methodology for Assessing and Certifying Stockpile Safety and Reliability

When the Primary and Secondary campaigns were established in 1999, they brought some organization and overall goals to the scientific research conducted across the weapons complex. For example, as we noted in April 2005, the Primary campaign set an initial goal in the 2005 to 2010 time frame for certifying the performance of the primary of a nuclear weapon to within a stated yield level. However, according to senior NNSA officials, NNSA still lacked a coherent strategy for relating the scientific work conducted by the weapons laboratories under the campaigns to the needs of the nuclear stockpile and the overall Stockpile Stewardship Program. This view was echoed by a NNSA advisory committee report, which stated in 2002 that the process used by the weapons laboratories to certify the safety and reliability of nuclear weapons was ill defined and unevenly applied, leading to major delays and inefficiencies in programs.

Starting in 2001, LLNL and LANL began developing what is intended to be a common methodology for assessing and certifying the performance and safety of nuclear weapons in the absence of nuclear testing. In 2003, the associate directors for nuclear weapons at LLNL and LANL published a white paper—entitled “National Certification Methodology for the Nuclear Weapon Stockpile”—that described this new methodology, which they referred to as the quantification of margins and uncertainties or QMU. According to the white paper, QMU is based on an adaptation of standard engineering practices and lends itself to the development of “rigorous, quantitative, and explicit criteria for judging the robustness of weapon system and component performance at a detailed level.” Moreover, the quantitative results of this process would enable NNSA and the weapons laboratories to set priorities for their activities and thereby make rational decisions about allocating program resources to the nuclear stockpile.

The process envisaged in the white paper focuses on creating a “watch list” of factors that, in the judgment of nuclear weapons experts, are the most critical to the operation and performance of a nuclear weapon. These factors include key operating characteristics and components of the nuclear weapon. For each identified, critical factor leading to a nuclear explosion, nuclear weapons experts would define performance metrics. These performance metrics would represent the experts’ best judgment of what constitutes acceptable behavior—i.e., the range of acceptable values for a critical function to successfully occur or for a critical component to function properly—as well as what constitutes unacceptable behavior or failure. To use an analogy, consider the operation of a gasoline engine. Some of the events critical to the operation of the engine would include the opening and closing of valves, the firing of the spark plugs, and the ignition of the fuel in each cylinder. Relevant performance metrics for the ignition of fuel in a cylinder would include information on the condition of the spark plugs (e.g., whether they are corroded) and the fuel/air mixture in the cylinder.

Once nuclear experts have identified the relevant performance metrics for each critical factor, according to the 2003 white paper, the goal of QMU is to quantify these metrics. Specifically, the QMU methodology seeks to quantify (1) how close each critical factor is to the point at which it would fail to perform as designed (i.e., the performance margin or the margin to failure) and (2) the uncertainty in calculating the margin. According to the white paper, the weapons laboratories would be able to use their calculated values of margins and uncertainties as a way to assess their confidence in the performance of a nuclear weapon. That is, the laboratories would establish a “confidence ratio” for each critical factor —they would divide their calculated value for the margin (“M”) by their calculations of the associated uncertainty (“U”) and arrive at a single number (“M/U”). According to the white paper, the weapons laboratories would only have confidence in the performance of a nuclear weapon if the margin “significantly” exceeds uncertainty for all critical issues. However, the white paper did not define what the term “significantly” meant.

In a broad range of key planning and management documents that have followed the issuance of the white paper, NNSA and the weapons laboratories have endorsed the use of the QMU methodology as the principal tool for assessing and certifying the safety and reliability of the nuclear stockpile in the absence of nuclear testing. For example, in its fiscal year 2006 implementation plan for the Primary campaign, NNSA stated as a strategic objective that it needs to develop the capabilities and understanding necessary to apply QMU as the assessment and certification methodology for the nuclear explosive package. In addition, in its fiscal year 2006 budget request, NNSA selected its progress toward the development and implementation of QMU as one of its major performance indicators. Finally, in the plans that NNSA uses to evaluate the performance of LANL and LLNL, NNSA has established an overall objective for LANL and LLNL to assess and certify the safety and reliability of nuclear weapons using a common QMU methodology.

Officials at NNSA and the weapons laboratories have also stated that QMU will be vital to certifying any weapon redesigns, such as are envisioned by the RRW program. For example, senior NNSA officials told us that the Stockpile Stewardship Program will not be sustainable if it only involves the continued refurbishment in perpetuity of existing weapons in the current nuclear stockpile. They stated that the accumulation of small changes over the extended lifetime of the current nuclear stockpile will result in increasing levels of uncertainty about its performance. If NNSA moves forward with the RRW program, according to NNSA documents and officials, the future goal of the weapons program will be to use QMU to replace existing stockpile weapons with an RRW whose safety and reliability could be assured with the highest confidence, without nuclear testing, for as long as the United States requires nuclear forces.

The Development and Implementation of QMU Is at an Early Stage and Important Differences Exist Among the Weapons Laboratories in their Application of QMU

According to NNSA and laboratory officials, the weapons laboratories have made progress in applying the principles of QMU to the certification of life extension programs and to the annual stockpile assessment process. For example, LLNL officials told us that they are applying QMU to the assessment of the W80, which is currently undergoing a life extension. They said that, in applying the QMU methodology, they tend to focus their efforts on identifying credible “failure modes,” which are based on observable problems, such as might be caused by the redesign of components in a nuclear weapon, changes to the manufacturing process for components, or the performance of a nuclear weapon under aged conditions. They said that, for the W80 life extension program, they have developed a list of failure modes and quantified the margins and uncertainties associated with these failure modes. Based on their calculations, they said that they have increased their confidence in the performance of the W80.

Similarly, LANL officials told us that they are applying QMU to the W76, which is also currently undergoing a life extension and is scheduled to finish its first production unit in 2007. They said that, in applying the QMU methodology, they tend to focus their efforts on defining “performance gates,” which are based on a number of critical points during the explosion of a nuclear weapon that separate the nuclear explosion into natural stages of operation. The performance gates identify the characteristics that a nuclear weapon must have at a particular time during its operation to meet its performance requirements (e.g., to reach its expected yield). LANL officials told us that they have developed a list of performance gates for the W76 life extension program and are beginning to quantify the margins and uncertainties associated with these performance gates.

Despite this progress, we found that QMU is still in its early stages of development and that important differences exist among the weapons laboratories in their application of QMU. To date, NNSA has commissioned two technical reviews of the implementation of QMU at the weapons laboratories. The first review was conducted by NNSA’s Office of Defense Programs Science Council (Science Council)—which advises NNSA on scientific matters across a range of activities, including those associated with the scientific campaigns—and resulted in a March 2004 report. The second review was conducted by the MITRE Corporation’s JASON panel and resulted in a February 2005 report. Both reports endorsed the use of QMU by the weapons laboratories and listed several potential benefits that QMU could bring to the nuclear weapons program. For example, according to the Science Council report, QMU will serve an important role in training the next generation of nuclear weapon designers and will quantify and increase NNSA’s confidence in the assessment and certification of the nuclear stockpile. According to the JASON report, QMU could become a useful management tool for directing investments in a given weapon system where they would be most effective in increasing confidence, as required by the life extension programs. In addition, the JASON report described how LANL and LLNL officials had identified potential failure modes in several weapon systems and calculated the associated margins and uncertainties. The report noted that, for most of these failure modes, the margin for success was large compared with the uncertainty in the performance.

However, according to both the Science Council and the JASON reports, the development and implementation of QMU is still in its early stages. For example, the JASON report described QMU as highly promising but unfinished, incomplete and evolving, and in the early stages of development. Moreover, the chair of the JASON panel on QMU told us in June 2005 that, during the course of his review, members of the JASON panel found that QMU was not mature enough to assess its reliability or usefulness. The reports also stated that the weapons laboratories have not fully developed or agreed upon the technical details supporting the implementation and application of QMU. For example, the JASON report stated that, in the course of its review, it became evident that there were a variety of differing and sometimes diverging reviews of what QMU really was and how it was working in practice. As an example, the report stated that some of the scientists, designers, and engineers at LANL and LLNL saw the role of expert judgment as an integral part of the QMU process, while others did not. In discussions with the weapons laboratories about the two reports, LANL officials told us that they believed that the details of QMU as a formal methodology are still evolving, while LLNL officials stated that QMU was “embryonic” and not fully developed.

While supporting QMU, the two reports noted that the weapons laboratories face challenges in successfully implementing a coherent and credible analytical method based on the QMU methodology. For example, in its 2004 report, the Science Council stated that, in its view, the QMU methodology is based on the following core assumptions: Computer simulations can accurately predict the behavior of a complex nuclear explosive system as a function of time.

It is sufficient for the assessment of the performance of a nuclear weapon to examine the simulation of the time evolution of a nuclear explosive system at a number of discrete time intervals and to determine whether the behavior of the system at each interval is within acceptable bounds.

The laboratories’ determinations of acceptable behavior can be made quantitatively—that is, they will make a quantitative estimate of a system’s margins and uncertainties.

Given these quantitative measures of the margins and uncertainties, it is possible to calculate the probability (or confidence level) that the nuclear explosive system will perform as desired.

However, the Science Council’s report noted that extraordinary degrees of complexity are involved in a rational implementation of QMU that are only beginning to be understood. For example, in order for the QMU methodology to have validity, it must sufficiently identify all critical failure modes, critical events, and associated performance metrics. However, as described earlier, the operation of an exploding nuclear weapon is highly integrated and nonlinear, occurs during a very short period of time, and reaches extreme temperatures and pressures. In addition, the United States does not possess a set of completely known and expressed laws and equations of nuclear weapons physics. Given these complexities, it will be difficult to demonstrate the successful implementation of QMU, according to the report. In addition, the Science Council stated that it was not presented with any evidence that there exists a method—even in principle—for calculating an overall probability that a nuclear explosive package will perform as designed from the set of quantitative margins and uncertainties at each time interval.

To address these and other issues, the two reports recommended that NNSA take steps to further define the technical details supporting the implementation of QMU and to integrate the activities of the three weapons laboratories in implementing QMU. For example, the 2004 Science Council report recommended that NNSA direct the associate directors for nuclear weapons at LANL and LLNL to undertake a major effort to define the details of QMU. In particular, the report recommended that a trilaboratory team be charged with defining a common language for QMU and identifying the important performance gates, failure modes, and other criteria in the QMU approach. The report stated that this agreed-upon “reference” set could then be used to support all analyses of stockpile issues. In addition, the report recommended that NNSA consider establishing annual or semiannual workshops for the three weapons laboratories to improve the identification, study, and prioritization of potential failure modes and other factors that are critical to the operation and performance of nuclear weapons.

Similarly, the 2005 JASON panel report noted that the meaning and implications of QMU are currently unclear. To rectify this problem, the report recommended that the associate directors for nuclear weapons at LANL and LLNL write a new, and authoritative, paper defining QMU and submit it to NNSA. Furthermore, the report recommended that the laboratories establish a formal process to (1) identify all failure modes and performance gates associated with QMU, using the same methodology for all weapon systems, and (2) establish better relationships between the concepts of failure modes and performance gates for all weapon systems in the stockpile.

However, NNSA and laboratory officials have not fully implemented these recommendations, particularly the recommendations of the Science Council. For example, while LLNL and LANL officials are drafting a new “white paper” on QMU that attempts to clarify some fundamental tenets of the methodology, officials from SNL are not involved in the drafting of this paper. In addition, NNSA has not required the three weapons laboratories to hold regular meetings or workshops to improve the identification, prioritization, and integration of failure modes, performance gates, and other critical factors.

According to NNSA’s Assistant Deputy Administrator for Research, Development, and Simulation, NNSA has not fully implemented the recommendations of the Science Council’s report partly because the report was intended more to give NNSA a sense of the status of the implementation of QMU than it was to provide recommendations. For example, the 2004 report states that the “friendly review,” as the report is referred to by NNSA, would not have budget implications and that the report’s findings and recommendations would be reported only to the senior management of the weapons laboratories. As a result, the Assistant Deputy Administrator told us that he had referred the recommendations to the directors of the weapons laboratories and told them to implement the recommendations as they saw fit.

Furthermore, LLNL and LANL officials disagreed with some of the statements in the Science Council report and stressed that, in using QMU, they do not attempt to assign an overall probability that the nuclear explosive package will perform as desired. That is, they do not attempt to add up calculations of margins and uncertainties for all the critical factors to arrive at a single estimate of margin and uncertainty, or a single confidence ratio, for the entire nuclear explosive package. Instead, they said that they focus on ensuring that the margin for each identified critical factor in the explosion of a nuclear weapon is greater than the uncertainty. However, they said that, for a given critical factor, they do combine various calculations of individual uncertainties that contribute to the total amount of uncertainty for that factor.

In addition, in addressing comments in the JASON report, LLNL and LANL officials stressed that QMU has always relied, and will continue to rely heavily, on the judgment of nuclear weapons experts. For example, LLNL officials told us that since there is no single definition of what constitutes a threshold for failure, they use expert judgment to decide what to put on their list of failure modes. They also said that the QMU methodology provides a way to make the entire annual assessment and certification process more transparent to peer review. Similarly, LANL officials said that they use expert judgment extensively in establishing performance metrics and threshold values for their performance gates. They said that expert judgment will always be a part of the scientific process and a part of QMU.

Beyond the issues raised in the two reports, we found that there are differences in the understanding and application of QMU among the three laboratories. For example, the three laboratories do not agree about the application of QMU to areas outside of the nuclear explosive package. Specifically, LLNL officials told us that the QMU methodology, as currently developed, only applies to the nuclear explosive package and not to the nonnuclear components that control the use, arming, and firing of the nuclear warhead. According to LLNL and LANL officials, SNL scientists can run hundreds of experiments to test their components and, therefore, can use normal statistical analysis in certifying the performance of nonnuclear components. As a result, according to LLNL and LANL officials, SNL does not have to cope with real uncertainty and does not “do” QMU. Furthermore, according to LLNL officials, SNL has chosen not to participate in the development of QMU with LLNL and LANL.

However, SNL officials told us that while some of the nonnuclear components are testable to a degree, SNL is as challenged as the other two weapons laboratories in certifying the performance of their systems without actual testing. For example, SNL officials said that they simply do not have enough money to perform enough tests on all of their nonnuclear components to be able to rely completely on statistical analysis to meet their safety performance levels. In addition, SNL scientists are not able to test their components under the conditions of a nuclear explosion but are still required to certify the performance of the components under these conditions. Thus, SNL officials told us that they had been using their own version of QMU for a long time.

SNL officials told us that they define QMU as a way to make risk-informed decisions about the effect of variabilities and uncertainties on the performance of a nuclear weapon, including the nonnuclear components that control the use, arming, and firing of the nuclear warhead. Moreover, they said that this kind of risk-informed approach is not unique to the nuclear weapons laboratories and is used extensively in areas such as nuclear reactor safety. However, they told us that they have been left out in the development of QMU by the two other weapons laboratories. Specifically, they said that while SNL scientists have worked with other scientists at LANL and LLNL at a “grass roots” level, there has only been limited cooperation and dialogue between upper-level management at the three laboratories concerning the development and implementation of QMU.

In addition, we found that while LLNL and LANL both agree on the fundamental tenets of QMU at a high level, their application of the QMU methodology differs in some important respects. For example, LLNL and LANL officials told us that, at a detailed level, the two laboratories are pursuing different approaches to calculating and combining uncertainties. For the W80 life extension program, LLNL officials showed us how they combined calculations of individual uncertainties that contributed to the total uncertainty for a key failure mode of the primary—the amount of primary yield necessary to drive the secondary. However, they said that the scientific support for their method for combining individual calculations of uncertainty was limited, and they stated that they are pursuing a variety of more sophisticated analyses to improve their current approach.

Moreover, the two laboratories are taking a different approach to generating a confidence ratio for each critical factor, as described in the 2003 white paper on QMU. For example, for the W80 life extension program, LLNL officials showed us how they calculated a single confidence ratio for a key failure mode of the primary, based on their calculations of margin and uncertainty. They said that the weapon systems for which they are responsible have a lot of margin built into them, and they feel comfortable generating this number. In contrast, in discussions with LANL officials about the W76 life extension program, LANL officials told us that they prefer not to calculate a single confidence ratio for a performance gate, partly because they are concerned that their customers (e.g., the Department of Defense) might think that the QMU methodology is more formal than it is currently.

In commenting on the differences between the two laboratories, NNSA officials stated that the two laboratories are pursuing complementary approaches, and that these differences are part of the rationale for a national policy decision to maintain two nuclear design laboratories. In addition, they stated that the confidence in the correctness of scientific research is improved by achieving the same answer through multiple approaches. LLNL officials also made similar comments, stating that the nation will benefit from some amount of independence between the laboratories to assure that the best methodology for assessing the stockpile in the absence of nuclear testing is achieved.

NNSA’s Management of the Development and Implementation of QMU Is Deficient in Four Key Areas

NNSA relies on its Primary and Secondary campaigns to manage the development and implementation of QMU. According to NNSA policies, campaign managers at NNSA headquarters are responsible for developing campaign plans and high-level milestones, overseeing the execution of these plans, and providing input to the evaluation of the performance of the weapons laboratories. However, NNSA’s management of these processes is deficient in four key areas. First, the planning documents that NNSA has established for the Primary and Secondary campaigns do not adequately integrate the scientific research currently conducted that supports the development and implementation of QMU. Second, NNSA has not developed a clear, consistent set of milestones to guide the development and implementation of QMU. Third, NNSA has not established formal requirements for conducting annual, technical reviews of the implementation of QMU or for certifying the completion of QMU-related milestones. Finally, NNSA has not established adequate performance measures to determine the progress of the laboratories in developing and implementing QMU.

Campaign Planning Documents Do Not Adequately Integrate the Scientific Activities Supporting QMU

As part of its planning structure, NNSA requires the use of program and implementation plans to set requirements and manage resources for the campaigns and other programs associated with the Stockpile Stewardship Program. Program plans are strategic in nature and identify the long-term goals, high-level milestones, and resources needed to support a particular program over a 7-year period, while implementation plans establish performance expectations for the program and each participating site for the current year of execution. According to NNSA policies, program and implementation plans should flow from and interact with each other using a set of cascading goals and requirements.

NNSA has established a single program plan, which it calls the “Science campaign program plan,” that encompasses the Primary and the Secondary campaigns, as well as two other campaigns—Advanced Radiography and Dynamic Materials Properties. NNSA has also established separate implementation plans for each of these campaigns, including the Primary and Secondary campaigns. According to NNSA, it relies on these plans—and in particular the plans related to the Primary and Secondary campaigns—to manage the development and implementation of QMU, as well as to determine the requirements for the experimental data and computer modeling needed to analyze and understand the different scientific phenomena that occur in a nuclear weapon during detonation.

However, the current Primary and Secondary campaign plans do not contain a comprehensive, integrated list of the relevant scientific research being conducted across the weapons complex to support the development and implementation of QMU. For example, according to the NNSA campaign manager for the Primary campaign, he had to hold a workshop in 2005 with officials from the weapons laboratories in order to catalogue all of the scientific activities that are currently performed under the heading of “primary assessment” regardless of the NNSA funding source. According to this official, the existing Primary campaign implementation plan does not provide the integration across NNSA programs that is needed to achieve the goals of the Primary campaign and to develop and implement QMU.

According to NNSA officials, the lack of integration has occurred in large part because a significant portion of the scientific research that is relevant to the Primary and Secondary campaigns is funded and carried out by different campaigns and other programs. Specifically, different NNSA campaign managers use different campaign planning documents to plan and oversee research and funding for activities that are directly relevant to the Primary and Secondary campaigns and the development and implementation of QMU. For example, the ASC campaign provides the supercomputing capability that the weapons laboratories use to simulate and predict the behavior of an exploding nuclear weapon. Moreover, the weapons laboratories rely on ASC supercomputers to quantify their uncertainties with respect to the accuracy of these computer simulations—a key component in the implementation of QMU. As a result, the ASC campaign plans and funds activities that are critical to the development and implementation of QMU.

To address this problem, according to NNSA officials, NNSA is taking steps to establish better relationships among the campaign plans. For example, NNSA is currently drafting a new plan—which it calls the Primary Assessment Plan—in an attempt to better coordinate the activities covered under the separate program and implementation plans. The draft plan outlines high-level research priorities, time lines, and proposed milestones necessary to support (1) NNSA’s responsibilities for the current stockpile, (2) primary physics design for the development of an RRW, and (3) certification of an RRW in the 2012 time frame and a second RRW in the 2018 time frame. According to NNSA officials, they expect to finalize this plan by the third quarter of fiscal year 2006. In addition, they expect to have a similar plan for the Secondary campaign finalized by December 2006 and are considering combining both plans into a full-system assessment plan. According to one NNSA official responsible for the Primary and Secondary campaigns, NNSA will revise the existing campaign program and implementation plans to be consistent with the Primary Assessment Plan.

More fundamentally, some nuclear weapons experts have suggested that NNSA’s planning structure should be reorganized to better reflect the use of QMU as NNSA’s main strategy for assessing and certifying the performance of nuclear weapons. For example, the chair of the LLNL Defense and Nuclear Technologies Director’s Review Committee—which conducts technical reviews of LLNL’s nuclear weapons activities for the University of California—told us that the current campaign structure has become a series of “stovepipes” that NNSA uses to manage stockpile stewardship. He said that in order for NNSA to realize its long-term goals for implementing QMU, NNSA is going to have to reorganize itself around something that he called an “uncertainty spreadsheet” for each element of a weapon’s performance (e.g., implosion of the primary, transfer of energy to the secondary, etc.), leading to the weapon’s yield. He said that the laboratories should develop a spreadsheet for each weapon in the stockpile that (1) identifies the major sources of uncertainty at each critical event in their assessment of the weapon’s performance and (2) relates the laboratory’s scientific activities and milestones to these identified sources of uncertainty. He said that the development and use of these spreadsheets would essentially capture the intent of the scientific campaigns and make them unnecessary.

NNSA Does Not Have a Clear, Consistent Set of QMU-Related Milestones

NNSA has established a number of milestones that relate to the development and implementation of QMU. Within the Science campaign program plan, NNSA has established a series of high-level milestones, which it calls “level-1” milestones. According to NNSA policies, level-1 milestones should be sufficient enough to allow strategic integration between sites involved in the campaigns and between programs in NNSA. Within the implementation plans for the Primary and Secondary campaigns, NNSA has established a number of lower-level milestones, which it calls “level-2” milestones, which NNSA campaign managers use to track major activities for the current year of execution. The level-1 milestones related to QMU are shown in table 4, and the level-2 milestones related to QMU for the Primary campaign are shown in table 5.

According to NNSA officials, the level-1 milestones in table 4 represent a two-stage path to systematically identify uncertainties and reduce them through analyzing past underground test results, developing new experimental capabilities, and performing new experiments to understand the relevant physical processes. According to these level-1 milestones, NNSA expects to complete the second stage or “cycle” of this process by fiscal year 2014 (i.e., milestone M20), at which time NNSA will have sufficiently reduced major sources of uncertainties and will have confidence in its ability to predict the performance of nuclear weapons in the absence of nuclear testing.

However, we identified several problems with the NNSA milestones related to the development and implementation of QMU. Specifically, the level-1 milestones in the Science campaign program plan have the following problems: The milestones are not well-defined and never explicitly mention QMU.

According to NNSA officials responsible for overseeing the Primary campaign, these milestones are too qualitative and too far in the future to enable NNSA to effectively plan for and oversee the implementation of QMU. They described these milestones as “fuzzy” and said that they need to be better defined. However, NNSA officials also stated that these milestones are not just for QMU but for the entire Science campaign, of which QMU is only a part.

The milestones conflict with the performance measures shown in other important NNSA management documents. Specifically, while the Science campaign program plan envisions a two-stage path to identify and reduce key uncertainties related to nuclear weapon operations using QMU by 2014, the performance measures in NNSA’s fiscal year 2006 budget request and in Appendix A of the Science campaign program plan call for the completion of QMU by 2010.

The milestones have not been integrated with other QMU-related level-1 milestones in other planning documents. For example, the current ASC campaign program plan contains a series of level-1 milestones for completing the certification of several weapon systems—including the B61, W80, W76, and W88—with quantified margins and uncertainties by the end of fiscal year 2007. However, these milestones do not appear in and are not referenced by the Science campaign program plan. Moreover, the ASC campaign manager told us that, until recently, he was not aware of the existence of the level-1 milestones for implementing QMU that are contained in the Science campaign program plan.

In addition, we found that neither the Science campaign program plan nor the Primary campaign implementation plan describe how the level-2 milestones on QMU in the Primary campaign implementation plan are related to the level-1 milestones on QMU in the Science campaign program plan. Consequently, it is unclear how the achievement of specific level-2 milestones—such as the development of probabilistic tools and methods to combine various sources of uncertainty for primary performance—will result in the achievement of level-1 milestones for the implementation of QMU or how NNSA expects to certify several major nuclear weapon systems using QMU before the QMU methodology is fully developed and implemented.

NNSA, as well as laboratory officials, agreed that there are weaknesses with the current QMU milestones. According to NNSA officials, when NNSA established the current tiered structure for campaign milestones in 2003, the different tiers of milestones served different purposes and, therefore, were never well-integrated. For example, NNSA officials said that the level-1 milestones were originally created to reflect measures that were deemed to be important to senior NNSA officials, while level-2 milestones were created to be used by NNSA campaign managers to perform more technical oversight of the weapons laboratories. Furthermore, according to NNSA officials, the current level-2 milestones are only representative of campaign activities conducted by the weapons laboratories. That is, the level-2 milestones were never designed to cover the entire scope of work being conducted by the weapons laboratories and are, therefore, not comprehensive in scope.

To address these problems, according to NNSA officials, NNSA is taking steps to develop better milestones to track the implementation of the QMU methodology. For example, in the draft Primary Assessment Plan, NNSA has established 19 “high-level” milestones that cover the time period from fiscal year 2006 to fiscal year 2018. According to these draft milestones, by fiscal year 2010, NNSA expects to “complete the experimental work and methodology development needed to demonstrate the ability of primary certification tools to support certification of existing stockpile system and RRW.” In addition, NNSA expects to certify a RRW in fiscal year 2012 and a second RRW in fiscal year 2018.

NNSA Has Not Established Formal Requirements for Conducting Technical Reviews or Certifying the Completion of QMU-Related Milestones

According to NNSA policies, campaign managers are required to track the status of level-1 and level-2 milestones and provide routine, formal reports on the status of their programs. For example, campaign managers are required to track, modify, and score the status of level-1 and level-2 milestones through the use of an Internet-based application called the Milestone Reporting Tool. On a quarterly basis, campaign managers assign one of four possible scores for each milestone listed in the application: (1) “blue” for completed milestones, (2) “green” for milestones that are on track to be finished by the end of the fiscal year, (3) “yellow” for milestones that may not be completed by the end of the fiscal year, and (4) “red” for milestones that will not be completed by the end of the fiscal year. At quarterly program review meetings, campaign managers brief senior-level NNSA officials on the status of major milestones, along with cost and expenditure data for their programs. In addition, campaign managers are responsible for conducting technical reviews of the campaigns for which they are responsible, at least annually, to ensure that campaign activities are being executed properly and that campaign milestones are being completed.

However, NNSA campaign managers have not met all of the NNSA requirements needed to effectively oversee the Primary and Secondary campaigns. For example, we found that the campaign managers for the Primary and Secondary campaigns have not established formal requirements for conducting annual, technical reviews of the implementation of QMU at the three weapons laboratories. Moreover, these officials have not established requirements for certifying the completion of level-2 milestones that relate to QMU. They could not provide us with documentation showing the specific activities or outcomes that they expected from the weapons laboratories in order to certify that the laboratories had completed the level-2 milestones for QMU. Instead, they relied more on ad hoc reviews of campaign activities and level-2 milestones as part of their oversight activities for their campaigns. According to the Primary campaign manager, the officials at the weapons laboratories are the principal managers of campaign activities. As a result, he views his role as more of a “sponsor” for his program and, therefore, does not require any written reports or evidence from the laboratories to certify that they have completed specific milestones.

In contrast, we found that the ASC campaign manager has established formal requirements for a variety of reoccurring technical reviews of activities associated with the ASC campaign. Specifically, the ASC campaign relies on semiannual reviews conducted by the ASC Predictive Science Committee—which provides an independent, technical review of the status of level-2 milestones—as well as on annual “principal investigators” meetings that provide a technical review of every program element within the ASC campaign. The ASC campaign manager told us that he relies on these technical reviews to oversee program activities because the quarterly program review meetings are not meant to help him manage his program but are really a way for senior-level NNSA officials to stay informed.

In addition, the ASC campaign manager has established detailed, formal requirements for certifying the completion of level-2 milestones for the ASC campaign. Specifically, the fiscal year 2006 implementation plan for the ASC campaign contains a detailed description of what NNSA expects from the completion of each level-2 milestone, including a description of completion criteria, the method by which NNSA will certify the completion of the milestone, and an assessment of the risk level associated with the completion of the milestone. The ASC campaign manager told us that, when NNSA officials created the level-2 milestones for the campaigns in 2003, the milestones were really just “sentences” and lacked the detailed criteria that would enable NNSA managers to adequately track and document the completion of major milestones. As a result, the ASC campaign has made a major effort in recent years to develop detailed, formal requirements to support the completion of ASC level-2 milestones.

NNSA Has Not Established Adequate Measures to Determine the Laboratories’ Performance in Developing and Implementing QMU

NNSA uses performance measurement data to inform resource decisions, improve the management and delivery of products and services, and justify budget requests. According to NNSA requirements, performance measurement data should explain in clear, concise, meaningful, and measurable terms what program officials expect to accomplish for a specific funding level over a fixed period of time. In addition, performance measurement data should include annual targets that describe specific outputs that can be measured, audited, and substantiated by the detailed technical milestones contained in documentation such as campaign implementation plans.

With respect to QMU, NNSA has established an overall annual performance target to measure the cumulative percentage of progress toward the development and implementation of the QMU methodology. Specifically, in its fiscal year 2006 budget request to the Congress, NNSA stated that it expects to complete the development and implementation of QMU by 2010 as follows: 25 percent complete by the end of fiscal year 2005, 40 percent complete by the end of fiscal year 2006, 55 percent complete by the end of fiscal year 2007, 70 percent complete by the end of fiscal year 2008, 85 percent complete by the end of fiscal year 2009, and 100 percent complete by the end of fiscal year 2010.

According to NNSA, it had progressed 10 percent toward its target of completing QMU by the end of fiscal year 2004. However, NNSA officials could not document how they can measure progress toward the performance target for developing and implementing QMU. Moreover, NNSA officials could not explain how the 2010 overall performance target for the completion and implementation of QMU is related to the level-1 milestones for QMU in the Science campaign program plan, which describes a two-stage process to identify and reduce key uncertainties in nuclear weapon performance using QMU by 2014. According to one NNSA official responsible for overseeing the Primary campaign, NNSA created this annual performance target because the Office of Management and Budget requires agencies to express some of their annual performance targets in percentage terms. However, this official said the actual percentages are not very meaningful, and he does not have any specific criteria for how to measure progress to justify the use of the percentages in the budget request.

NNSA has also established broad performance measures to evaluate the performance of LANL and LLNL. Specifically, in its performance evaluation plans for LANL and LLNL for fiscal year 2006, NNSA has established the following three performance measures: Use progress toward quantifying margins and uncertainty, and experience in application, to further refine and document the QMU methodology.

Demonstrate application of a common assessment methodology (i.e., QMU) in major warhead assessments and the certification of Life Extension Program warheads.

Complete the annual assessment of the safety, reliability, and performance of all warhead types in the stockpile, including reaching conclusions on whether nuclear testing is required to resolve any issues.

However, the plan that NNSA uses to evaluate the performance of SNL does not contain any performance measures or targets specifically related to QMU, and the performance evaluation plans for LANL and LLNL do not contain any annual targets that can be measured and linked to the specific performance measures related to QMU. Instead, the plans state that NNSA will rely on LLNL and LANL officials to develop the relevant targets and related dates for each performance measure, as well as to correlate the level-1 and level-2 milestones with these measures. When asked why these plans do not meet NNSA’s own requirements, NNSA officials said that they have not included specific annual performance targets in the plans because to do so would make it harder for them to finalize the plans and adjust to changes in NNSA’s budget. However, they said that NNSA is planning on implementing more stringent plans that will include annual performance targets when the next contract for LANL and LLNL is developed. In addition, NNSA officials told us that they recognize the need to develop performance measures related to QMU for SNL and anticipate implementing these changes in the fiscal year 2007 performance evaluation plan.

NNSA officials told us that they have used more specific measures, such as the completion of level-2 milestones, in their assessment of the weapons laboratories’ performance since fiscal year 2004. However, we also found problems with the way NNSA has assessed the performance of the weapons laboratories in implementing QMU. For example, in NNSA’s annual performance appraisal of LANL for fiscal year 2004, NNSA states that LANL had completed 75 percent of the work required to develop “QMU logic” for the W76 life extension by the end of fiscal year 2004. However, NNSA officials could not document how they are able to measure progress toward the development and implementation of QMU logic for the W76 life extension. Again, an NNSA official responsible for overseeing the Primary campaign told us that the actual percentages are not very meaningful, and that he did not have any specific criteria for how to measure progress to justify the use of the percentage in the appraisal.

In a recent report, we recognized the difficulties of developing useful results-oriented performance measures for programs such as those geared toward research and development programs. For programs that can take years to observe program results, it can be difficult to identify performance measures that will provide information on the annual progress they are making toward achieving program results. However, we also recognize that such efforts have the potential to provide important information to decision makers.

NNSA officials told us that they recognize the need for developing appropriate measures to ensure that adequate progress is being maintained toward achieving the goals and milestones of the campaigns. However, according to NNSA, very few products of the scientific campaigns involve the repetition of specific operations whose costs can be monitored effectively as a measure of performance. As a result, the best measure of progress for the scientific campaigns is through scientific review by qualified technical peers at appropriate points in the program. However, NNSA has not established any performance measures or targets for implementing QMU that require periodic scientific peer reviews or define what is meant by “appropriate” points in the program.

Conclusions

Faced with an aging nuclear stockpile, as well as an aging workforce, NNSA needs a methodologically rigorous, transparent, and explainable approach for how it will continue to assess and certify the safety and reliability of the nation’s nuclear weapons stockpile, now and into the foreseeable future, without underground testing. After over a decade of conducting stockpile stewardship, NNSA’s selection of QMU as its methodology for assessment and certification represents a positive step toward a methodologically rigorous, transparent, and explainable approach that can be carried out by a new cadre of weapons designers. However, important technical and management details must be resolved before NNSA can say with certainty that it has a sound and agreed upon approach.

First, NNSA must take steps to ensure that all three nuclear weapons laboratories—not just LANL and LLNL—are in agreement about how QMU is to be defined and applied. While we recognize that there will be methodological differences between LANL and LLNL in the detailed application of QMU to specific weapon systems, we believe that it is fundamentally important that these differences be understood and, if need be, reconciled, to ensure that QMU achieves the goal of a common methodology with rigorous, quantitative, and explicit criteria, as envisioned by the original 2003 white paper on QMU. More importantly, we believe that SNL has an important role in the development and application of QMU to the entire warhead, and we find the continuing disagreement over the application of QMU to areas outside of the nuclear explosive package to be disconcerting. There have been several recommendations calling for a new, technical paper defining QMU, as well as the establishment of regular forums to further develop the QMU methodology and reconcile any differences in approach. We believe the NNSA needs to fully implement these recommendations.

Second, NNSA has not made effective use of its current planning and program management structure to ensure that all of the research needed to support QMU is integrated and that scarce scientific resources are being used efficiently. We believe that NNSA must establish an integrated management approach involving planning, oversight, and evaluation methods that are all clearly linked to the overall goal of the development and application of QMU. In particular, we believe that NNSA needs clear, consistent, and realistic milestones and regular, technical reviews of the development of QMU in order to ensure sound progress. Finally, while we support the development of QMU and believe it must be effectively managed, we also believe it is important to recognize and acknowledge that the development and application of QMU, especially the complexities involved in analyzing and combining uncertainties related to potential failure modes and performance margins, represents a daunting research challenge that may not be achievable in the time constraints created by an aging nuclear stockpile.

Recommendations for Executive Action

To ensure that the weapons laboratories will have the proper tools in place to support the continued assessment of the existing stockpile or the certification of redesigned nuclear components under the RRW program, we recommend that the Administrator of NNSA take the following two actions: Require the three weapons laboratories to formally document an agreed upon, technical description of the QMU methodology that clearly recognizes and reconciles any methodological differences.

Establish a formal requirement for periodic collaboration between the three weapons laboratories to increase their mutual understanding of the development and implementation of QMU.

To ensure that NNSA can more effectively manage the development and implementation of QMU, we recommend that the Administrator of NNSA take the following three actions: Develop an integrated plan for implementing QMU that contains (1) clear, consistent, and realistic milestones for the development and implementation of QMU across the weapons complex and (2) formal requirements for certifying the completion of these milestones.

Establish a formal requirement for conducting annual, technical reviews of the scientific research conducted by the weapons laboratories that supports the development and implementation of QMU.

Revise the performance evaluation plans for the three weapons laboratories so that they contain annual performance targets that can be measured and linked to specific milestones related to QMU.

Agency Comments and Our Evaluation

We provided NNSA with a draft of this report for their review and comment. Overall, NNSA agreed that there was a need for an agreed-upon technical approach for implementing QMU and that NNSA needed to improve the management of QMU through clearer, long-term milestones and better integration across the program. However, NNSA stated that QMU had already been effectively implemented and that we had not given NNSA sufficient credit for its success. In addition, NNSA raised several issues about our conclusions and recommendations regarding their management of the QMU effort. The complete text of NNSA’s comments on our draft report is presented in appendix I. NNSA also made technical clarifications, which we incorporated in this report as appropriate.

With respect to whether QMU has already been effectively implemented, during the course of our work, LANL and LLNL officials showed us examples of where they used the QMU methodology to examine specific issues associated with the stockpile. At the same time, during our discussions with laboratory officials, as well as with the Chairs of the JASON panel on QMU, the Office of Defense Programs Science Counsel, and the Strategic Advisory Group Stockpile Assessment Team of the U.S. Strategic Command, there was general agreement that the application of the QMU methodology was still in the early stages of development. As NNSA pointed out in its letter commenting on our report, to implement QMU, the weapons laboratories need to make a number of improvements, including techniques for combining different kinds of uncertainties, as well as developing better models for a variety of complex processes that occur during a nuclear weapon explosion. In addition, the successful implementation of QMU will continue to rely on the expert judgment and the successful completion of major scientific facilities such as the National Ignition Facility. We have modified our report to more fully recognize that QMU is being used by the laboratories to address stockpile issues and to more completely characterize its current state of development. At the same time, however, because QMU is still under development, we continue to believe that NNSA needs to make more effective use of its current planning and program management structure.

NNSA raised several specific concerns about our conclusions and recommendations. First, NNSA disagreed with our conclusion and associated recommendations that NNSA take steps to ensure that all three nuclear weapons laboratories are in agreement about how QMU is to be defined and applied. NNSA stated that we overemphasized the differences between LANL and LLNL in implementing QMU and that, according to NNSA, LANL and LLNL have a “common enough” agreement on QMU to go forward with its implementation. Moreover, NNSA stated that our recommendations blur very clear distinctions between SNL and the two nuclear design labs. According to NNSA, QMU is applied to issues regarding the nuclear explosive package, which is the mission of LANL and LLNL.

While we believe that some of the technical differences between the laboratories remain significant, we have revised our report to more accurately reflect the nature of the differences between LANL and LLNL. With respect to SNL, we would again point out that SNL officials are still required to certify the performance of nuclear weapon components under the conditions of a nuclear explosion and, thus, use similar elements of the QMU methodology. Therefore, we continue to believe that all three laboratories, as well as NNSA, would benefit from efforts to more formally document the QMU methodology and regularly meet to increase their mutual understanding. As evidence of the benefits of this approach, we would note that LLNL and LANL are currently developing a revised “white paper” on QMU, and that in discussions with one of the two authors, he agreed that inclusion of SNL in the development of the draft white paper could be beneficial.

Second, NNSA made several comments with respect to our recommendation that NNSA develop an integrated plan for implementing QMU that contains clear, consistent, and realistic milestones. For example, NNSA stated that they expect to demonstrate the success of the implementation of QMU and the scientific campaigns by the performance of a scientifically defensible QMU analysis for each required certification problem. In addition, NNSA stated that the 2010 budget target and the 2014 milestone were developed for different purposes and measure progress at different times. According to NNSA, the 2010 target describes developing QMU to the point that it can be applied to certification of a system (e.g., the W88) without underground testing, while the 2014 milestone is intended to be for the entire Science campaign effort.

However, as we state in our report, and as acknowledged by NNSA officials responsible for the Primary and Secondary campaigns, there continue to be problems with the milestones that NNSA has established for implementing QMU. Among these problems is the fact that these milestones are not well-defined and conflict with other performance measures that NNSA has established for QMU. Moreover, in its comments on our report, NNSA agreed that better integration and connectivity of milestones between various program elements would improve the communications of the importance of program goals and improve the formality of coordination of program activities, “which is currently accomplished in an informal and less visible manner.” Given this acknowledgment by NNSA, we continue to believe that an integrated plan for implementing QMU, rather than NNSA’s current ad hoc approach, is warranted.

Third, NNSA made several comments regarding our recommendation that NNSA establish a formal requirement for conducting annual, technical reviews of the scientific research conducted by the weapons laboratories that supports the development and implementation of QMU. NNSA stated that it believes the ad hoc reviews it conducts, such as the JASON review, provide sufficient information on scientific achievements, difficulties, and required redirection to manage these programs effectively. As a result, NNSA stated that it has not selected a single review process to look at overall success in the implementation of QMU but expects to continue to rely on ad hoc reviews.

We agree that reviews, such as the JASON review, are helpful, and we relied heavily on the JASON review, as well as other reviews as part of our analysis. However, as we point out in the report, the issue is that the campaign managers for the Primary and Secondary campaigns do not meet all of NNSA’s own requirements for providing effective oversight, which include the establishment of formal requirements for conducting technical reviews of campaign activities. Therefore, we believe that NNSA needs to take steps to implement its own policies. In addition, we believe that the ASC campaign provides a good role model for how the Primary and Secondary campaigns should be managed.

Finally, NNSA made several comments with respect to our recommendation for NNSA to revise the performance evaluation plans for the laboratories so that they contain annual performance targets that can be measured and linked to specific milestones related to QMU. Specifically, NNSA stated that the implementation of QMU is an area where it is difficult to establish a meaningful metric. According to NNSA, since QMU is implicitly evaluated in every review of the components of the science campaign, NNSA does not believe it is necessary to formally state an annual QMU requirement. However, as we point out in the report, the current performance evaluation plans for LANL and LLNL do not meet NNSA’s own requirements for the inclusion of annual performance targets that can be measured and linked to the specific performance measures related to QMU. More fundamentally, since NNSA has placed such emphasis on the development and implementation of QMU in the years ahead, we continue to believe that NNSA needs to develop more meaningful criteria for assessing the laboratories’ progress in developing and implementing QMU.

We are sending copies of this report to the Administrator, NNSA; the Director of the Office of Management and Budget; and appropriate congressional committees. We also will make copies available to others upon request. In addition, the report will be available at no charge on the GAO Web site at http://www.gao.gov.

If you or your staff have any questions about this report or need additional information, please contact me at (202) 512-3841 or aloisee@gao.gov. Contact points for our Offices of Congressional Relations or Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix II.

Comments from the National Nuclear Security Administration

GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the individual named above, James Noel, Assistant Director; Jason Holliday; Keith Rhodes; Peter Ruedel; and Carol Herrnstadt Shulman made key contributions to this report.