Nuclear Weapons: NNSA Needs to Refine and More Effectively Manage Its New Approach for Assessing and Certifying Nuclear Weapons
Published: 2006-02-03T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: In 1992, the United States began a unilateral moratorium on the testing of nuclear weapons. To compensate for the lack of testing, the Department of Energy's National Nuclear Security Administration (NNSA) developed the Stockpile Stewardship Program to assess and certify the safety and reliability of the nation's nuclear stockpile without nuclear testing. In 2001, NNSA's weapons laboratories began developing what is intended to be a common framework for a new methodology for assessing and certifying the safety and reliability of the nuclear stockpile without nuclear testing. GAO was asked to evaluate (1) the new methodology NNSA is developing and (2) NNSA's management of the implementation of this new methodology.
section_title: What GAO Found
paragraphs: NNSA has endorsed the use of the "quantification of margins and uncertainties" (QMU) methodology as its principal method for assessing and certifying the safety and reliability of the nuclear stockpile. Starting in 2001, Los Alamos National Laboratory (LANL) and Lawrence Livermore National Laboratory (LLNL) officials began developing QMU, which focuses on creating a common "watch list" of factors that are the most critical to the operation and performance of a nuclear weapon. QMU seeks to quantify (1) how close each critical factor is to the point at which it would fail to perform as designed (i.e., the margin to failure) and (2) the uncertainty that exists in calculating the margin, in order to ensure that the margin is sufficiently larger than the uncertainty. According to NNSA and laboratory officials, they intend to use their calculations of margins and uncertainties to more effectively target their resources, as well as to certify any redesigned weapons envisioned by the Reliable Replacement Warhead program. According to NNSA and weapons laboratory officials, they have made progress in applying the principles of QMU to the assessment and certification of nuclear warheads in the stockpile. NNSA has commissioned two technical reviews of the implementation of QMU. While strongly supporting QMU, the reviews found that the development and implementation of QMU was still in its early stages and recommended that NNSA further define the technical details supporting the implementation of QMU and integrate the activities of the three weapons laboratories in implementing QMU. GAO also found important differences in the understanding and application of QMU among the weapons laboratories. For example, while LLNL and LANL both agree on the fundamental tenets of QMU at a high level, they are pursuing different approaches to calculating and combining uncertainties. NNSA uses a planning structure that it calls "campaigns" to organize and fund its scientific research. According to NNSA policies, campaign managers at NNSA headquarters are responsible for developing plans and high-level milestones, overseeing the execution of these plans, and providing input to the evaluation of the performance of the weapons laboratories. However, NNSA's management of these processes is deficient in four key areas. First, NNSA's existing plans do not adequately integrate the scientific research currently conducted across the weapon complex to support the development and implementation of QMU. Second, NNSA has not developed a clear, consistent set of milestones to guide the development and implementation of QMU. Third, NNSA has not established formal requirements for conducting annual, technical reviews of the implementation of QMU at the three laboratories or for certifying the completion of QMU-related milestones. Finally, NNSA has not established adequate performance measures to determine the progress of the three laboratories in developing and implementing QMU.

section_title: Letter
paragraphs: In 1992, the United States began a unilateral moratorium on the testing of  nuclear weapons. Prior to the moratorium, underground nuclear testing  was a critical component of the evaluation and certification of the  performance of a nuclear weapon. Confidence in the continued  performance of stockpiled weapons relied heavily on the expert judgment  of weapon designers who had significant experience with successful  nuclear tests. In addition, the training of new weapon designers depended  on continued nuclear testing. In 1993, the Department of Energy (DOE), at  the direction of the President and the Congress, established the Stockpile  Stewardship Program to ensure the preservation of the United States’ core  intellectual and technical competencies in nuclear weapons without  testing. The National Nuclear Security Administration (NNSA), a  separately organized agency within DOE, is now responsible for carrying  out the Stockpile Stewardship Program, which includes activities  associated with the research, design, development, simulation, modeling,  and nonnuclear testing of nuclear weapons. The three nuclear weapons  design laboratories—Lawrence Livermore National Laboratory (LLNL) in  California, Los Alamos National Laboratory (LANL) in New Mexico, and  Sandia National Laboratories (SNL) in California and New Mexico—use the  results of these activities to annually assess the safety and reliability of the  nation’s nuclear weapons stockpile and to certify to the President that the  resumption of underground nuclear weapons testing is not needed.
When the moratorium began in 1992, DOE (and subsequently NNSA) faced  several challenges in fulfilling its new mission of stockpile stewardship.   For example, since both expected and unexpected changes occur as the  nuclear stockpile ages, NNSA has become more concerned with gaining a  detailed understanding of how such changes might affect the safety and  reliability of stockpiled weapons. However, unlike the rest of a nuclear  weapon, the nuclear explosive package—which contains the primary and  the secondary—cannot be tested simply by evaluating individual  components. Specifically, because the operation of the nuclear explosive  package is highly integrated, nonlinear, occurs during a very short period of  time, and reaches extreme temperatures and pressures, there are portions  of the nuclear explosive package that cannot be tested outside of a nuclear  explosion. In addition, although the United States conducted about 1,000  nuclear weapons tests prior to the moratorium, only a few tests were  designed to collect data on uncertainties associated with a particular part  of the nuclear explosive package. As a result, much of the scientific basis  for the examination of an exploding nuclear weapon must be extrapolated  from other phenomena. Finally, since nuclear testing is no longer available  to train new weapons designers, NNSA and the weapons laboratories are  faced with the need to develop a rigorous, transparent, and explainable  approach to all aspects of the weapon design process, including the  assessment and certification of the performance of nuclear weapons.
To address these challenges, in 1999, DOE established 18 programs—which  it referred to as “campaigns”—six of which were intended to develop the  scientific knowledge, tools, and methods required to provide confidence in  the assessment and certification of the safety and reliability of the nuclear  stockpile in the absence of nuclear testing. These scientific campaigns  include the (1) Primary Assessment Technologies (Primary), (2) Secondary  Assessment Technologies (Secondary), (3) Advanced Simulation and  Computing (ASC), (4) Advanced Radiography, (5) Dynamic Materials  Properties, and (6) Inertial Confinement Fusion and High Yield (ICF)  campaigns. In particular, the Primary and Secondary campaigns are  designed to analyze and understand the different scientific phenomena that  occur in the primary and secondary stages of a nuclear weapon during  detonation. As such, the Primary and Secondary campaigns are intended to  set the requirements for the computer models and experimental data  provided by the other campaigns that are needed to assess and certify the  safety and reliability of nuclear weapons.
While the campaign structure brought increased organization to the  scientific research conducted across the weapons complex, NNSA still  lacked a coherent strategy for relating the scientific research conducted by  the weapons laboratories to the needs of the nuclear stockpile and the  Stockpile Stewardship Program. Consequently, in 2001, LLNL and LANL  began developing what is intended to be a common framework for a new  methodology for assessing and certifying the safety and reliability of  warheads in the nuclear stockpile in the absence of nuclear testing.
The Stockpile Stewardship Program is now over 10 years old, NNSA’s  campaign structure is in its sixth year, and 4 years have passed since LLNL  and LANL began their effort to develop a new assessment and certification  methodology. As the weapons in the nuclear stockpile continue to age, and  as more experienced weapon designers and other scientists and  technicians retire, NNSA is faced with increased urgency in meeting the  goals of the Stockpile Stewardship Program. Furthermore, NNSA has  recently created an effort, known as the Reliable Replacement Warhead  (RRW) program, to study a new approach to maintaining nuclear warheads  over the long term. The RRW program would redesign weapon components  to be easier to manufacture, maintain, dismantle, and certify without  nuclear testing, potentially allowing NNSA to transition to a smaller, more  efficient weapons complex. NNSA’s ability to successfully manage these  efforts will have a dramatic impact on the future of the U.S. nuclear  stockpile and, ultimately, will affect the President’s decision of whether a  return to nuclear testing is required to maintain confidence in the safety  and reliability of the stockpile.
In this context, you asked us to evaluate (1) the new methodology NNSA is  developing for assessing and certifying the safety and reliability of the  nuclear stockpile in the absence of nuclear testing and (2) NNSA’s  management of the implementation of this methodology.
To evaluate the new methodology NNSA is developing for assessing and  certifying the safety and reliability of the nuclear stockpile in the absence  of nuclear testing, we reviewed relevant policy and planning documents  from NNSA and the three weapons laboratories, including implementation  plans and program plans for the six scientific campaigns. We focused our  work principally on the Primary and Secondary campaigns because the  primary and secondary are the key components of the nuclear explosive  package and because the Primary and Secondary campaigns are intended  to set the requirements for the experimental data and computer models  needed to assess and certify the performance of nuclear weapons. We also  reviewed relevant reports, including those from NNSA’s Office of Defense Programs Science Council, the MITRE Corporation’s JASON panel,  University of California review committees for LANL and LLNL, and the  Strategic Advisory Group Stockpile Assessment Team for U.S. Strategic  Command. In addition, we interviewed officials from NNSA headquarters  and site offices, as well as contractors who operate NNSA sites. Our  primary source of information was NNSA’s Office of Defense Programs. We  also met with officials at LANL, LLNL, and SNL. Finally, we interviewed  nuclear weapons experts, senior scientists, and other relevant officials  outside of NNSA and the laboratories, including members of NNSA’s Office  of Defense Programs Science Council, the JASON panel, University of  California review committees for LANL and LLNL, the Strategic Advisory  Group Stockpile Assessment Team for U.S. Strategic Command, and the  Deputy Assistant to the Secretary of Defense (Nuclear Matters) for the  Department of Defense.
To evaluate NNSA’s management of the implementation of its new  methodology to assess and certify the safety and reliability of nuclear  weapons in the absence of nuclear testing, we reviewed relevant NNSA  policy, planning, and evaluation documents, including the Office of Defense  Program’s Program Management Manual, campaign program and  implementation plans, contractor performance evaluation plans and  reports, and internal reviews of NNSA management. We also reviewed  contractor planning and evaluation documents, including LANL, LLNL, and  SNL performance evaluation plans and reports. Finally, we met with  campaign managers and other officials at NNSA headquarters and site  offices, LANL, LLNL, and SNL. We performed our work between August  2004 and December 2005 in accordance with generally accepted  government auditing standards.
subsections: section_title: Background
paragraphs: Most modern nuclear warheads contain a nuclear explosive package,  which contains the primary and the secondary, and a set of nonnuclear  components. The nuclear detonation of the primary produces energy that  drives the secondary, which produces further nuclear energy of a militarily  significant yield. The nonnuclear components control the use, arming, and  firing of the warhead. All nuclear weapons developed to date rely on  nuclear fission to initiate their explosive release of energy. Most also rely  on nuclear fusion to increase their total energy yield. Nuclear fission  occurs when the nucleus of a heavy, unstable atom (such as uranium-235)  is split into two lighter parts, which releases neutrons and produces large  amounts of energy. Nuclear fusion occurs when the nuclei of two light  atoms (such as deuterium and tritium) are joined, or fused, to form a  heavier atom, with an accompanying release of neutrons and larger  amounts of energy.
The U.S. nuclear stockpile consists of nine weapon types. (See table 1.) The  lifetimes of the weapons currently in the stockpile have been extended well  beyond the minimum life for which they were originally  designed—generally about 20 years—increasing the average age of the  stockpile and, for the first time, leaving NNSA with large numbers of  weapons that are close to 30 years old.
Established in 1993, the Stockpile Stewardship Program faces two main  technical challenges: provide (1) a better scientific understanding of the  basic phenomena associated with nuclear weapons and (2) an improved  capability to predict the impact of aging and remanufactured components  on the safety and reliability of nuclear weapons. Specifically,   An exploding nuclear weapon creates the highest pressures, greatest  temperatures, and most extreme densities ever made by man on earth,  within some of the shortest times ever measured. When combined, these  variables exist nowhere else in nature. While the United States  conducted about 1,000 nuclear weapons tests prior to the moratorium,  these tests were conducted mainly to look at broad indicators of  weapon performance (such as the yield of a weapon) and were often not  designed to collect data on specific properties of nuclear weapons  physics. After more than 60 years of developing nuclear weapons, while  many of the physical processes are well understood and accurately  modeled, the United States still does not possess a set of completely  known and expressed laws and equations of nuclear weapons physics  that link the physical event to first principles.
As nuclear weapons age, a number of physical changes can take place.
The effects of aging are not always gradual, and the potential for  unexpected changes in materials causes significant concerns as to  whether weapons will continue to function properly. Replacing aging  components is, therefore, essential to ensure that the weapon will  function as designed. However, it may be difficult or impossible to  ensure that all specifications for the manufacturing of new components  are precisely met, especially since each weapon was essentially  handmade. In addition, some of the manufacturing process lines used  for the original production have been disassembled.
In 1995, the President established an annual assessment and reporting  requirement designed to help ensure that nuclear weapons remain safe and  reliable without underground testing. As part of this requirement, the three  weapons laboratories are required to issue a series of reports and letters  that address the safety, reliability, performance, and military effectiveness  of each weapon type in the stockpile. The letters, submitted to the  Secretary of Energy individually by the laboratory directors, summarize the  results of the assessment reports and, among other things, express the  directors’ conclusions regarding whether an underground nuclear test is  needed and the adequacy of various tools and methods currently in use to  evaluate the stockpile.
To address these challenges, in 1999 DOE developed a new three-part  program structure for the Stockpile Stewardship Program that included a  series of campaigns, which DOE defined as technically challenging,  multiyear, multifunctional efforts to develop and maintain the critical  capabilities needed to continue assessing the safety and reliability of the  nuclear stockpile into the foreseeable future without underground testing.  DOE originally created 18 campaigns that were designed to focus its efforts  in science and computing, applied science and engineering, and production  readiness. Six of these campaigns currently focus on the development and  improvement of the scientific knowledge, tools, and methods required to  provide confidence in the assessment and certification of the safety and  reliability of the nuclear stockpile in the absence of nuclear testing. These  six campaigns are as follows:  The Primary and Secondary campaigns were established to analyze and  understand the different scientific phenomena that occur in the primary  and secondary stages of a nuclear weapon during detonation. As such,  the Primary and Secondary campaigns are intended to support the  development and implementation of the QMU methodology and to set  the requirements for the computers, computer models, and  experimental data needed to assess and certify the performance of  nuclear weapons.
The ASC campaign provides the leading-edge supercomputers and  models that are used to simulate the detonation and performance of  nuclear weapons.
Two campaigns—Advanced Radiography and Dynamic Materials  Properties—provide data from laboratory experiments to support  nuclear weapons theory and computational modeling. For example, the  Advanced Radiography campaign conducts experiments that measure  how stockpile materials behave when exposed to explosively driven  shocks. One of the major facilities being built to support this campaign  is the Dual Axis Radiographic Hydrodynamic Test Facility at LANL.
The ICF campaign develops experimental capabilities and conducts  experiments to examine phenomena at high temperature and pressure  regimes that approach but do not equal those occurring in a nuclear  weapon. As a result, scientists currently have to extrapolate from the  results of these experiments to understand similar phenomena in a  nuclear weapon. One of the major facilities being built as part of this  campaign is the National Ignition Facility at LLNL.
The other two program activities associated with the Stockpile  Stewardship Program are “Directed Stockpile Work” and “Readiness in  Technical Base and Facilities.” Directed Stockpile Work includes the  activities that directly support specific weapons in the stockpile, such as  the Stockpile Life Extension Program, which employs a standardized  approach for planning and carrying out nuclear weapons refurbishment  activities to extend the operational lives of the weapons in the stockpile  well beyond their original design lives. The life extension for the W87 was  completed in 2004, and three other weapon systems—the B61, W76, and  W80—are currently undergoing life extensions. Each life extension  program is specific to that weapon type, with different parts being replaced  or refurbished for each weapon type. Readiness in Technical Base and  Facilities includes the physical infrastructure and operational readiness  required to conduct campaign and Directed Stockpile Work activities  across the nuclear weapons complex. The complex includes the three  nuclear weapons design laboratories (LANL, LLNL, and SNL), the Nevada  Test Site, and four production plants—the Pantex Plant in Texas, the Y-12  Plant in Tennessee, a portion of the Savannah River Site in South Carolina,  and the Kansas City Plant in Missouri.
From fiscal year 2001 through fiscal year 2005, NNSA spent over $7 billion  on the six scientific campaigns (in inflation-adjusted dollars). (See table 2.)  NNSA has requested almost $7 billion in funding for these campaigns over  the next 5 years. (See table 3.)
Within NNSA, the Office of Defense Programs is responsible for managing  the campaigns and the Stockpile Stewardship Program in general. Within  this office, two organizations share responsibility for overall management  of the scientific campaigns: the Office of the Assistant Deputy  Administrator for Research, Development, and Simulation and the Office of  the Assistant Deputy Administrator for Inertial Confinement Fusion and  the National Ignition Facility Project. The first office oversees campaign  activities associated with the Primary and Secondary campaigns—as well  as the ASC, Advanced Radiography, and Dynamic Materials Properties  campaigns—with a staff of about 13 people. The second office oversees  activities associated with the ICF campaign with a single staff person.  Actual campaign activities are conducted by scientists and other staff at the  three weapons laboratories. LANL and LLNL conduct activities associated  with the nuclear explosive package, while SNL performs activities  associated with the nonnuclear components that control the use, arming,  and firing of the nuclear warhead.
section_title: The QMU Methodology Is Highly Promising but Still in the Early Stages of Development
paragraphs: NNSA has endorsed the use of a new common methodology, known as the  quantification of margins and uncertainties, or QMU, for assessing and  certifying the safety and reliability of the nuclear stockpile. NNSA and  laboratory officials told us that they have made progress in applying the  principles of QMU to the certification and assessment of nuclear warheads  in the stockpile. However, QMU is still in its early stages of development,  and important differences exist among the three laboratories in their  application of QMU. To date, NNSA has commissioned two technical  reviews of the implementation of QMU at the weapons laboratories. While  strongly supporting QMU, the reviews found that the development and  implementation of QMU was still in its early stages. The reviews  recommended that NNSA take steps to further define the technical details  supporting the implementation of QMU and integrate the activities of the  three weapons laboratories in implementing QMU. However, NNSA and the  weapons laboratories have not fully implemented these recommendations.  Beyond the issues raised in the two reports, we also found differences in  the understanding and application of QMU among the three laboratories.
subsections: section_title: NNSA Has Endorsed QMU as a New, Common Methodology for Assessing and Certifying Stockpile Safety and Reliability
paragraphs: When the Primary and Secondary campaigns were established in 1999, they  brought some organization and overall goals to the scientific research  conducted across the weapons complex. For example, as we noted in April  2005, the Primary campaign set an initial goal in the 2005 to 2010 time  frame for certifying the performance of the primary of a nuclear weapon to  within a stated yield level. However, according to senior NNSA officials,  NNSA still lacked a coherent strategy for relating the scientific work  conducted by the weapons laboratories under the campaigns to the needs  of the nuclear stockpile and the overall Stockpile Stewardship Program.  This view was echoed by a NNSA advisory committee report, which stated  in 2002 that the process used by the weapons laboratories to certify the  safety and reliability of nuclear weapons was ill defined and unevenly  applied, leading to major delays and inefficiencies in programs.
Starting in 2001, LLNL and LANL began developing what is intended to be a  common methodology for assessing and certifying the performance and  safety of nuclear weapons in the absence of nuclear testing. In 2003, the  associate directors for nuclear weapons at LLNL and LANL published a  white paper—entitled “National Certification Methodology for the Nuclear  Weapon Stockpile”—that described this new methodology, which they  referred to as the quantification of margins and uncertainties or QMU.  According to the white paper, QMU is based on an adaptation of standard  engineering practices and lends itself to the development of “rigorous,  quantitative, and explicit criteria for judging the robustness of weapon  system and component performance at a detailed level.” Moreover, the  quantitative results of this process would enable NNSA and the weapons  laboratories to set priorities for their activities and thereby make rational  decisions about allocating program resources to the nuclear stockpile.
The process envisaged in the white paper focuses on creating a “watch list”  of factors that, in the judgment of nuclear weapons experts, are the most  critical to the operation and performance of a nuclear weapon. These  factors include key operating characteristics and components of the  nuclear weapon. For each identified, critical factor leading to a nuclear  explosion, nuclear weapons experts would define performance metrics.  These performance metrics would represent the experts’ best judgment of  what constitutes acceptable behavior—i.e., the range of acceptable values  for a critical function to successfully occur or for a critical component to  function properly—as well as what constitutes unacceptable behavior or  failure. To use an analogy, consider the operation of a gasoline engine.  Some of the events critical to the operation of the engine would include the  opening and closing of valves, the firing of the spark plugs, and the ignition  of the fuel in each cylinder. Relevant performance metrics for the ignition  of fuel in a cylinder would include information on the condition of the  spark plugs (e.g., whether they are corroded) and the fuel/air mixture in the  cylinder.
Once nuclear experts have identified the relevant performance metrics for  each critical factor, according to the 2003 white paper, the goal of QMU is  to quantify these metrics. Specifically, the QMU methodology seeks to  quantify (1) how close each critical factor is to the point at which it would  fail to perform as designed (i.e., the performance margin or the margin to  failure) and (2) the uncertainty in calculating the margin. According to the  white paper, the weapons laboratories would be able to use their calculated  values of margins and uncertainties as a way to assess their confidence in  the performance of a nuclear weapon. That is, the laboratories would  establish a “confidence ratio” for each critical factor —they would divide  their calculated value for the margin (“M”) by their calculations of the  associated uncertainty (“U”) and arrive at a single number (“M/U”).  According to the white paper, the weapons laboratories would only have  confidence in the performance of a nuclear weapon if the margin  “significantly” exceeds uncertainty for all critical issues. However, the  white paper did not define what the term “significantly” meant.
In a broad range of key planning and management documents that have  followed the issuance of the white paper, NNSA and the weapons  laboratories have endorsed the use of the QMU methodology as the  principal tool for assessing and certifying the safety and reliability of the  nuclear stockpile in the absence of nuclear testing. For example, in its  fiscal year 2006 implementation plan for the Primary campaign, NNSA  stated as a strategic objective that it needs to develop the capabilities and  understanding necessary to apply QMU as the assessment and certification  methodology for the nuclear explosive package. In addition, in its fiscal  year 2006 budget request, NNSA selected its progress toward the  development and implementation of QMU as one of its major performance  indicators. Finally, in the plans that NNSA uses to evaluate the performance  of LANL and LLNL, NNSA has established an overall objective for LANL  and LLNL to assess and certify the safety and reliability of nuclear weapons  using a common QMU methodology.
Officials at NNSA and the weapons laboratories have also stated that QMU  will be vital to certifying any weapon redesigns, such as are envisioned by  the RRW program. For example, senior NNSA officials told us that the  Stockpile Stewardship Program will not be sustainable if it only involves  the continued refurbishment in perpetuity of existing weapons in the  current nuclear stockpile. They stated that the accumulation of small  changes over the extended lifetime of the current nuclear stockpile will  result in increasing levels of uncertainty about its performance. If NNSA  moves forward with the RRW program, according to NNSA documents and  officials, the future goal of the weapons program will be to use QMU to  replace existing stockpile weapons with an RRW whose safety and  reliability could be assured with the highest confidence, without nuclear  testing, for as long as the United States requires nuclear forces.
section_title: The Development and Implementation of QMU Is at an Early Stage and Important Differences Exist Among the Weapons Laboratories in their Application of QMU
paragraphs: According to NNSA and laboratory officials, the weapons laboratories have  made progress in applying the principles of QMU to the certification of life  extension programs and to the annual stockpile assessment process. For  example, LLNL officials told us that they are applying QMU to the  assessment of the W80, which is currently undergoing a life extension.  They said that, in applying the QMU methodology, they tend to focus their  efforts on identifying credible “failure modes,” which are based on  observable problems, such as might be caused by the redesign of  components in a nuclear weapon, changes to the manufacturing process  for components, or the performance of a nuclear weapon under aged  conditions. They said that, for the W80 life extension program, they have  developed a list of failure modes and quantified the margins and  uncertainties associated with these failure modes. Based on their  calculations, they said that they have increased their confidence in the  performance of the W80.
Similarly, LANL officials told us that they are applying QMU to the W76,  which is also currently undergoing a life extension and is scheduled to  finish its first production unit in 2007. They said that, in applying the QMU  methodology, they tend to focus their efforts on defining “performance  gates,” which are based on a number of critical points during the explosion  of a nuclear weapon that separate the nuclear explosion into natural stages  of operation. The performance gates identify the characteristics that a  nuclear weapon must have at a particular time during its operation to meet  its performance requirements (e.g., to reach its expected yield). LANL  officials told us that they have developed a list of performance gates for the  W76 life extension program and are beginning to quantify the margins and  uncertainties associated with these performance gates.
Despite this progress, we found that QMU is still in its early stages of  development and that important differences exist among the weapons  laboratories in their application of QMU. To date, NNSA has commissioned  two technical reviews of the implementation of QMU at the weapons  laboratories. The first review was conducted by NNSA’s Office of Defense  Programs Science Council (Science Council)—which advises NNSA on  scientific matters across a range of activities, including those associated  with the scientific campaigns—and resulted in a March 2004 report. The  second review was conducted by the MITRE Corporation’s JASON panel  and resulted in a February 2005 report. Both reports endorsed the use of  QMU by the weapons laboratories and listed several potential benefits that  QMU could bring to the nuclear weapons program. For example, according  to the Science Council report, QMU will serve an important role in training  the next generation of nuclear weapon designers and will quantify and  increase NNSA’s confidence in the assessment and certification of the  nuclear stockpile. According to the JASON report, QMU could become a  useful management tool for directing investments in a given weapon  system where they would be most effective in increasing confidence, as  required by the life extension programs. In addition, the JASON report  described how LANL and LLNL officials had identified potential failure  modes in several weapon systems and calculated the associated margins  and uncertainties. The report noted that, for most of these failure modes,  the margin for success was large compared with the uncertainty in the  performance.
However, according to both the Science Council and the JASON reports,  the development and implementation of QMU is still in its early stages. For  example, the JASON report described QMU as highly promising but  unfinished, incomplete and evolving, and in the early stages of  development. Moreover, the chair of the JASON panel on QMU told us in  June 2005 that, during the course of his review, members of the JASON  panel found that QMU was not mature enough to assess its reliability or  usefulness. The reports also stated that the weapons laboratories have not  fully developed or agreed upon the technical details supporting the  implementation and application of QMU. For example, the JASON report  stated that, in the course of its review, it became evident that there were a  variety of differing and sometimes diverging reviews of what QMU really  was and how it was working in practice. As an example, the report stated  that some of the scientists, designers, and engineers at LANL and LLNL saw  the role of expert judgment as an integral part of the QMU process, while  others did not. In discussions with the weapons laboratories about the two  reports, LANL officials told us that they believed that the details of QMU as  a formal methodology are still evolving, while LLNL officials stated that  QMU was “embryonic” and not fully developed.
While supporting QMU, the two reports noted that the weapons  laboratories face challenges in successfully implementing a coherent and  credible analytical method based on the QMU methodology. For example,  in its 2004 report, the Science Council stated that, in its view, the QMU  methodology is based on the following core assumptions:  Computer simulations can accurately predict the behavior of a complex  nuclear explosive system as a function of time.
It is sufficient for the assessment of the performance of a nuclear  weapon to examine the simulation of the time evolution of a nuclear  explosive system at a number of discrete time intervals and to  determine whether the behavior of the system at each interval is within  acceptable bounds.
The laboratories’ determinations of acceptable behavior can be made  quantitatively—that is, they will make a quantitative estimate of a  system’s margins and uncertainties.
Given these quantitative measures of the margins and uncertainties, it is  possible to calculate the probability (or confidence level) that the  nuclear explosive system will perform as desired.
However, the Science Council’s report noted that extraordinary degrees of  complexity are involved in a rational implementation of QMU that are only  beginning to be understood. For example, in order for the QMU  methodology to have validity, it must sufficiently identify all critical failure  modes, critical events, and associated performance metrics. However, as  described earlier, the operation of an exploding nuclear weapon is highly  integrated and nonlinear, occurs during a very short period of time, and  reaches extreme temperatures and pressures. In addition, the United States  does not possess a set of completely known and expressed laws and  equations of nuclear weapons physics. Given these complexities, it will be  difficult to demonstrate the successful implementation of QMU, according  to the report. In addition, the Science Council stated that it was not  presented with any evidence that there exists a method—even in  principle—for calculating an overall probability that a nuclear explosive  package will perform as designed from the set of quantitative margins and  uncertainties at each time interval.
To address these and other issues, the two reports recommended that  NNSA take steps to further define the technical details supporting the  implementation of QMU and to integrate the activities of the three weapons  laboratories in implementing QMU. For example, the 2004 Science Council  report recommended that NNSA direct the associate directors for nuclear  weapons at LANL and LLNL to undertake a major effort to define the  details of QMU. In particular, the report recommended that a trilaboratory  team be charged with defining a common language for QMU and identifying  the important performance gates, failure modes, and other criteria in the  QMU approach. The report stated that this agreed-upon “reference” set  could then be used to support all analyses of stockpile issues. In addition,  the report recommended that NNSA consider establishing annual or  semiannual workshops for the three weapons laboratories to improve the  identification, study, and prioritization of potential failure modes and other  factors that are critical to the operation and performance of nuclear  weapons.
Similarly, the 2005 JASON panel report noted that the meaning and  implications of QMU are currently unclear. To rectify this problem, the  report recommended that the associate directors for nuclear weapons at  LANL and LLNL write a new, and authoritative, paper defining QMU and  submit it to NNSA. Furthermore, the report recommended that the  laboratories establish a formal process to (1) identify all failure modes and  performance gates associated with QMU, using the same methodology for  all weapon systems, and (2) establish better relationships between the  concepts of failure modes and performance gates for all weapon systems in  the stockpile.
However, NNSA and laboratory officials have not fully implemented these  recommendations, particularly the recommendations of the Science  Council. For example, while LLNL and LANL officials are drafting a new  “white paper” on QMU that attempts to clarify some fundamental tenets of  the methodology, officials from SNL are not involved in the drafting of this  paper. In addition, NNSA has not required the three weapons laboratories  to hold regular meetings or workshops to improve the identification,  prioritization, and integration of failure modes, performance gates, and  other critical factors.
According to NNSA’s Assistant Deputy Administrator for Research,  Development, and Simulation, NNSA has not fully implemented the  recommendations of the Science Council’s report partly because the report  was intended more to give NNSA a sense of the status of the  implementation of QMU than it was to provide recommendations. For  example, the 2004 report states that the “friendly review,” as the report is  referred to by NNSA, would not have budget implications and that the  report’s findings and recommendations would be reported only to the  senior management of the weapons laboratories. As a result, the Assistant  Deputy Administrator told us that he had referred the recommendations to  the directors of the weapons laboratories and told them to implement the  recommendations as they saw fit.
Furthermore, LLNL and LANL officials disagreed with some of the  statements in the Science Council report and stressed that, in using QMU,  they do not attempt to assign an overall probability that the nuclear  explosive package will perform as desired. That is, they do not attempt to  add up calculations of margins and uncertainties for all the critical factors  to arrive at a single estimate of margin and uncertainty, or a single  confidence ratio, for the entire nuclear explosive package. Instead, they  said that they focus on ensuring that the margin for each identified critical  factor in the explosion of a nuclear weapon is greater than the uncertainty.  However, they said that, for a given critical factor, they do combine various  calculations of individual uncertainties that contribute to the total amount  of uncertainty for that factor.
In addition, in addressing comments in the JASON report, LLNL and LANL  officials stressed that QMU has always relied, and will continue to rely  heavily, on the judgment of nuclear weapons experts. For example, LLNL  officials told us that since there is no single definition of what constitutes a  threshold for failure, they use expert judgment to decide what to put on  their list of failure modes. They also said that the QMU methodology  provides a way to make the entire annual assessment and certification  process more transparent to peer review. Similarly, LANL officials said that  they use expert judgment extensively in establishing performance metrics  and threshold values for their performance gates. They said that expert  judgment will always be a part of the scientific process and a part of QMU.
Beyond the issues raised in the two reports, we found that there are  differences in the understanding and application of QMU among the three  laboratories. For example, the three laboratories do not agree about the  application of QMU to areas outside of the nuclear explosive package.  Specifically, LLNL officials told us that the QMU methodology, as currently  developed, only applies to the nuclear explosive package and not to the  nonnuclear components that control the use, arming, and firing of the  nuclear warhead. According to LLNL and LANL officials, SNL scientists can  run hundreds of experiments to test their components and, therefore, can  use normal statistical analysis in certifying the performance of nonnuclear  components. As a result, according to LLNL and LANL officials, SNL does  not have to cope with real uncertainty and does not “do” QMU.  Furthermore, according to LLNL officials, SNL has chosen not to  participate in the development of QMU with LLNL and LANL.
However, SNL officials told us that while some of the nonnuclear  components are testable to a degree, SNL is as challenged as the other two  weapons laboratories in certifying the performance of their systems  without actual testing. For example, SNL officials said that they simply do  not have enough money to perform enough tests on all of their nonnuclear  components to be able to rely completely on statistical analysis to meet  their safety performance levels. In addition, SNL scientists are not able to  test their components under the conditions of a nuclear explosion but are  still required to certify the performance of the components under these  conditions. Thus, SNL officials told us that they had been using their own  version of QMU for a long time.
SNL officials told us that they define QMU as a way to make risk-informed  decisions about the effect of variabilities and uncertainties on the  performance of a nuclear weapon, including the nonnuclear components  that control the use, arming, and firing of the nuclear warhead. Moreover,  they said that this kind of risk-informed approach is not unique to the  nuclear weapons laboratories and is used extensively in areas such as  nuclear reactor safety. However, they told us that they have been left out in  the development of QMU by the two other weapons laboratories.  Specifically, they said that while SNL scientists have worked with other  scientists at LANL and LLNL at a “grass roots” level, there has only been  limited cooperation and dialogue between upper-level management at the  three laboratories concerning the development and implementation of  QMU.
In addition, we found that while LLNL and LANL both agree on the  fundamental tenets of QMU at a high level, their application of the QMU  methodology differs in some important respects. For example, LLNL and  LANL officials told us that, at a detailed level, the two laboratories are  pursuing different approaches to calculating and combining uncertainties.  For the W80 life extension program, LLNL officials showed us how they  combined calculations of individual uncertainties that contributed to the  total uncertainty for a key failure mode of the primary—the amount of  primary yield necessary to drive the secondary. However, they said that the  scientific support for their method for combining individual calculations of  uncertainty was limited, and they stated that they are pursuing a variety of  more sophisticated analyses to improve their current approach.
Moreover, the two laboratories are taking a different approach to  generating a confidence ratio for each critical factor, as described in the  2003 white paper on QMU. For example, for the W80 life extension  program, LLNL officials showed us how they calculated a single confidence  ratio for a key failure mode of the primary, based on their calculations of  margin and uncertainty. They said that the weapon systems for which they  are responsible have a lot of margin built into them, and they feel  comfortable generating this number. In contrast, in discussions with LANL  officials about the W76 life extension program, LANL officials told us that  they prefer not to calculate a single confidence ratio for a performance  gate, partly because they are concerned that their customers (e.g., the  Department of Defense) might think that the QMU methodology is more  formal than it is currently.
In commenting on the differences between the two laboratories, NNSA  officials stated that the two laboratories are pursuing complementary  approaches, and that these differences are part of the rationale for a  national policy decision to maintain two nuclear design laboratories. In  addition, they stated that the confidence in the correctness of scientific  research is improved by achieving the same answer through multiple  approaches. LLNL officials also made similar comments, stating that the  nation will benefit from some amount of independence between the  laboratories to assure that the best methodology for assessing the stockpile  in the absence of nuclear testing is achieved.
section_title: NNSA’s Management of the Development and Implementation of QMU Is Deficient in Four Key Areas
paragraphs: NNSA relies on its Primary and Secondary campaigns to manage the  development and implementation of QMU. According to NNSA policies,  campaign managers at NNSA headquarters are responsible for developing  campaign plans and high-level milestones, overseeing the execution of  these plans, and providing input to the evaluation of the performance of the  weapons laboratories. However, NNSA’s management of these processes is  deficient in four key areas. First, the planning documents that NNSA has  established for the Primary and Secondary campaigns do not adequately  integrate the scientific research currently conducted that supports the  development and implementation of QMU. Second, NNSA has not  developed a clear, consistent set of milestones to guide the development  and implementation of QMU. Third, NNSA has not established formal  requirements for conducting annual, technical reviews of the  implementation of QMU or for certifying the completion of QMU-related  milestones. Finally, NNSA has not established adequate performance  measures to determine the progress of the laboratories in developing and  implementing QMU.
subsections: section_title: Campaign Planning Documents Do Not Adequately Integrate the Scientific Activities Supporting QMU
paragraphs: As part of its planning structure, NNSA requires the use of program and  implementation plans to set requirements and manage resources for the  campaigns and other programs associated with the Stockpile Stewardship  Program. Program plans are strategic in nature and identify the long-term  goals, high-level milestones, and resources needed to support a particular  program over a 7-year period, while implementation plans establish  performance expectations for the program and each participating site for  the current year of execution. According to NNSA policies, program and  implementation plans should flow from and interact with each other using  a set of cascading goals and requirements.
NNSA has established a single program plan, which it calls the “Science  campaign program plan,” that encompasses the Primary and the Secondary  campaigns, as well as two other campaigns—Advanced Radiography and  Dynamic Materials Properties. NNSA has also established separate  implementation plans for each of these campaigns, including the Primary  and Secondary campaigns. According to NNSA, it relies on these  plans—and in particular the plans related to the Primary and Secondary  campaigns—to manage the development and implementation of QMU, as  well as to determine the requirements for the experimental data and  computer modeling needed to analyze and understand the different  scientific phenomena that occur in a nuclear weapon during detonation.
However, the current Primary and Secondary campaign plans do not  contain a comprehensive, integrated list of the relevant scientific research  being conducted across the weapons complex to support the development  and implementation of QMU. For example, according to the NNSA  campaign manager for the Primary campaign, he had to hold a workshop in  2005 with officials from the weapons laboratories in order to catalogue all  of the scientific activities that are currently performed under the heading of  “primary assessment” regardless of the NNSA funding source. According to  this official, the existing Primary campaign implementation plan does not  provide the integration across NNSA programs that is needed to achieve  the goals of the Primary campaign and to develop and implement QMU.
According to NNSA officials, the lack of integration has occurred in large  part because a significant portion of the scientific research that is relevant  to the Primary and Secondary campaigns is funded and carried out by  different campaigns and other programs. Specifically, different NNSA  campaign managers use different campaign planning documents to plan  and oversee research and funding for activities that are directly relevant to  the Primary and Secondary campaigns and the development and  implementation of QMU. For example, the ASC campaign provides the  supercomputing capability that the weapons laboratories use to simulate  and predict the behavior of an exploding nuclear weapon. Moreover, the  weapons laboratories rely on ASC supercomputers to quantify their  uncertainties with respect to the accuracy of these computer  simulations—a key component in the implementation of QMU. As a result,  the ASC campaign plans and funds activities that are critical to the  development and implementation of QMU.
To address this problem, according to NNSA officials, NNSA is taking steps  to establish better relationships among the campaign plans. For example,  NNSA is currently drafting a new plan—which it calls the Primary  Assessment Plan—in an attempt to better coordinate the activities covered  under the separate program and implementation plans. The draft plan  outlines high-level research priorities, time lines, and proposed milestones  necessary to support (1) NNSA’s responsibilities for the current stockpile,  (2) primary physics design for the development of an RRW, and (3)  certification of an RRW in the 2012 time frame and a second RRW in the  2018 time frame. According to NNSA officials, they expect to finalize this  plan by the third quarter of fiscal year 2006. In addition, they expect to have  a similar plan for the Secondary campaign finalized by December 2006 and  are considering combining both plans into a full-system assessment plan.  According to one NNSA official responsible for the Primary and Secondary  campaigns, NNSA will revise the existing campaign program and  implementation plans to be consistent with the Primary Assessment Plan.
More fundamentally, some nuclear weapons experts have suggested that  NNSA’s planning structure should be reorganized to better reflect the use of  QMU as NNSA’s main strategy for assessing and certifying the performance  of nuclear weapons. For example, the chair of the LLNL Defense and  Nuclear Technologies Director’s Review Committee—which conducts  technical reviews of LLNL’s nuclear weapons activities for the University of  California—told us that the current campaign structure has become a  series of “stovepipes” that NNSA uses to manage stockpile stewardship. He  said that in order for NNSA to realize its long-term goals for implementing  QMU, NNSA is going to have to reorganize itself around something that he  called an “uncertainty spreadsheet” for each element of a weapon’s  performance (e.g., implosion of the primary, transfer of energy to the  secondary, etc.), leading to the weapon’s yield. He said that the laboratories  should develop a spreadsheet for each weapon in the stockpile that (1)  identifies the major sources of uncertainty at each critical event in their  assessment of the weapon’s performance and (2) relates the laboratory’s  scientific activities and milestones to these identified sources of  uncertainty. He said that the development and use of these spreadsheets  would essentially capture the intent of the scientific campaigns and make  them unnecessary.
section_title: NNSA Does Not Have a Clear, Consistent Set of QMU-Related Milestones
paragraphs: NNSA has established a number of milestones that relate to the  development and implementation of QMU. Within the Science campaign  program plan, NNSA has established a series of high-level milestones,  which it calls “level-1” milestones. According to NNSA policies, level-1  milestones should be sufficient enough to allow strategic integration  between sites involved in the campaigns and between programs in NNSA.  Within the implementation plans for the Primary and Secondary  campaigns, NNSA has established a number of lower-level milestones,  which it calls “level-2” milestones, which NNSA campaign managers use to  track major activities for the current year of execution. The level-1  milestones related to QMU are shown in table 4, and the level-2 milestones  related to QMU for the Primary campaign are shown in table 5.
According to NNSA officials, the level-1 milestones in table 4 represent a  two-stage path to systematically identify uncertainties and reduce them  through analyzing past underground test results, developing new  experimental capabilities, and performing new experiments to understand  the relevant physical processes. According to these level-1 milestones,  NNSA expects to complete the second stage or “cycle” of this process by  fiscal year 2014 (i.e., milestone M20), at which time NNSA will have  sufficiently reduced major sources of uncertainties and will have  confidence in its ability to predict the performance of nuclear weapons in  the absence of nuclear testing.
However, we identified several problems with the NNSA milestones related  to the development and implementation of QMU. Specifically, the level-1  milestones in the Science campaign program plan have the following  problems:  The milestones are not well-defined and never explicitly mention QMU.
According to NNSA officials responsible for overseeing the Primary  campaign, these milestones are too qualitative and too far in the future  to enable NNSA to effectively plan for and oversee the implementation  of QMU. They described these milestones as “fuzzy” and said that they  need to be better defined. However, NNSA officials also stated that  these milestones are not just for QMU but for the entire Science  campaign, of which QMU is only a part.
The milestones conflict with the performance measures shown in other  important NNSA management documents. Specifically, while the  Science campaign program plan envisions a two-stage path to identify  and reduce key uncertainties related to nuclear weapon operations  using QMU by 2014, the performance measures in NNSA’s fiscal year  2006 budget request and in Appendix A of the Science campaign  program plan call for the completion of QMU by 2010.
The milestones have not been integrated with other QMU-related level-1  milestones in other planning documents. For example, the current ASC  campaign program plan contains a series of level-1 milestones for  completing the certification of several weapon systems—including the  B61, W80, W76, and W88—with quantified margins and uncertainties by  the end of fiscal year 2007. However, these milestones do not appear in  and are not referenced by the Science campaign program plan.  Moreover, the ASC campaign manager told us that, until recently, he was  not aware of the existence of the level-1 milestones for implementing  QMU that are contained in the Science campaign program plan.
In addition, we found that neither the Science campaign program plan nor  the Primary campaign implementation plan describe how the level-2  milestones on QMU in the Primary campaign implementation plan are  related to the level-1 milestones on QMU in the Science campaign program  plan. Consequently, it is unclear how the achievement of specific level-2  milestones—such as the development of probabilistic tools and methods to  combine various sources of uncertainty for primary performance—will  result in the achievement of level-1 milestones for the implementation of  QMU or how NNSA expects to certify several major nuclear weapon  systems using QMU before the QMU methodology is fully developed and  implemented.
NNSA, as well as laboratory officials, agreed that there are weaknesses  with the current QMU milestones. According to NNSA officials, when  NNSA established the current tiered structure for campaign milestones in  2003, the different tiers of milestones served different purposes and,  therefore, were never well-integrated. For example, NNSA officials said  that the level-1 milestones were originally created to reflect measures that  were deemed to be important to senior NNSA officials, while level-2  milestones were created to be used by NNSA campaign managers to  perform more technical oversight of the weapons laboratories.  Furthermore, according to NNSA officials, the current level-2 milestones  are only representative of campaign activities conducted by the weapons  laboratories. That is, the level-2 milestones were never designed to cover  the entire scope of work being conducted by the weapons laboratories and  are, therefore, not comprehensive in scope.
To address these problems, according to NNSA officials, NNSA is taking  steps to develop better milestones to track the implementation of the QMU  methodology. For example, in the draft Primary Assessment Plan, NNSA  has established 19 “high-level” milestones that cover the time period from  fiscal year 2006 to fiscal year 2018. According to these draft milestones, by  fiscal year 2010, NNSA expects to “complete the experimental work and  methodology development needed to demonstrate the ability of primary  certification tools to support certification of existing stockpile system and  RRW.” In addition, NNSA expects to certify a RRW in fiscal year 2012 and a  second RRW in fiscal year 2018.
section_title: NNSA Has Not Established Formal Requirements for Conducting Technical Reviews or Certifying the Completion of QMU-Related Milestones
paragraphs: According to NNSA policies, campaign managers are required to track the  status of level-1 and level-2 milestones and provide routine, formal reports  on the status of their programs. For example, campaign managers are  required to track, modify, and score the status of level-1 and level-2  milestones through the use of an Internet-based application called the  Milestone Reporting Tool. On a quarterly basis, campaign managers assign  one of four possible scores for each milestone listed in the application: (1)  “blue” for completed milestones, (2) “green” for milestones that are on  track to be finished by the end of the fiscal year, (3) “yellow” for milestones  that may not be completed by the end of the fiscal year, and (4) “red” for  milestones that will not be completed by the end of the fiscal year. At  quarterly program review meetings, campaign managers brief senior-level  NNSA officials on the status of major milestones, along with cost and  expenditure data for their programs. In addition, campaign managers are  responsible for conducting technical reviews of the campaigns for which  they are responsible, at least annually, to ensure that campaign activities  are being executed properly and that campaign milestones are being  completed.
However, NNSA campaign managers have not met all of the NNSA  requirements needed to effectively oversee the Primary and Secondary  campaigns. For example, we found that the campaign managers for the  Primary and Secondary campaigns have not established formal  requirements for conducting annual, technical reviews of the  implementation of QMU at the three weapons laboratories. Moreover, these  officials have not established requirements for certifying the completion of  level-2 milestones that relate to QMU. They could not provide us with  documentation showing the specific activities or outcomes that they  expected from the weapons laboratories in order to certify that the  laboratories had completed the level-2 milestones for QMU. Instead, they  relied more on ad hoc reviews of campaign activities and level-2 milestones  as part of their oversight activities for their campaigns. According to the  Primary campaign manager, the officials at the weapons laboratories are  the principal managers of campaign activities. As a result, he views his role  as more of a “sponsor” for his program and, therefore, does not require any  written reports or evidence from the laboratories to certify that they have  completed specific milestones.
In contrast, we found that the ASC campaign manager has established  formal requirements for a variety of reoccurring technical reviews of  activities associated with the ASC campaign. Specifically, the ASC  campaign relies on semiannual reviews conducted by the ASC Predictive  Science Committee—which provides an independent, technical review of  the status of level-2 milestones—as well as on annual “principal  investigators” meetings that provide a technical review of every program  element within the ASC campaign. The ASC campaign manager told us that  he relies on these technical reviews to oversee program activities because  the quarterly program review meetings are not meant to help him manage  his program but are really a way for senior-level NNSA officials to stay  informed.
In addition, the ASC campaign manager has established detailed, formal  requirements for certifying the completion of level-2 milestones for the  ASC campaign. Specifically, the fiscal year 2006 implementation plan for  the ASC campaign contains a detailed description of what NNSA expects  from the completion of each level-2 milestone, including a description of  completion criteria, the method by which NNSA will certify the completion  of the milestone, and an assessment of the risk level associated with the  completion of the milestone. The ASC campaign manager told us that,  when NNSA officials created the level-2 milestones for the campaigns in  2003, the milestones were really just “sentences” and lacked the detailed  criteria that would enable NNSA managers to adequately track and  document the completion of major milestones. As a result, the ASC  campaign has made a major effort in recent years to develop detailed,  formal requirements to support the completion of ASC level-2 milestones.
section_title: NNSA Has Not Established Adequate Measures to Determine the Laboratories’ Performance in Developing and Implementing QMU
paragraphs: NNSA uses performance measurement data to inform resource decisions,  improve the management and delivery of products and services, and justify  budget requests. According to NNSA requirements, performance  measurement data should explain in clear, concise, meaningful, and  measurable terms what program officials expect to accomplish for a  specific funding level over a fixed period of time. In addition, performance  measurement data should include annual targets that describe specific  outputs that can be measured, audited, and substantiated by the detailed  technical milestones contained in documentation such as campaign  implementation plans.
With respect to QMU, NNSA has established an overall annual performance  target to measure the cumulative percentage of progress toward the  development and implementation of the QMU methodology. Specifically, in  its fiscal year 2006 budget request to the Congress, NNSA stated that it  expects to complete the development and implementation of QMU by 2010  as follows:   25 percent complete by the end of fiscal year 2005,  40 percent complete by the end of fiscal year 2006,  55 percent complete by the end of fiscal year 2007,  70 percent complete by the end of fiscal year 2008,   85 percent complete by the end of fiscal year 2009, and  100 percent complete by the end of fiscal year 2010.
According to NNSA, it had progressed 10 percent toward its target of  completing QMU by the end of fiscal year 2004. However, NNSA officials  could not document how they can measure progress toward the  performance target for developing and implementing QMU. Moreover,  NNSA officials could not explain how the 2010 overall performance target  for the completion and implementation of QMU is related to the level-1  milestones for QMU in the Science campaign program plan, which  describes a two-stage process to identify and reduce key uncertainties in  nuclear weapon performance using QMU by 2014. According to one NNSA  official responsible for overseeing the Primary campaign, NNSA created  this annual performance target because the Office of Management and  Budget requires agencies to express some of their annual performance  targets in percentage terms. However, this official said the actual  percentages are not very meaningful, and he does not have any specific  criteria for how to measure progress to justify the use of the percentages in  the budget request.
NNSA has also established broad performance measures to evaluate the  performance of LANL and LLNL. Specifically, in its performance evaluation  plans for LANL and LLNL for fiscal year 2006, NNSA has established the  following three performance measures:  Use progress toward quantifying margins and uncertainty, and  experience in application, to further refine and document the QMU  methodology.
Demonstrate application of a common assessment methodology (i.e.,  QMU) in major warhead assessments and the certification of Life  Extension Program warheads.
Complete the annual assessment of the safety, reliability, and  performance of all warhead types in the stockpile, including reaching  conclusions on whether nuclear testing is required to resolve any issues.
However, the plan that NNSA uses to evaluate the performance of SNL  does not contain any performance measures or targets specifically related  to QMU, and the performance evaluation plans for LANL and LLNL do not  contain any annual targets that can be measured and linked to the specific  performance measures related to QMU. Instead, the plans state that NNSA  will rely on LLNL and LANL officials to develop the relevant targets and  related dates for each performance measure, as well as to correlate the  level-1 and level-2 milestones with these measures. When asked why these  plans do not meet NNSA’s own requirements, NNSA officials said that they  have not included specific annual performance targets in the plans because  to do so would make it harder for them to finalize the plans and adjust to  changes in NNSA’s budget. However, they said that NNSA is planning on  implementing more stringent plans that will include annual performance  targets when the next contract for LANL and LLNL is developed. In  addition, NNSA officials told us that they recognize the need to develop  performance measures related to QMU for SNL and anticipate  implementing these changes in the fiscal year 2007 performance evaluation  plan.
NNSA officials told us that they have used more specific measures, such as  the completion of level-2 milestones, in their assessment of the weapons  laboratories’ performance since fiscal year 2004. However, we also found  problems with the way NNSA has assessed the performance of the  weapons laboratories in implementing QMU. For example, in NNSA’s  annual performance appraisal of LANL for fiscal year 2004, NNSA states  that LANL had completed 75 percent of the work required to develop “QMU  logic” for the W76 life extension by the end of fiscal year 2004. However,  NNSA officials could not document how they are able to measure progress  toward the development and implementation of QMU logic for the W76 life  extension. Again, an NNSA official responsible for overseeing the Primary  campaign told us that the actual percentages are not very meaningful, and  that he did not have any specific criteria for how to measure progress to  justify the use of the percentage in the appraisal.
In a recent report, we recognized the difficulties of developing useful  results-oriented performance measures for programs such as those geared  toward research and development programs. For programs that can take  years to observe program results, it can be difficult to identify performance  measures that will provide information on the annual progress they are  making toward achieving program results. However, we also recognize that  such efforts have the potential to provide important information to  decision makers.
NNSA officials told us that they recognize the need for developing  appropriate measures to ensure that adequate progress is being maintained  toward achieving the goals and milestones of the campaigns. However,  according to NNSA, very few products of the scientific campaigns involve  the repetition of specific operations whose costs can be monitored  effectively as a measure of performance. As a result, the best measure of  progress for the scientific campaigns is through scientific review by  qualified technical peers at appropriate points in the program. However,  NNSA has not established any performance measures or targets for  implementing QMU that require periodic scientific peer reviews or define  what is meant by “appropriate” points in the program.
section_title: Conclusions
paragraphs: Faced with an aging nuclear stockpile, as well as an aging workforce,  NNSA needs a methodologically rigorous, transparent, and explainable  approach for how it will continue to assess and certify the safety and  reliability of the nation’s nuclear weapons stockpile, now and into the  foreseeable future, without underground testing. After over a decade of  conducting stockpile stewardship, NNSA’s selection of QMU as its  methodology for assessment and certification represents a positive step  toward a methodologically rigorous, transparent, and explainable approach  that can be carried out by a new cadre of weapons designers. However,  important technical and management details must be resolved before  NNSA can say with certainty that it has a sound and agreed upon approach.
First, NNSA must take steps to ensure that all three nuclear weapons  laboratories—not just LANL and LLNL—are in agreement about how QMU  is to be defined and applied. While we recognize that there will be  methodological differences between LANL and LLNL in the detailed  application of QMU to specific weapon systems, we believe that it is  fundamentally important that these differences be understood and, if need  be, reconciled, to ensure that QMU achieves the goal of a common  methodology with rigorous, quantitative, and explicit criteria, as  envisioned by the original 2003 white paper on QMU. More importantly, we  believe that SNL has an important role in the development and application  of QMU to the entire warhead, and we find the continuing disagreement  over the application of QMU to areas outside of the nuclear explosive  package to be disconcerting. There have been several recommendations  calling for a new, technical paper defining QMU, as well as the  establishment of regular forums to further develop the QMU methodology  and reconcile any differences in approach. We believe the NNSA needs to  fully implement these recommendations.
Second, NNSA has not made effective use of its current planning and  program management structure to ensure that all of the research needed to  support QMU is integrated and that scarce scientific resources are being  used efficiently. We believe that NNSA must establish an integrated  management approach involving planning, oversight, and evaluation  methods that are all clearly linked to the overall goal of the development  and application of QMU. In particular, we believe that NNSA needs clear,  consistent, and realistic milestones and regular, technical reviews of the  development of QMU in order to ensure sound progress. Finally, while we  support the development of QMU and believe it must be effectively  managed, we also believe it is important to recognize and acknowledge that  the development and application of QMU, especially the complexities  involved in analyzing and combining uncertainties related to potential  failure modes and performance margins, represents a daunting research  challenge that may not be achievable in the time constraints created by an  aging nuclear stockpile.
section_title: Recommendations for Executive Action
paragraphs: To ensure that the weapons laboratories will have the proper tools in place  to support the continued assessment of the existing stockpile or the  certification of redesigned nuclear components under the RRW program,  we recommend that the Administrator of NNSA take the following two  actions:  Require the three weapons laboratories to formally document an agreed  upon, technical description of the QMU methodology that clearly  recognizes and reconciles any methodological differences.
Establish a formal requirement for periodic collaboration between the  three weapons laboratories to increase their mutual understanding of  the development and implementation of QMU.
To ensure that NNSA can more effectively manage the development and  implementation of QMU, we recommend that the Administrator of NNSA  take the following three actions:  Develop an integrated plan for implementing QMU that contains (1)  clear, consistent, and realistic milestones for the development and  implementation of QMU across the weapons complex and (2) formal  requirements for certifying the completion of these milestones.
Establish a formal requirement for conducting annual, technical reviews  of the scientific research conducted by the weapons laboratories that  supports the development and implementation of QMU.
Revise the performance evaluation plans for the three weapons  laboratories so that they contain annual performance targets that can be  measured and linked to specific milestones related to QMU.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided NNSA with a draft of this report for their review and  comment. Overall, NNSA agreed that there was a need for an agreed-upon  technical approach for implementing QMU and that NNSA needed to  improve the management of QMU through clearer, long-term milestones   and better integration across the program. However, NNSA stated that  QMU had already been effectively implemented and that we had not given  NNSA sufficient credit for its success. In addition, NNSA raised several  issues about our conclusions and recommendations regarding their  management of the QMU effort. The complete text of NNSA’s comments on  our draft report is presented in appendix I. NNSA also made technical  clarifications, which we incorporated in this report as appropriate.
With respect to whether QMU has already been effectively implemented,  during the course of our work, LANL and LLNL officials showed us  examples of where they used the QMU methodology to examine specific  issues associated with the stockpile. At the same time, during our  discussions with laboratory officials, as well as with the Chairs of the  JASON panel on QMU, the Office of Defense Programs Science Counsel,  and the Strategic Advisory Group Stockpile Assessment Team of the U.S.  Strategic Command, there was general agreement that the application of  the QMU methodology was still in the early stages of development. As  NNSA pointed out in its letter commenting on our report, to implement  QMU, the weapons laboratories need to make a number of improvements,  including techniques for combining different kinds of uncertainties, as well  as developing better models for a variety of complex processes that occur  during a nuclear weapon explosion. In addition, the successful  implementation of QMU will continue to rely on the expert judgment and  the successful completion of major scientific facilities such as the National  Ignition Facility. We have modified our report to more fully recognize that  QMU is being used by the laboratories to address stockpile issues and to  more completely characterize its current state of development. At the same  time, however, because QMU is still under development, we continue to  believe that NNSA needs to make more effective use of its current planning  and program management structure.
NNSA raised several specific concerns about our conclusions and  recommendations. First, NNSA disagreed with our conclusion and  associated recommendations that NNSA take steps to ensure that all three  nuclear weapons laboratories are in agreement about how QMU is to be  defined and applied. NNSA stated that we overemphasized the differences  between LANL and LLNL in implementing QMU and that, according to  NNSA, LANL and LLNL have a “common enough” agreement on QMU to go  forward with its implementation. Moreover, NNSA stated that our  recommendations blur very clear distinctions between SNL and the two  nuclear design labs. According to NNSA, QMU is applied to issues  regarding the nuclear explosive package, which is the mission of LANL and  LLNL.
While we believe that some of the technical differences between the  laboratories remain significant, we have revised our report to more  accurately reflect the nature of the differences between LANL and LLNL.  With respect to SNL, we would again point out that SNL officials are still  required to certify the performance of nuclear weapon components under  the conditions of a nuclear explosion and, thus, use similar elements of the  QMU methodology. Therefore, we continue to believe that all three  laboratories, as well as NNSA, would benefit from efforts to more formally  document the QMU methodology and regularly meet to increase their  mutual understanding. As evidence of the benefits of this approach, we  would note that LLNL and LANL are currently developing a revised “white  paper” on QMU, and that in discussions with one of the two authors, he  agreed that inclusion of SNL in the development of the draft white paper  could be beneficial.
Second, NNSA made several comments with respect to our  recommendation that NNSA develop an integrated plan for implementing  QMU that contains clear, consistent, and realistic milestones. For example,  NNSA stated that they expect to demonstrate the success of the  implementation of QMU and the scientific campaigns by the performance  of a scientifically defensible QMU analysis for each required certification  problem. In addition, NNSA stated that the 2010 budget target and the 2014  milestone were developed for different purposes and measure progress at  different times. According to NNSA, the 2010 target describes developing  QMU to the point that it can be applied to certification of a system (e.g., the  W88) without underground testing, while the 2014 milestone is intended to  be for the entire Science campaign effort.
However, as we state in our report, and as acknowledged by NNSA officials  responsible for the Primary and Secondary campaigns, there continue to be  problems with the milestones that NNSA has established for implementing  QMU. Among these problems is the fact that these milestones are not  well-defined and conflict with other performance measures that NNSA has  established for QMU. Moreover, in its comments on our report, NNSA  agreed that better integration and connectivity of milestones between  various program elements would improve the communications of the  importance of program goals and improve the formality of coordination of  program activities, “which is currently accomplished in an informal and  less visible manner.” Given this acknowledgment by NNSA, we continue to  believe that an integrated plan for implementing QMU, rather than NNSA’s  current ad hoc approach, is warranted.
Third, NNSA made several comments regarding our recommendation that  NNSA establish a formal requirement for conducting annual, technical  reviews of the scientific research conducted by the weapons laboratories  that supports the development and implementation of QMU. NNSA stated  that it believes the ad hoc reviews it conducts, such as the JASON review,  provide sufficient information on scientific achievements, difficulties, and  required redirection to manage these programs effectively. As a result,  NNSA stated that it has not selected a single review process to look at  overall success in the implementation of QMU but expects to continue to  rely on ad hoc reviews.
We agree that reviews, such as the JASON review, are helpful, and we relied  heavily on the JASON review, as well as other reviews as part of our  analysis. However, as we point out in the report, the issue is that the  campaign managers for the Primary and Secondary campaigns do not meet  all of NNSA’s own requirements for providing effective oversight, which  include the establishment of formal requirements for conducting technical  reviews of campaign activities. Therefore, we believe that NNSA needs to  take steps to implement its own policies. In addition, we believe that the  ASC campaign provides a good role model for how the Primary and  Secondary campaigns should be managed.
Finally, NNSA made several comments with respect to our  recommendation for NNSA to revise the performance evaluation plans for  the laboratories so that they contain annual performance targets that can  be measured and linked to specific milestones related to QMU. Specifically,  NNSA stated that the implementation of QMU is an area where it is difficult  to establish a meaningful metric. According to NNSA, since QMU is  implicitly evaluated in every review of the components of the science  campaign, NNSA does not believe it is necessary to formally state an  annual QMU requirement. However, as we point out in the report, the  current performance evaluation plans for LANL and LLNL do not meet  NNSA’s own requirements for the inclusion of annual performance targets  that can be measured and linked to the specific performance measures  related to QMU. More fundamentally, since NNSA has placed such  emphasis on the development and implementation of QMU in the years  ahead, we continue to believe that NNSA needs to develop more  meaningful criteria for assessing the laboratories’ progress in developing  and implementing QMU.
We are sending copies of this report to the Administrator, NNSA; the  Director of the Office of Management and Budget; and appropriate  congressional committees. We also will make copies available to others  upon request. In addition, the report will be available at no charge on the  GAO Web site at http://www.gao.gov.
If you or your staff have any questions about this report or need additional  information, please contact me at (202) 512-3841 or aloisee@gao.gov.  Contact points for our Offices of Congressional Relations or Public Affairs  may be found on the last page of this report. GAO staff who made major  contributions to this report are listed in appendix II.
section_title: Comments from the National Nuclear Security Administration
section_title: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the individual named above, James Noel, Assistant Director;  Jason Holliday; Keith Rhodes; Peter Ruedel; and Carol Herrnstadt Shulman  made key contributions to this report.