Nuclear Nonproliferation: NNSA's Threat Assessment Process Could Be Improved
Published: 2015-10-30T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The threat posed by the proliferation of nuclear and radiological weapons remains one of the most pressing U.S. national security challenges, and these threats are evolving. Information produced by the OTH initiative about future proliferation threats is intended to support long-term DNN planning and other DNN management decisions.
The Joint Explanatory Statement Accompanying the National Defense Authorization Act for Fiscal Year 2014 includes a provision for GAO to assess NNSA's nuclear nonproliferation programs. This report (1) describes how NNSA assessed potential proliferation threats through its OTH initiative and assesses the limitations, if any, in the process used by the initiative and (2) examines the extent to which NNSA used information about the potential threats assessed through the OTH initiative in DNN organization and planning decisions.
GAO analyzed NNSA documentation on the OTH initiative and NNSA planning documents. GAO also interviewed DOE and NNSA officials, DOE national laboratory representatives, and external subject matter experts and external validators in peer review groups involved in OTH activities.
section_title: What GAO Found
paragraphs: The National Nuclear Security Administration (NNSA)—a semiautonomous agency within the Department of Energy (DOE)—has used a variety of established methods in its “Over the Horizon” (OTH) initiative to assess potential proliferation threats, but the implementation of these methods had several limitations. NNSA used established methods in its OTH initiative, including: (1) a literature review of studies on proliferation threats; (2) structured interviews with participants from the U.S. government, nongovernmental organizations, and industry; and (3) a peer review of OTH findings by external subject matter experts. Although NNSA used established methods, the way NNSA implemented them had several limitations. For example, NNSA officials did not document the rationale for selecting individuals for structured interviews to show that those selected provided a balanced range of views and were sufficiently knowledgeable. NNSA officials also did not have detailed records or analyses of these interviews. Established methods and common methodological practices call for structured interviews to be documented and analyzed. GAO has found that such practices help ensure the reliability and validity of the information collected. Another limitation was that the peer review was not conducted in a way consistent with established standards. For example, NNSA officials did not document the results of the peer review, as called for by established peer review standards, and some reviewers told GAO that it was unclear how their comments had been incorporated into a 2012 OTH report. These limitations raise concerns about the quality of the analyses produced, and about the usefulness of the OTH initiative, as it has been implemented so far, as a planning tool in NNSA's Office of Defense Nuclear Nonproliferation (DNN).
It is unclear how information generated by the OTH initiative informed recent organization and planning decisions in the DNN office, even though NNSA established the initiative partly to support decision making on such matters. For instance, in January 2015, NNSA reorganized DNN programs by consolidating five DNN program offices into four offices. The extent to which the OTH initiative informed the reorganization is unclear because NNSA officials could not provide documentation or examples of links between OTH findings and elements of the reorganization. Instead, NNSA officials told GAO that the OTH initiative was one of several sources that informed the reorganization and that there were other reasons for it, including consolidating similar functions to achieve efficiencies. In addition, in March 2015, NNSA issued an unclassified strategic plan that was presented as defining and describing NNSA program missions—including DNN programs—to prevent, counter, and respond to future nuclear proliferation and terrorism threats. However, the extent to which the OTH initiative informed the plan was unclear because the plan and the OTH initiative covered different time frames, and because the plan and a draft classified appendix contained conflicting information about the role of the initiative in the plan's development. GAO is not making a recommendation on these matters because NNSA is creating a new strategic planning function that will oversee the OTH process and manage integration of OTH and other long-range studies into future versions of the NNSA strategic plan on nuclear threats.
section_title: What GAO Recommends
paragraphs: In conducting any future proliferation threat assessments, through the OTH initiative or another process, NNSA should better implement established methods, including literature reviews, structured interviews, and peer reviews. NNSA agreed with this recommendation and is taking action to address it.

section_title: Letter
paragraphs: The threat posed by the proliferation of weapons of mass destruction,  including concerns that nonstate actors or additional countries could  obtain nuclear or radiological weapons, remains one of the most pressing  U.S. national security challenges. To address this threat, the National  Nuclear Security Administration (NNSA)—a semiautonomous agency  within the Department of Energy (DOE)—implements nuclear  nonproliferation programs worldwide under its Office of Defense Nuclear  Nonproliferation (DNN). DNN’s mission is to develop and implement  policy and technical solutions to eliminate proliferation-sensitive materials  and limit or prevent the spread of materials, technology, and expertise  related to nuclear and radiological weapons and programs around the  world. DNN programs include efforts to limit or prevent the spread of  nuclear and radiological materials and associated technology and  expertise, advance technologies that detect nuclear and radiological  proliferation worldwide, and eliminate or secure inventories of surplus  materials and infrastructure usable for nuclear weapons.
Some DNN programs have been under way for 20 years or more—for  instance, NNSA’s nonproliferation research and development (R&D)  activities have been implemented under different program names since  the mid-1960s. In addition, nuclear security in Russia and the former  Soviet republics has been a major focus of DNN programs created in the  aftermath of the cold war and the collapse of the Soviet Union in 1991.  Following the terrorist attacks of September 11, 2001, NNSA undertook  new nonproliferation initiatives to better secure stockpiles of weapon- usable nuclear materials and radiological sources in dozens of other  countries around the world.
The proliferation threat environment is changing and evolving. In the last  several years, some nuclear proliferation threats have been substantially  reduced. For example, as a result of the President’s 2009 global nuclear  material security initiative, progress has been made to reduce the threat  posed by vulnerable nuclear material stockpiles around the world.  However, some nuclear nonproliferation efforts have stalled, exemplified  by discontinuation of many joint nuclear security activities between the  United States and Russia following Russia’s invasion of Ukraine in 2014,  which has resulted in diminished DNN program engagement with Russian  nuclear material sites. In addition, some proliferation concerns may  intensify, and new threats may emerge, for example, as commercial  nuclear power expands internationally.
In response to the changing nonproliferation environment, DOE and  NNSA have undertaken several efforts. In particular, in 2010, NNSA  began an initiative, known as the “Over the Horizon” (OTH) initiative, to  identify and assess future nuclear and radiological proliferation threats  and related trends over the next 5 to 10 years—beyond NNSA’s 5-year  budget planning horizon—and to consider the implications for the future  of DNN programs. According to NNSA officials and documentation,  establishment of the OTH initiative was intended to institutionalize long- term DNN planning, and the information produced by the initiative would,  among other things, support DNN program planning and organization  decisions.
In addition, in January 2015, NNSA reorganized the DNN programs into a  new program office structure. Furthermore, in response to a  recommendation contained in an August 2014 interim report by the  Secretary of Energy Advisory Board’s Task Force on Nuclear  Nonproliferation, NNSA, in March 2015, issued the first strategic plan to  define and describe DNN missions and those of other NNSA programs to  prevent, counter, and respond to nuclear terrorism and proliferation.
In this context, the Joint Explanatory Statement Accompanying the  National Defense Authorization Act for Fiscal Year 2014 included a  provision for us to examine the status and future of NNSA’s nuclear  nonproliferation programs. This report (1) describes how NNSA assessed  potential future proliferation threats through its OTH initiative and  assesses the limitations, if any, in the process used by the initiative and  (2) examines the extent to which NNSA used information about the  potential future proliferation threats assessed through the OTH initiative in  DNN organization and planning decisions.
To address our first objective, we analyzed NNSA documentation and  interviewed NNSA officials about the process used by the OTH initiative,  including the key dates, activities, reports, and other milestones  associated with the process. We also interviewed other agency officials,  national laboratory representatives, and independent nonproliferation  experts that DNN officials said were engaged in the OTH initiative  process. We compared the methods NNSA used to established methods  for designing evaluations, as discussed in GAO-developed guidance and  criteria adopted by DOE that were used or recommended by the Office of  Management and Budget and the National Academy of Sciences. To  address our second objective, we interviewed NNSA officials regarding  the DNN reorganization and the rationale for changing the DNN program  structure. In addition, we reviewed the March 2015 NNSA strategic plan  and draft classified appendix describing the planned activities of DNN and  other NNSA programs to prevent, counter, and respond to nuclear  terrorism and proliferation and we interviewed NNSA officials about the  plan. A more detailed discussion of our objectives, scope, and  methodology is presented in appendix I.
We conducted this performance audit from May 2014 to October 2015 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: NNSA’s Deputy Administrator for Defense Nuclear Nonproliferation  directs the DNN office. The DNN Deputy Administrator reports to the  NNSA Administrator and, through the NNSA Administrator, to the  Secretary of Energy. Other NNSA staff in the DNN Deputy Administrator’s  office include a Principal Assistant Deputy Administrator, a Chief  Operating Officer, and a Senior Advisor. Each DNN program office is  headed by an Assistant Deputy Administrator (ADA). In August 2012,  DNN established a DNN Management Council to serve as its corporate  management mechanism. This council, consisting of the NNSA Deputy  Administrator for DNN, the DNN Principal Assistant Deputy Administrator,  the DNN Chief Operating Officer, the DNN Senior Advisor, and the ADAs,  has formulated various DNN program decisions—including on budget,  staffing, and priorities—and has addressed other issues relating to the  DNN programs on an ad hoc basis.
In January 2015, NNSA reorganized the DNN programs into a new  program office structure. Prior to the reorganization, DNN activities were  arranged into five program offices: Office of Global Threat Reduction;  Office of Defense Nuclear Nonproliferation R&D; Office of  Nonproliferation and International Security; Office of International Material  Protection and Cooperation; and Office of Fissile Materials Disposition.  Under the January 2015 DNN reorganization, NNSA officials consolidated  these five DNN program offices into four program offices: Office of  Defense Nuclear Nonproliferation R&D; Office of Material Management  and Minimization; Office of Global Material Security; and Office of  Nonproliferation and Arms Control. Appendix III provides more  information on the DNN reorganization.
In March 2015, NNSA issued a new unclassified strategic plan for NNSA  efforts to address future nuclear proliferation and terrorism threats for  fiscal years 2016 to 2020 in response to an August 2014 interim report by  the Secretary of Energy Advisory Board Task Force on Nuclear  Nonproliferation. Among other things, that task force recommended that  DOE produce a biannual report on nonweapons national security  activities—especially nonproliferation programs—comparable to an  annual report NNSA currently produces on the management of the  nuclear weapons programs over the next 25 years, known as the  Stockpile Stewardship and Management Plan. The DOE Secretary  agreed with this recommendation and asked NNSA to begin drafting such  a report. The March 2015 plan was presented as the first strategic plan to  define and describe DOE and NNSA missions to prevent, counter, and  respond to the threats of nuclear proliferation and terrorism, including the  DNN programs. For each DNN program, the plan identifies program  objectives, priorities, and performance measures; program activities,  accomplishments, and challenges; and program plans over NNSA’s fiscal  year 2016 to fiscal year 2020 budget horizon. In addition, the plan states  that NNSA will issue a classified appendix containing more details on  nuclear and radiological risks, threats, and vulnerabilities, and will update  portions of the plan annually following the release of the President’s  budget request.
section_title: NNSA Used a Variety of Established Methods to Assess Future Proliferation Threats, but the Way It Implemented the Methods Had Several Limitations
paragraphs: The OTH initiative has gone through several phases since it began in  2010 and, as the initiative has evolved, NNSA has used a variety of  established methods—consistent with those in GAO-developed guidance  for designing evaluations and criteria adopted by DOE that were used or  recommended by the Office of Management and Budget and the National  Academy of Sciences—to assess research and development programs’  relevance, quality, and performance. NNSA used these methods  throughout the OTH initiative process to identify and assess future  threats, and they included a literature review, structured interviews and  workshops, and a peer review of OTH findings by external subject matter  experts. NNSA also formed a working group that held meetings and  interviews to update and refine threats identified in earlier phases.  However, although NNSA used established methods, the way NNSA  implemented them had several limitations—including the absence of  documented analysis from the literature review, limited documentation of  the structured interviews, and limited application of peer review  standards. Taken together, the limitations we found in the OTH process  raise concerns about the quality of the analyses produced and about the  usefulness of the OTH initiative, as it has been implemented so far, as a  DNN planning tool.
subsections: section_title: The OTH Process to Identify Future Proliferation Threats Evolved over Several Phases and Used a Variety of Established Methods
paragraphs: NNSA began the OTH initiative process in 2010 to identify and assess  possible future nuclear and radiological proliferation threats and related  trends over the next decade to inform DNN program planning and other  DNN management decisions. Since the initiative’s inception, the process  has evolved over three phases and encompassed a variety of established  methods. It has produced three different reports, with a fourth due later in  2015. (See app. II for a list of the unclassified proliferation-related threats  and trends identified in each of the three reports to date.) Table 1  summarizes the three OTH phases, which are described in greater detail  below.
According to NNSA documents and officials, NNSA officials initiated  phase 1 of OTH work in October 2010 within the NIS program office to  identify key threats and trends that could impact the emerging nuclear  security and nonproliferation environment, discuss their implications for  the NIS program, and provide recommendations regarding both general  direction and specific areas of emphasis for the program for calendar  years 2016 to 2020. This NIS specific phase of OTH work included the  use of several established methods consistent with those in GAO- developed guidance, including a literature review encompassing several  dozen U.S. government documents and other publications, two  workshops, and structured interviews with participants from the U.S.  government, nongovernmental organizations, and industry.
According to GAO-developed guidance of established methods for  designing evaluations, a key first step in designing an evaluation is to  collect background information—such as by conducting a literature  review—to understand a program’s history, related policies, and  knowledge base. Such a review can help focus evaluation questions on  knowledge gaps, identify design and data collection options used in the  past, and provide important context. In addition, to aggregate and  generalize results, it is important to collect uniform data from everyone  interviewed, such as through structured interviews or self-administered  questionnaires, which help ensure that the same questions are asked of  all. According to the guidance, designing a consistent set of responses  into the data collection process, such as by using a structured interview,  helps establish the uniformity of data across units in the sample.
The NIS program office produced a report from this phase in July 2011.  The report identified 10 overarching trends that could impact the future of  the NIS program, grouped under three broad categories: (1) the evolving  strategic landscape, (2) proliferation and nuclear terrorism challenges,  and (3) existing and potential stresses on current nonproliferation and  nuclear security regimes. The report also examined over 50 potential  proliferation-related “wild cards”—events considered to be generally  unanticipated, even if sometimes speculated, extremely difficult to  prepare for, and outside of the U.S. government’s control—that could  impact the trends and planning through 2020, such as terrorist or state  use of a nuclear weapon. Issues related to the organization of the NIS  program and other resource-related issues were not within the scope of  analysis in this phase of OTH study work. The report provided  recommendations for the U.S. government, DNN, and the NIS program,  along with ideas to support their implementation. The report outlined an  overall “logic flow” to the approach that had led to the recommendations,  described the methodology and the activities that were conducted,  identified documents reviewed in the literature review, and identified the  individuals who were interviewed or participated in workshops.
subsections: section_title: OTH Phase 2 (March 2012 to December 2012)
paragraphs: The second phase of OTH work was initiated following the July 2011  report completed in the first OTH phase. According to NNSA officials, the  NNSA Deputy Administrator for DNN asked, after being briefed on the  findings from the 2011 NIS program-specific OTH report, that a second  phase of OTH work be undertaken to identify potential security and  proliferation threats and trends and the implications for all DNN programs  for calendar years 2017 to 2021. In this second phase of OTH work,  which began in March 2012, NNSA developed formal terms of reference  for OTH work specifying the objective, approach, and methodology for  evaluating such threats and trends. (See app. IV for an explanation of  these terms of reference.) For example, the stated OTH initiative  objective was to produce a cross-cutting analysis to help DNN position  itself to best meet nuclear security and nonproliferation challenges over  the next 5 to 10 years.
In this phase, NNSA also formed an OTH working group, consisting of  representatives from each DNN program office, tasked with developing  an updated analysis. To do its work, this working group met weekly,  received briefings from each DNN program office and the DOE Office of  Intelligence and Counterintelligence, and convened a peer review group  to provide external validation of study results consistent with established  methods. Federal agencies and other organizations, such as the  National Academies of Sciences, have established standards and  principles for peer review of technical and scientific work. For instance, a  National Academies report states that peer review is characterized, in  part, as being a documented, critical review of assumptions, calculations  and methodology, performed by a person having technical expertise in  the subject matter to be reviewed who is independent and external of the  work being reviewed. As was the case in the first phase of OTH work,  analysis of DNN organizational issues and other resources-related  issues, such as budget and personnel, were not within the scope of this  phase.
At the end of the second phase, DNN issued a report in December 2012  describing 12 threats and trends affecting nuclear and radiological  security and associated implications for DNN during calendar years 2017  to 2021. According to this report, the findings were based on a synthesis  of views expressed by the working group participants, information derived  from the NIS program-level work in the first OTH phase, and by the  external validators in the peer review group. Specific threats and trends  are presented in the report under three broad categories: (1) global  security dynamics, (2) state-level proliferation, and (3) the  terrorist/nonstate actor threat.
This 2012 report also identified the implications that these threat trends  may have for U.S. national security and DNN, and discussed potential  opportunities for DNN leadership to consider in addressing these threats.  For example, under the global security dynamics category, the report  discusses the threats and trends stemming from the persistence and  escalation of regional conflicts, and suggests the need for proactive— rather than reactive—country-specific efforts to promote nuclear and  radiological material security in conflict-prone regions. The report further  states that to achieve the goal of mitigating the risk of unsecured nuclear  and radiological materials in conflict-prone regions, DNN should consider  various approaches for implementing nuclear and radiological material  security in countries of concern and coordinate and maintain “whole-of- government” crisis mitigation plans in coordination with DOE, the  Department of Defense, and the Department of State.
section_title: OTH Phase 3 (ongoing, initiated mid-2013)
paragraphs: Phase 3 of OTH work began in mid-2013, and it is ongoing. According to  NNSA documentation, the OTH working group concluded during this  phase that the threats identified in phase 2 and in the December 2012  report will remain fundamental concerns for DNN through calendar year  2023. Those threats include potential state-based proliferation and  attempts by nonstate actors to acquire nuclear materials. Nevertheless,  the OTH working group has reevaluated and further refined the potential  proliferation-related threats identified in the second OTH phase study  through use of internal working group meetings and interviews with NNSA  program officials. According to NNSA officials, these program officials  include staff members from DNN program offices, NNSA’s Office of  Emergency Operations, and NNSA’s Office of Counterterrorism and  Counterproliferation, as well as officials from the DOE Office of  Intelligence and Counterintelligence.
An OTH report produced by NNSA in October 2013 presented an  updated list and descriptions of 10 threats and trends affecting nuclear  and radiological security and nonproliferation, focused on calendar years  2018 to 2023. A key objective, according to NNSA officials, was to  provide more specificity to these 10 proliferation threats and trends in  order to better support long-range DNN program planning. To that end,  NNSA officials involved in the OTH process told us that, following the  publication of the October 2013 report, members of the OTH working  group worked with officials from the DOE Office of Intelligence and  Counterintelligence to prepare more detailed analyses and “map” the  threats and trends identified in the October 2013 report to specific regions  and countries. The OTH working group is planning to produce a report on  the results of the threat mapping process in the summer of 2015.
See figure 1 for a timeline illustrating the three phases of the OTH  process and the main activities conducted under each phase.
section_title: The Way NNSA Implemented Established Methods Had Several Limitations
paragraphs: Through reviews of NNSA documentation and interviews with NNSA  officials, we identified several limitations in the way NNSA implemented  the established methods it used (i.e., the literature review, structured  interviews, and peer review by external subject matter experts) through  the OTH initiative to identify and assess future proliferation-related threats  and trends, which raise concerns about the quality and usefulness of the  analyses produced through the process.
subsections: section_title: Absence of Documented Analysis from Literature Review
paragraphs: The 2011 OTH report, other NNSA documents related to the OTH  initiative, and NNSA officials noted that the process used by the OTH  initiative to assess the nonproliferation environment and identify emerging  threats and trends over the next decade included a review of relevant  literature, conducted in the first OTH phase, consistent with established  methods as discussed earlier. The literature collected by NNSA for this  review included several dozen U.S. strategy documents, policy speeches,  and scholarly articles on topics ranging from overarching regional trends,  to technological advances, to predictions related to countries of concern.  We found limitations in the literature review conducted in the first OTH  phase based in part on GAO-developed guidance for designing  evaluations. According to this guidance, which is relevant to conducting  evaluations, such as those conducted by the OTH initiative, a systematic  approach to designing evaluations enhances quality, credibility, and  usefulness.
However, NNSA did not follow consistent common methodological  practices in its literature review. For example, NNSA did not document  the methodology or scope for developing this literature review list, the  criteria they used for selecting the sources cited in their review, or to  whom the literature review was disseminated, according to an NNSA  official. NNSA officials also did not prepare a summary analysis of the  findings from the literature review or have other documentation showing  how the materials and information gleaned in the literature review were  used in the development of the 2011 report on threats and trends through  the OTH initiative process consistent with common methodological  practice. In response, an NNSA official told us that the collected  documents—consisting of four binders of documents altogether—were  intended to serve as OTH background and be available for others in the  office to read if they chose to do so.
Because NNSA officials did not document the methodology and scope or  a written summary or analysis of the findings from the OTH literature  review used in the 2011 report, it is difficult to assess how the background  information was used to provide context in developing the potential  proliferation threats and trends through the OTH initiative. These  conditions, in turn, raise potential quality concerns related to the validity  and credibility of the literature review results.
section_title: Limited Documentation of Structured Interviews
paragraphs: According to NNSA documents we reviewed, an important component of  the OTH process was conducting structured interviews in the first OTH  phase, consistent with established methods as discussed earlier. NNSA  conducted these interviews with more than 40 staff members, officials  and experts from DOE and NNSA, the national laboratories, National  Security Council, Department of State, Department of Defense, the  Intelligence Community, and nongovernmental organizations. According  to the NNSA documents, the interviews were intended to help gather a  range of perspectives on the key threats and trends that could  characterize the future strategic environment, as well as to gather ideas  on potential programmatic needs and priorities.
We found, based on our interviews with NNSA officials, that the OTH  structured interview process conducted in the first OTH phase had  limitations including the following:
According to NNSA officials, NNSA did not document the rationale for  selecting individuals to be interviewed to show that the people  selected as data sources provided a balanced range of views, were  sufficiently knowledgeable, and that their opinions may be considered  candid and accurate, consistent with common methodological  practice. Instead, NNSA officials involved in the OTH initiative said  that they relied on their own subject matter knowledge to consult with  experts with whom they were familiar.
According to NNSA officials, NNSA did not have detailed records or  analyses of these interviews, although this is called for by established  methods and common methodological practices. NNSA officials  provided us with a list of the questions that were asked to participants  in the structured interviews and a document containing quotes  excerpted from these interviews. However, this document did not  identify or attribute remarks to any of the interview participants, and  NNSA officials said that they did not have individual records or  analyses or attempt to systematically synthesize information from  these interviews.
To ensure consistency and to prevent biased interpretation of results, it is  common methodological practice to document individual interviews in  records and to synthesize information obtained in interviews. We have  found that such practices also help ensure the reliability and validity of the  information being collected.
It is unclear how DNN used information from the interviews to develop the  potential proliferation threats and trends of concern identified in OTH  reports. For instance, it is not possible to establish how various views  were reconciled or reflected in the OTH analysis and reports. Because of  the limitations, NNSA does not have assurance that the structured  interviews—in terms of selection of individuals to be interviewed, record  keeping, and analysis—provided OTH with objective, balanced, and  independent views and quality information.
section_title: Limited Application of Peer Review Standards
paragraphs: NNSA officials conducted a peer review process to validate the findings  developed during the second OTH phase consistent with established  practice, as discussed earlier. Specifically, according to NNSA officials,  the peer review by external validators consisted of a June 2012  presentation of OTH findings to a group of seven individuals who were  briefed on the OTH results and were asked to comment at the meeting.  However, this process was not carried out in a way consistent with  established peer review standards as we discussed earlier. For example,  a 2004 DOE peer review guide defines peer review as a rigorous, formal,  and documented evaluation process using objective criteria and qualified  and independent reviewers to make a judgment of the  technical/scientific/business merit, the actual or anticipated results, and  the productivity and management effectiveness of programs and/or  projects.
Based on our interviews with NNSA officials and individuals whom NNSA  identified as external subject matter experts, we identified the following  limitations in the peer review process that were not consistent with  established peer review standards:
According to NNSA officials, NNSA did not have a methodology  explaining how individuals were chosen as validators to ensure a  balanced range of feedback. However, established peer review  standards state the importance of having a process for selecting peer  reviewers that ensures a broad and balanced range of expertise and  perspectives.
Some of these individuals told us that they were not provided written  documents to review in advance of or during the meeting, such as a  draft of the OTH phase 2 report. However, according to established  peer review standards, peer reviewers should be provided with  relevant documentation in advance of the peer review session, in  order to give reviewers time to familiarize themselves with the subject  of the review.
According to NNSA officials, NNSA did not document the results of  the peer review process, such as through a summary document or  record of analysis, consistent with established peer review standards  that state the importance of receiving written comments from all peer  reviewers and producing a record of these comments.
As a result of the departures from these established standards during the  OTH peer review process, it is not clear whether the process provided  independent, objective opinions reflecting all individual inputs and the full  range of reviewer comments, or how these opinions were reconciled and  reflected in OTH reports to ensure quality information.
Taken together, the limitations we found in the OTH process raise  concerns about the quality of the analyses produced and about the  usefulness of the OTH initiative—as it has been implemented so far—as  a DNN planning tool.
section_title: It Is Not Clear How Recent NNSA Nonproliferation Organization and Planning Decisions Were Informed by the Over the Horizon Initiative
paragraphs: The OTH initiative was intended to help inform DNN management in its  decisions about the future DNN organization and resources needed to  address future threats, but it is not clear how information generated by the  OTH initiative informed the January 2015 DNN reorganization. In addition,  the OTH initiative was established to institutionalize long-term DNN  planning and support development of DNN strategic plans, but the extent  to which the OTH initiative informed the March 2015 DOE and NNSA  strategic plan to reduce global nuclear threats is unclear. According to  NNSA officials, the OTH initiative is in the process of being integrated into  a new DNN strategic planning and integration function that will be led out  of the office of the DNN Deputy Administrator, and which will manage  integration of OTH studies into strategic planning documents, such as  future versions of the NNSA strategic plan to reduce global nuclear  threats.
subsections: section_title: It Is Not Clear How OTH Assessments Informed DNN’s Recent Reorganization
paragraphs: The OTH initiative was intended to develop and provide information on  potential future proliferation threats to help inform DNN management in its  decisions about the future DNN organization and resources needed to  address those threats. In January 2015, NNSA reorganized the DNN  programs into a new program office structure by consolidating the  previous five DNN program offices into four offices. However, it is not  clear how information generated by the OTH initiative informed the  decision to reorganize DNN programs into the new structure.
In October 2014, before the January 2015 reorganization, NNSA officials  in the DNN Deputy Administrator’s office stated that the then-planned  DNN reorganization was informed by the potential future proliferation- related threats and trends identified through the OTH process— specifically those identified in the July 2011 and December 2012 reports  prepared during the first and second phases of the OTH initiative.  However, these officials could not provide us with documentation or point  to specific examples that could demonstrate direct links between the  findings in the 2011 and 2012 OTH reports and elements of the DNN  reorganization, such as how changes in certain threats or concerns about  new or emerging threats prompted changes in program structure. In a  subsequent meeting, NNSA officials in the DNN Deputy Administrator’s  office stated that the information produced through the OTH process  about the changing proliferation threat environment provided general  context for the decision to reorganize the DNN programs. They also  stated that the initiative was one of several sources of information that  DNN management considered in developing the new DNN structure;  according to these officials, other sources included information about  evolving proliferation threats provided by the intelligence community.
NNSA officials in the DNN Deputy Administrator’s office also stated that  there were other drivers behind the reorganization, and that it was not  solely a threat-based reorganization. Specifically, these officials stated  that a key reorganization goal was to achieve greater efficiency in DNN  program management by consolidating similar nonproliferation functions  that were fragmented among several DNN program offices. For instance,  NNSA officials stated that some programs were spread among several  DNN program offices working on issues related to the security of nuclear  material overseas and that the reorganization would consolidate those  functions under a single program office. In addition, these officials told us  the reorganization was motivated by uncertain prospects for future joint  nuclear security work in Russia and by the completion or review of certain  DNN programs, such as an assessment of alternative options for U.S.  plutonium disposition. NNSA officials stated that the new structure will  provide DNN the capability and flexibility to respond to changes in the  future proliferation threat environment.
section_title: The Extent to Which OTH Threat Assessments Informed DOE’s and NNSA’s Plan for Countering Nuclear Threats Is Unclear
paragraphs: The OTH initiative was established to institutionalize long-term DNN  planning and inform development of DNN strategic plans. In March 2015,  NNSA issued a strategic plan to reduce global nuclear threats that was  presented as the first strategic plan to define and describe NNSA  program missions—including DNN programs—to prevent, counter, and  respond to future threats of nuclear proliferation and terrorism. The  unclassified plan states that a classified appendix will be issued, which  will provide more details about future nuclear and radiological threats and  NNSA activities to address the evolving global nuclear security  environment. On the basis of our review of the unclassified plan and a  draft of the classified appendix, as well as our interviews with NNSA  officials, we found that the extent to which the OTH initiative informed the  new plan is unclear for two reasons.
First, the time frames covered by the March 2015 plan and the OTH  initiative are not aligned. The March 2015 plan focuses on NNSA program  plans over the agency’s 5-year budget planning horizon—i.e., fiscal years  2016 to 2020. However, the objective of the OTH initiative is to assist  DNN by analyzing nuclear and radiological security and nonproliferation  challenges 5 to 10 years into the future, or beyond NNSA’s 5-year budget  planning horizon. For example, in October 2013, the OTH initiative  produced an updated analysis of nuclear and radiological security threats  and trends for the 2018 to 2023 time frame. Because the March 2015  plan focuses on NNSA program objectives, activities, and plans for the  next 5 fiscal years, it is not clear how information produced through the  OTH initiative would align with that planning effort.
Second, the extent to which the OTH initiative informed development of  the new NNSA plan is unclear because of conflicting information in the  unclassified plan and the draft of the classified appendix regarding the  contributions of the initiative in supporting both documents. The  unclassified plan includes several references to the OTH initiative,  including the future nuclear security threats and trends identified through  the initiative, and the role it will play in developing future versions of the  plan. For instance, the document’s executive summary and conclusion  state that NNSA will continue to use OTH studies to validate that its  efforts are focused on addressing current threats and to anticipate  emerging and evolving threats. The plan also states that information from  OTH studies, along with information from external sources, will be used to  conduct cross-program and program-specific risk assessments and  priority setting. In addition, the document describes 5 proliferation-related  threat trends that the plan states were identified by the OTH initiative.  These 5 threat trends, described in general terms in the plan’s  introduction, consolidate most of the 10 threats and trends for the 2018 to  2023 time frame identified in the October 2013 OTH report. (See app. II  for a list of the unclassified proliferation-related threats and trends  identified in the October 2013 OTH report and the March 2015 plan.)
In April 2015, we reviewed a copy of the draft classified appendix to the  March 2015 plan, and we discussed the appendix and the unclassified  plan with the NNSA official who coordinated production of both  documents. Based on our review of the documents and our discussion  with the NNSA official, we did not find a clear connection between the  plan and the OTH initiative. The draft classified appendix did not contain  any references to the OTH initiative or findings of future proliferation  related threats generated through its process. Instead, the NNSA official  stated that the details on the future proliferation-related threats and trends  described in the classified appendix were derived from a classified  analysis prepared by an entity within the intelligence community, not from  the completed OTH studies. Furthermore, the NNSA official told us that  the OTH initiative was on a “separate but parallel track” with preparation  of the strategic plan, and that findings from the OTH process were used  only to provide general setup and context for the plan—specifically the 5  threat trends discussed in the unclassified plan’s introduction.
The official said that information from the OTH studies did not provide the  threat basis for the new plan because the threats and trends identified by  the initiative were stated in terms that were too general or academic.  However, he noted that the OTH initiative was still completing regional  studies to map proliferation-related threats and trends identified in the  October 2013 OTH report to specific countries and regions, and he told  us that findings from the regional studies could inform the next version of  the strategic plan, to be issued in 2016.
NNSA officials told us that the OTH initiative is in the process of being  integrated into a new DNN strategic planning and integration function that  was implemented as part of the reorganization, which will be led out of  the office of the DNN Deputy Administrator. Among other things, the  NNSA official responsible for managing this effort will lead the analysis  and coordination of the OTH process and other long-range studies, and  manage the integration of these studies into strategic planning  documents, such as future versions of the NNSA strategic plan to reduce  global nuclear threats.
section_title: Conclusions
paragraphs: DNN programs remain critical to addressing the serious threats  associated with the proliferation of nuclear and radiological weapons.  NNSA has taken an important step through the OTH initiative to identify  and assess proliferation-related threats and trends over the next decade  and evaluate what they mean for the future of the DNN programs. NNSA  used established methods for designing the OTH studies, but we found  several limitations in the way the NNSA implemented these methods.  These limitations raise concerns about the quality and usefulness of the  analyses produced through the OTH process, including absence of  documented analysis from its literature review; limited documentation of  structured interviews; and limited application of established peer review  standards. In turn, these limitations raise concerns about the usefulness  of the OTH initiative—as it has been implemented so far—as a DNN  planning tool. NNSA could enhance the usefulness of the OTH initiative  as it moves forward by better implementing established methods— including literature reviews, structured interviews, and peer reviews  associated with threat identification and assessment.
section_title: Recommendation for Executive Action
paragraphs: To ensure that any future NNSA effort—through the OTH initiative or  another process—to assess proliferation threats and the implications for  DNN produces high-quality information, we recommend that the NNSA  Administrator better implement established methods, including literature  reviews, structured interviews, and peer reviews.
section_title: Agency Comments
paragraphs: We provided a draft of this report to NNSA for comment.  In written  comments provided by NNSA (reproduced in app. V), NNSA generally  agreed with our findings and recommendation to improve its OTH threat  assessment process.  Specifically, NNSA stated that as the OTH initiative  continues to evolve and mature, NNSA will incorporate the lessons  learned from our report into its standard processes. NNSA also noted that  it has created an Office of Strategic Planning and Implementation within  DNN to manage the OTH initiative and other information and planning  tools.
We are sending copies of this report to the appropriate congressional  committees, the NNSA Administrator, and other interested parties. In  addition, the report is available at no charge on the GAO website at  http://www.gao.gov.
If you or your staff members have any questions about this report, please  contact me at (202) 512-3841 or trimbled@gao.gov. Contact points for  our Offices of Congressional Relations and Public Affairs may be found   on the last page of this report. GAO staff who made major contributions to  this report are listed in appendix VI.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: Our objectives were to (1) describe how the National Nuclear Security  Administration (NNSA) assessed potential future proliferation threats  through its “Over the Horizon” (OTH) initiative and assess the limitations,  if any, in the process used by the initiative and (2) examine the extent to  which NNSA used information about the potential future proliferation  threats assessed through the OTH initiative in DNN organization and  planning decisions.
To address our first objective, we interviewed members of management  in the Office of Defense Nuclear Nonproliferation (DNN)—including the  Deputy Administrator for Defense Nuclear Nonproliferation, the DNN  Principal Assistant Deputy Administrator, the DNN Senior Advisor, and  DNN program office Assistant Deputy Administrators—and other NNSA  officials involved in implementing the OTH initiative to obtain views and  information on the goals of and the process used by the OTH initiative,  such as key dates, activities, reports, and other milestones associated  with the process. We also analyzed documentation NNSA officials  provided regarding the OTH process and findings from it. In particular, we  analyzed the OTH published reports: Office of Nonproliferation and  International Security: Over the Horizon Study, 2016-2020, July 2011;  Office of Defense Nuclear Nonproliferation Over the Horizon Opportunity  Analysis, 2017-2021 December 2012; and Threats and Trends Impacting  Nuclear and Radiological Security and Nonproliferation in 2018 to 2023,  October 2013. We also interviewed individuals who participated in OTH  process activities. NNSA officials provided us documents that identified  participants—including NNSA officials, other agency officials, national  laboratory representatives, and independent nonproliferation experts—in  four activities of the OTH process: (1) October 2010 workshop; (2)  November 2010-February 2011 structured interviews; (3) April 2011  workshop; (4) June 2012 meeting of external validators in a peer reviewer  group. We established a criterion to select a nonprobability sample of  participants in these sessions; we selected individuals that participated in  multiple activities of the OTH. This allowed us to gain more  comprehensive responses across activities. In selecting participants using  this criterion, the list of 146 participants across the four OTH initiative  activities was reduced to the 32 participants who were involved in more  than one OTH activity. We excluded participants from other U.S.  agencies, as well as participants who had only served as activity  facilitators. We also excluded NNSA personnel to better ensure candid  responses and to aim to avoid a conflict of interest. This narrowed our list  down to 18 participants. We contacted all 18 of these participants and  scheduled interviews based on their availability. We were able to  schedule and conduct interviews with 15 of these participants. We  determined that the responses from these participants were consistent  enough to constitute saturation, and we did not schedule any additional  interviews. We compared the methods NNSA used to established  methods for designing evaluations, as discussed in GAO-developed  guidance, and in criteria adopted by the Department of Energy that were  used or recommended by the Office of Management and Budget and the  National Academy of Sciences.
To address our second objective, we interviewed NNSA officials  regarding the DNN reorganization and the rationale for changing the DNN  program structure, and the extent to which information provided through  the OTH initiative regarding future proliferation-related threats and trends  informed the restructuring of the DNN programs. In addition, we reviewed  the March 2015 NNSA strategic plan and draft classified appendix  describing the planned activities of DNN and other NNSA programs to  prevent, counter, and respond to nuclear terrorism and proliferation  threats over the next 5 years, and we interviewed NNSA officials about  the plan and the extent to which it was informed by information generated  through the OTH initiative. Because it was outside the scope of our  review, we did not assess the plan or the reorganization themselves, for  example, by comparing them against established criteria for effective  strategic planning or effective organizational transformation.
We conducted this performance audit from May 2014 to October 2015 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Appendix II: Nuclear and Radiological Proliferation Threats and Trends Identified Through NNSA’s “Over the Horizon” (OTH) Initiative as Presented in NNSA Reports
subsections: section_title: Office of Nonproliferation and International Security: Over the Horizon Study, 2016-2020, July 2011 (Report from first OTH phase) Continued power transition, increasing multipolarity, and regional instabilities
paragraphs: Prevent, Counter, and  Respond—A Strategic Plan  to Reduce Global Nuclear  Threats (FY 2016-2020),  March 2015  Securing and managing  nuclear and radiological  materials will be challenged  by the significant amounts of  these materials (and possibly  increased amounts of  weapon-usable nuclear  materials), including in  regions of concern, as well as  by the erosion of control  within weak or failing states.
Possessing nuclear weapons  capabilities still could be seen  as salient and desirable for  some state and nonstate  actors hostile to U.S. and  allied interests, putting strains  on monitoring, verifying, and  maintaining arms control and  nonproliferation regimes.
The global expansion of civil  nuclear power and the wide  use of radiological sources  may accelerate the spread of  dual-use technology and  knowledge and increase  demands on safety, security,  safeguards, and emergency  response systems.
Expanding global trade  volumes and sophistication of  illicit procurement networks  will increase the opportunities  for state and nonstate actors  to acquire dual-use nuclear  equipment and technology.
section_title: Office of Defense Nuclear Nonproliferation (DNN) Over the Horizon Opportunity Analysis, 2017-2021, December 2012 (Report from second OTH phase) Increased sophistication of trafficking networks
paragraphs: Prevent, Counter, and  Respond—A Strategic Plan  to Reduce Global Nuclear  Threats (FY 2016-2020),  March 2015  Rapidly changing  technologies and greater  diffusion of dual-use  knowledge are expected to  provide more ways for  terrorists to threaten nuclear  security systems and easier  acquisition pathways to  nuclear weapons capabilities.
Rise in “lone wolf” operations  Persistent “insider threats”
section_title: Appendix III: Defense Nuclear Nonproliferation Program Reorganization
paragraphs: In January 2015, the National Nuclear Security Administration (NNSA)  reorganized the Defense Nuclear Nonproliferation (DNN) programs into a  new program office structure. Prior to the reorganization, DNN activities  were arranged into the following five program offices:
Office of Global Threat Reduction. This office implemented the  Global Threat Reduction Initiative (GTRI) that consisted of three  subprograms: a Highly Enriched Uranium (HEU) Reactor Conversion  subprogram that supported conversion of domestic and international  civilian research reactors and isotope production facilities from HEU to  low-enriched uranium (LEU); a Nuclear and Radiological Material  Protection subprogram, which worked to install security upgrades on  high-priority nuclear and radiological materials at civilian sites in the  United States and other countries; and a Nuclear and Radiological  Material Removal subprogram that included activities to (1) support  removal and disposal of U.S.-origin HEU, Russian–origin HEU, and  other high-risk nuclear materials; (2) develop capabilities to address  nuclear materials associated with “emerging threats” and rapidly  denuclearize a country through nuclear material removal when such  opportunities emerge, such as in Libya in 2004; (3) support removal  and disposal of excess or abandoned radiological materials in other  countries; and (4) support rapid removal and disposal of radiological  materials in the United States.
Office of Defense Nuclear Nonproliferation R&D. This office  implemented the DNN Research and Development (R&D) Program,  which included two subprograms supporting long-term basic and  applied research, development, and testing of new technologies: a  Proliferation Detection subprogram to (1) improve U.S. capabilities to  detect and monitor nuclear weapons production and proliferation of  nuclear weapon-usable materials and (2) a Nuclear Detonation  Detection subprogram to detect nuclear explosions worldwide.
Office of Nonproliferation and International Security (NIS). This  office implemented the NIS program that included four subprograms  to: (1) develop NNSA nonproliferation and arms control policy and  support implementation of bilateral and multilateral nonproliferation  requirements stemming from national nonproliferation initiatives,  agreements, and treaties; (2) support the International Atomic Energy  Agency’s (IAEA) safeguards system and international nuclear  security, including a safeguards policy component that maintains  qualified and knowledgeable safeguards staff at U.S. national  laboratories and IAEA and implements U.S. safeguards obligations at  DOE facilities, a safeguards engagement component that provides  safeguards implementation training to other countries, a safeguards  technology development component that manages a support program  to IAEA’s safeguards department for safeguards R&D and resolution  of technical safeguards issues, and an international nuclear security  component that conducts assessments of the physical protection of  U.S.-obligated nuclear material overseas and works with foreign  partners and IAEA to develop and implement nuclear security best  practices; (3) promote nuclear controls, by strengthening national  nuclear export control systems, developing international nuclear  forensics capabilities, conducting technical reviews of domestic  export licenses, and providing technical support to U.S. government  agencies involved in weapons of mass destruction (WMD) interdiction;  and (4) support nuclear verification, including activities in support of  nuclear warhead dismantlement and fissile materials transparency,  nuclear noncompliance verification, and implementation of HEU  transparency arrangements.
Office of International Material Protection and Cooperation  (IMPC). This office implemented the IMPC program that consisted of  two major subprograms, a nuclear Material Protection, Control, and  Accounting (MPC&A) program and a Second Line of Defense  program. The MPC&A program included several subprograms: a  Weapons Material Protection subprogram to provide MPC&A  upgrades at nuclear weapon material sites in Russia and other  countries; a Material Consolidation and Civilian Sites subprogram to  consolidate and convert weapon-usable nuclear material stocks in  Russia, and to improve security at civilian sites in Russia and in other  locations outside of Russia; a National Infrastructure and  Sustainability subprogram to assist Russia and other countries in  developing national MPC&A infrastructure and supporting the  sustainability of U.S.-funded security upgrades. The Second Line of  Defense subprogram strengthens the capacity and commitment of  foreign countries to deter, detect, and interdict illicit trafficking of  nuclear and other radioactive materials by working with foreign  governments to deploy radiation detection systems and by providing  training, maintenance, and sustainability assistance.
Office of Fissile Materials Disposition. This office implemented two  programs: (1) an HEU Disposition program to downblend surplus U.S.  HEU to LEU and (2) a Plutonium Disposition program to dispose of  surplus U.S. plutonium and support disposition of Russian weapon- grade plutonium. The Plutonium Disposition program included  budgeted funding for construction of facilities in the United States to  dispose of surplus U.S. plutonium.
Under the January 2015 DNN reorganization, NNSA officials consolidated  these five DNN program offices into the following four program offices:
Office of Defense Nuclear Nonproliferation R&D. This office and  the DNN R&D program remain unchanged under the new DNN  structure.
Office of Material Management and Minimization. This office will  implement some program activities previously conducted under the  Office of Global Threat Reduction and the Office of Fissile Materials  Disposition. Specifically, this office will implement (1) HEU reactor  conversion activities conducted previously through the GTRI program;  (2) activities to remove and dispose of U.S.-origin HEU, Russian– origin HEU, and other high-risk nuclear materials, and to develop  capabilities to address “emerging threats” and rapidly denuclearize a  country through nuclear material removal that were conducted  previously through the GTRI program; and (3) the U.S. uranium  disposition and U.S. and Russian plutonium disposition program  activities, including management of related plutonium disposition  facility construction in the United States, that were conducted  previously through the Fissile Materials Disposition program. In the  fiscal year 2016 budget request, NNSA has placed funding for  construction of the plutonium disposition facilities under a separate  program entry in the budget: “Nonproliferation Construction.”  However, organizationally, this office will oversee the nonproliferation  construction program budget.
Office of Global Material Security. This office will incorporate the  program activities previously conducted under the IMPC program. In  addition, the Global Material Security Program will implement the  domestic and international radiological removal and nuclear and  radiological material protection functions that were previously under  the GTRI program. This new program will also implement two  activities conducted previously under the NIS program: international  nuclear forensics cooperation efforts and the portion of the  international nuclear security program activities involving engagement  with foreign partners and IAEA on nuclear security best practices.
Office of Nonproliferation and Arms Control. This office will  continue to implement most of the activities conducted previously  under the NIS program. However, as noted above, international  forensics cooperation activities and the portion of international nuclear  security activities focused on promoting nuclear security best  practices with foreign partner governments and IAEA have been  realigned under the Global Material Security Program. This new office  will continue to implement international nuclear security program  activities to assess physical protection of U.S.-obligated nuclear  materials in overseas locations.
section_title: Appendix IV: Terms of Reference for the “Over the Horizon” Initiative
paragraphs: The following are the terms of reference for the National Nuclear Security  Administration’s “Over the Horizon” (OTH) initiative working group created  in March 2012, at the beginning of the second OTH study phase.
subsections: section_title: Objective:
paragraphs: Produce a cross-cutting analysis to help Defense Nuclear  Nonproliferation (DNN) position itself to best meet nuclear and  radiological security and nonproliferation challenges over the next 5 to 10  years.
section_title: Approach:
paragraphs: DNN will establish a working group with representation from all DNN  offices with the Office of Nonproliferation and International Security (NIS)  as the Chair. The working group will provide regular updates to the DNN  Assistant Deputy Administrators/Associate Assistant Deputy  Administrators. Work will begin immediately.
section_title: Methodology:
paragraphs: Define the threat environment and its trends as insight to how the threats  may evolve over the next 5 to 10 years, focusing on aspects of the threat  most relevant to DNN. This would be informed by the already completed  NIS effort.  1. Identify core strengths of the DNN functional bureaus. What do we do  best? What are our strongest assets? What do we bring to the table that  is unique, particularly with respect to the interagency process?  2. On the basis of the above, identify prospective and emerging nuclear  proliferation and nuclear/radiological security challenges, gaps, needs,  and opportunities that will require greater attention by the USG and its  allies, and where it makes sense for DNN to take a proactive role in  meeting these challenges.  3. Based on these challenges, and the status of programmatic efforts,  identify gaps, needs, and opportunities. Taking into account that  unanticipated events may occur between now and 2021 that could involve  a DNN response.  4. Develop a set of recommendations for DNN offices to factor into their  future programs and activities consistent with the analysis above.  Recommendations will consider scope of programs, resources, and  organizational structure.
section_title: Appendix V: Comments from the National Nuclear Security Administration
section_title: Appendix VI: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the individual named above, Jonathan Gill (Assistant  Director), Antoinette Capaccio, William Hoehn, Cynthia Norris, Sophia  Payind, Steven Putansu, Dan Royer, and Kiki Theodoropoulos made key  contributions to this report.