Nuclear Nonproliferation: Better Information Needed on Results of National Nuclear Security Administration's Research and Technology Development Projects
Published: 2017-02-03T00:00:00

Fast Facts:
NNSA manages research and technology development projects intended to help stop nuclear proliferation. What does it have to show for its $1 billion investment on these projects in fiscal years 2012–2015? There has been progress, but NNSA hasn't tracked or documented results consistently.
We looked at a sample of 91 projects. Of those, 88 had made technological progress by, for example, building instrument hardware. Of those, 33 were transitioned to users for further development or use, and 17 of those are currently in use. We recommended that NNSA consistently track and document its project results.

Highlights:
section_title: Why GAO Did This Study
paragraphs: Nuclear proliferation is a national security threat. DNN's two research and technology development programs develop technical solutions to prevent nuclear proliferation. From fiscal years 2012 through 2015, these programs obligated over $1.1 billion on 511 projects. The DNN R&D program obligated over $1 billion of this amount.
A House report included a provision for GAO to review the programs. This report evaluates, among other objectives, (1) the extent to which projects managed by NNSA's research and technology development programs resulted in advanced, transitioned, or deployed nonproliferation technologies and (2) how NNSA measures the performance of its research and technology development programs and projects.
GAO reviewed documents on a non-generalizable sample of 91 research and technology development projects, selected from all program areas to include projects with the largest amount of funding and to capture a broad range of the programs' technical functions. GAO also reviewed DNN's program objectives and publicly reported performance measures and interviewed NNSA officials and technology users at other agencies.
section_title: What GAO Found
paragraphs: The full extent to which research and technology development projects managed by two programs in the National Nuclear Security Administration's (NNSA) Office of Defense Nuclear Nonproliferation (DNN) have resulted in advanced (progressed technologies or science supporting them), transitioned (provided to users for further development or deployment), or deployed (used in the field) technologies is unclear because NNSA does not consistently track and document all of these outcomes. Specifically, the DNN Research and Development (DNN R&D) and Nonproliferation and Arms Control (NPAC) programs track and document technology advancements resulting from their projects, such as in project reports. However, NNSA officials acknowledged that DNN R&D—by far the larger program—does not consistently track and document whether its projects result in technologies transitioned to or deployed by end users. In contrast, the NPAC program tracks and documents these project outcomes. By not consistently tracking and documenting technology transition and deployment outcomes, NNSA is unable to demonstrate the full results of its projects. GAO gathered information from various sources on a sample of 91 projects. Of these, 88 projects advanced technologies by, for example, building instrument hardware or developing models for data analysis; the other 3 did not advance technologies but assessed potential applications of existing technologies. Among the 88 projects that advanced technologies, 33 also resulted in technologies being transitioned, including software used to analyze nuclear detonations. Finally, of these 33 projects, 17 also resulted in deployed technologies, including an enrichment monitoring tool used in Iran and space-based nuclear detonation sensors. Reasons that some technologies in the sample did not move beyond the advancement or transition stage included that the technology needed further development or evaluation before being used.
Limitations with the clarity of the DNN R&D program's publicly reported performance measures make them difficult to interpret, potentially hindering users' ability to determine the program's progress; better documentation of project performance against baseline targets may enhance NNSA's performance assessments. The DNN R&D program's performance measures have clarity limitations because they do not, for example, define measurement criteria or provide context justifying how the program determined that it met its performance targets; this may cause external users of the measures, including Congress, to have difficulty interpreting NNSA's assessment of performance. In a December 2011 report, GAO recommended that NNSA clarify its publicly reported measures for DNN's R&D program. GAO continues to believe that NNSA should do so. Regarding project-level performance, NNSA uses project plans to establish baseline targets for projects' scope and completion date, and tracks progress by communicating with project managers and reviewing documents such as quarterly and final project reports. However, NNSA officials said that final project reports do not document their assessment of performance against baseline targets and that there is no common template for final project reports, which GAO confirmed in a review of several final project reports. Documenting such assessments could enhance NNSA's ability to assess project performance against goals, consistent with federal internal control standards.
section_title: What GAO Recommends
paragraphs: GAO recommends that NNSA consistently track and document results of DNN R&D projects and document assessments of final project results against baseline performance targets. NNSA neither agreed nor disagreed with the first recommendation and partially agreed with the second, but agreed to take actions in response to both recommendations.

section_title: Letter
paragraphs: The threat posed by the proliferation of nuclear weapons is one of the  most pressing national security challenges facing the United States. To  address this challenge, the National Nuclear Security Administration  (NNSA)—a separately organized agency within the Department of Energy  (DOE)—implements nuclear nonproliferation programs worldwide under  its Office of Defense Nuclear Nonproliferation  (DNN). DNN’s mission is  to develop and implement policy and technical solutions to eliminate  proliferation-sensitive materials and limit or prevent the spread of  materials, technology, and expertise related to nuclear and radiological  weapons and programs around the world. As part of this mission, DNN  supports programs that conduct research and develop technologies for  use in treaty monitoring, safeguarding nuclear materials, and detecting  proliferation-related  activities and nuclear detonations. DNN’s nuclear  nonproliferation research and development (R&D) activities have been  implemented under different program names since the mid-1960s.
We have previously identified challenges affecting NNSA’s nuclear  nonproliferation R&D activities. For example, in August 2002, we found  that NNSA did not have a formal process to identify the needs of  technology users in the U.S. government’s nonproliferation community for  two of the three research areas managed at that time by NNSA’s  Nonproliferation  and Verification R&D Program—the predecessor of  DNN’s current nuclear nonproliferation  R&D program. Additionally,  we  found then that NNSA had inadequate tools to monitor project progress.
NNSA agreed with our recommendations that the agency offer technology  users opportunities to provide input through all phases of R&D projects  and that it upgrade the R&D program’s project management information  system to track progress in meeting projects’ milestones. More recently,  we found in a December 2011 report that some DNN program  performance measures did not include key attributes of successful  measures, such as being clear, reliable, and balanced. For instance, we  found that performance measures for DNN’s Nonproliferation  and  Verification R&D program were unclear because the measures assessed  progress using unspecified criteria and milestones from a classified R&D  requirements document. Because these performance measures were  linked to an unstated secondary set of criteria, we found that a third party  unfamiliar with the classified requirements would have difficulty  interpreting the measures and discerning whether the criteria for  measuring progress were appropriate, sufficient, and up to date. NNSA  neither agreed nor disagreed with our recommendation that the agency  clarify its performance measures for evaluating the Nonproliferation and  Verification R&D program. We discuss this recommendation later in this  report.
The House Armed Services Committee Report 114-102, which  accompanied H.R. 1735, the National Defense Authorization Act for  Fiscal Year 2016, included a provision for us to review NNSA’s  nonproliferation R&D programs. This report (1) evaluates the extent to  which projects managed by NNSA’s research and technology  development programs have resulted in advanced, transitioned, or  deployed nonproliferation technologies, (2) evaluates how NNSA  measures the performance of its research and technology development  programs and projects, and (3) describes how DNN programs decide  which research and technology development projects to pursue.
To evaluate the extent to which NNSA’s research and technology  development projects have resulted in advanced, transitioned, or  deployed nonproliferation technologies, we obtained and analyzed  information on a nongeneralizable sample of 91 research and technology  development projects managed by the two DNN programs responsible for such projects. The information included project data from a program  database. We selected these 91 projects—which were undertaken  between fiscal years 2012 and 2015—for the sample based on funding  level and project technical area, among other factors, to cover a broad  range of program activities. We also interviewed officials in NNSA  headquarters, project managers at DOE and NNSA national laboratories,  and officials in other agencies that are end users of technologies and  other outcomes that result from DNN’s projects. Specifically, we  interviewed officials at the Departments of Defense (DOD), Homeland  Security (DHS), and State, as well as officials at the Preparatory  Commission for the Comprehensive Nuclear-Test-Ban Treaty  Organization (CTBTO) and the International  Atomic Energy Agency  (IAEA). We also reviewed documents that establish program goals and  federal internal control standards.
To evaluate how NNSA measures the performance of its research and  technology development programs and projects, we reviewed externally  reported and internal program– and project-level measures that NNSA  uses to assess performance in DNN’s research and technology  development programs. While we also reviewed internal program-level  measures, we focused our evaluation on external, publicly reported  performance measures. Specifically, we reviewed measures developed  for and reported on in NNSA’s budget materials in response to the GPRA  Modernization Act of 2010 (GPRAMA), which updated the Government  Performance and Results Act of 1993. For project-level measures, we  reviewed documentation on the 91 projects we selected for our  nongeneralizable sample; specifically, we reviewed documents that  establish baseline targets for projects’ scope, cost, and completion date,  as well as documents that contain information about project results. We  also interviewed DNN officials to obtain their views about program and  project management and performance. In addition, we reviewed  documents that establish criteria for successful performance measures  and federal internal control standards.
To describe how DNN decides which projects to pursue, we reviewed  documents that establish the nonproliferation mission needs of DOE,  interagency, and IAEA stakeholders; those that establish program  requirements and project selection criteria; and project proposals. We  also interviewed NNSA and other agency officials and project managers  at the national laboratories to understand the decision-making processes  that DNN uses to define broad program areas and select projects for  funding.
To assess the reliability  of the data we analyzed on the projects in our  nongeneralizable sample, we interviewed NNSA officials and reviewed  documentation, and we determined that the data were sufficiently reliable  for presenting the information contained in this report on NNSA project  characteristics and funding. Additional details on our scope and  methodology can be found in appendix I.
We conducted this performance audit from November 2015 to February  2017 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: Within DNN, two programs manage research and technology  development projects: the DNN Research and Development (DNN R&D)  program and the Nonproliferation and Arms Control (NPAC) program.  These two programs have established program areas to focus their  research and technology development activities.
DNN R&D program areas. Within the DNN R&D program, two  program areas pursue research and technology development  projects: Nuclear Detonation Detection and Proliferation Detection.  The Nuclear Detonation Detection program area develops and  provides global monitoring capabilities for detecting foreign nuclear  weapon detonations, in order to meet both treaty monitoring and  military needs. The Proliferation Detection program area develops  capabilities to detect special nuclear materials (which include  plutonium and highly enriched uranium), to detect weapons  production and movement, and to enhance transparent nuclear  reductions and monitoring. These two program areas each are divided  into three functional areas.
The Nuclear Detonation Detection program area’s three functional
Space-based Nuclear Detonation Detection, which develops  and builds space-based sensors for nuclear explosion  detection;
Ground-based Nuclear Detonation Detection, which produces  and updates modeling and analysis capabilities—such as  software codes and algorithms that help distinguish  underground nuclear detonations from natural seismic  events—and provides other technical support for the nation’s  ground-based nuclear explosion monitoring networks; and
Forensics-based Nuclear Detonation Detection, which  conducts R&D to advance capabilities in the field of nuclear  forensics analysis.
The Proliferation Detection program area’s three functional areas
Nuclear Weaponization and Material  Production Detection,  which supports the development of technology to detect and  characterize the production of nuclear weapons and related  materials by foreign entities;
Nuclear Weapons and Material  Security, which, among other  things, develops tools for nuclear security, treaty monitoring  and verification, and operational interdiction and nuclear  security efforts across NNSA; and
Enabling Capabilities, which develops cross-cutting  technologies applicable to multiple NNSA and interagency  missions.
NPAC program areas. Within the NPAC program, two program areas  pursue technology development projects:  International  Nuclear Safeguards (Safeguards), among other  things, develops technologies to detect and deter undeclared  nuclear materials and activities. Specifically, Safeguards’ activities  include developing and transferring tools, technologies, and  approaches to improve U.S., IAEA,  and IAEA  member states’  capabilities in undertaking IAEA safeguards activities, such as  monitoring uranium enrichment levels in nuclear facilities.
Nuclear Verification (Verification), among other things, develops  and deploys technologies to maintain the United States’ ability to  monitor and verify nuclear reduction agreements, detect treaty  violations, and verify other nuclear nonproliferation commitments.  The goals of the Verification program area include developing  technologies to verifiably and irreversibly disable nuclear facilities  in countries of concern.
Figure 1 illustrates the structure of the DNN programs and functional  areas that manage research and technology development projects.
From fiscal year 2012 through fiscal year 2015, DNN R&D and NPAC  obligated a total of more than $1.1 billion on 511 research and technology  development projects. The DNN R&D program obligated more than $1  billion on 420 projects, and the NPAC program obligated about $73  million on 91 projects. According to NNSA officials we interviewed, the  Space-based Nuclear Detonation Detection functional area expenditures  were mostly for production activities that resulted in technologies  deployed to end users, rather than for basic and applied R&D, in contrast  with the other DNN R&D program functional areas. DNN R&D and  NPAC projects are undertaken at DOE and NNSA national laboratories  and sites; these projects generally last about 3 years from beginning to  end, according to program officials. Table 1 shows the number of projects  undertaken and funding obligated in each program and functional area  from fiscal years 2012 through 2015.
According to NNSA officials, the end users of technologies resulting from  DNN’s research and technology development programs include DOD,  including its Defense Threat Reduction Agency, U.S. Strategic Command,  and Air Force Technical Applications Center; DHS; State; IAEA and other  international organizations, including the CTBTO; international  governments; universities; and private industry, including small business.  NNSA itself also uses some of the technologies that result from DNN’s  research and technology development programs.
section_title: NNSA Does Not Consistently Track and Document the Full Extent of Nonproliferation Technology Advancement, Transition, and Deployment
paragraphs: The full extent to which research and technology development projects  managed by NNSA’s DNN R&D and NPAC programs have resulted in  advanced, transitioned, or deployed technologies is unclear because  NNSA does not consistently track and document all of these project  outcomes. However, by reviewing a range of information on a  nongeneralizable sample of 91 DNN research and technology  development projects, as well as interviewing end users of technologies  that resulted from some of the projects, we were able to determine the  outcomes of these projects. More specifically, we found that 88 of the 91  projects in the sample resulted in technologies being advanced—that is,  the project progressed the technology itself or the scientific knowledge  behind the technology. Additionally,  we found that, among these 88  projects, 33 resulted in technologies being transitioned—that is, provided  to users outside of the project team for further development or  deployment. Finally, we found that 17 of these 33 projects resulted in  deployed technologies—that is, we were able to confirm that a technology  was being actively used in the field by a federal agency or foreign partner.
subsections: section_title: NNSA Tracks and Documents Nonproliferation Technology Advancement, but Does Not Do So Consistently for Technology Transition and Deployment
paragraphs: The full extent to which NNSA’s research and technology development  projects result in advanced, transitioned, or deployed technologies is  unclear because NNSA does not consistently track and document all of  these project outcomes. Both the DNN R&D and NPAC programs track  and document projects’ results in advancing technologies. For instance,  the DNN R&D program uses an online system, WebPMIS, to track project  costs and maintain an archive of project-related documentation. This  documentation includes project reports that provide information on the  scientific and technological advancements that result from a project.  Similarly, the NPAC program provided documentation of technology  advancement for our review. However, the DNN R&D program, which is  by far the larger program, does not consistently track and document  projects that result in technologies being transitioned or deployed. NNSA  officials acknowledged that the DNN R&D program does not use the  WebPMIS system, or otherwise collect and maintain information, to track  and document whether its projects result in technologies that have been  transitioned to or deployed by end users. As a result, the DNN R&D  program was not able to readily provide us with information on the extent  to which its projects result in transitioned or deployed technologies. In  contrast, we found that the NPAC program tracks and documents the  extent to which its safeguards– and verification-related projects have  resulted in technology transition or deployment, as well as tracking  scientific and technological advancements. For example, NNSA officials  provided us with information on all NPAC projects for the past 5 fiscal  years that showed whether technologies that projects developed have  been deployed; they also provided documentation of technologies that  have been transitioned to end users. Both the DNN R&D and NPAC  programs list technology transition—which includes providing  technologies to end users, who may then deploy them—as a goal in  strategic planning documents. In addition, NPAC program officials told us  that their program evaluates its success in part based on the extent to  which end users deploy technologies that the NPAC program develops.  Similarly, DNN R&D program officials said that meeting end users’  technology needs constitutes an important part of their program mission.  Moreover, under federal internal control standards, management should  design control activities to achieve objectives and respond to risks, and  should record, communicate, and use quality information to enable  managers to carry out internal control responsibilities and evaluate  program performance in achieving key objectives.
Differing program emphases—both between the DNN R&D and NPAC  programs and among the DNN R&D program’s functional areas—may  account for the inconsistencies we found in NNSA’s tracking and  documentation of transition– and deployment-related project outcomes.  For instance, the NPAC program generally focuses on developing  technologies at higher readiness levels to prepare them for transition and  deployment in the field, according to NNSA officials. Conversely, most  of the DNN R&D program’s functional areas conduct R&D on  technologies that are further from being deployment-ready because they  require more basic work at lower technology readiness levels, with some  exceptions, NNSA officials told us. Notably, the DNN R&D program’s  Space-based Nuclear Detonation Detection functional area has a history,  dating to the 1960s, of researching, developing, and producing  successive upgrades to technologies used by the U.S. Atomic Energy  Detection System and its space-based element, the U.S. Nuclear  Detonation Detection System, to monitor and detect nuclear detonations,  NNSA officials stated. Officials in the DNN R&D program office said that  well-defined DOD technology requirements and delivery schedules drive  these research, development, and production activities; as a result, the  Space-based Nuclear Detonation Detection functional area has a  stronger focus on technology production and deployment than the other  DNN R&D program functional areas, which typically do not have similarly  specific requirements and schedules. relevant data from reliable internal and external sources, processing the data into quality  information, and then considering the data to make informed decisions and evaluate  program performance. providing end users with tools or knowledge that are sufficiently ready  for the end user to take the final steps necessary for the technology’s  use in mission-specific purposes, such as adapting a detection  algorithm or producing a field-ready detector array.
It can be difficult to trace a link between a project and a given result— for example, when another agency builds on a project’s work; when a  discrete project does not lead directly to an outcome, but results in a  “spin-off” project or is part of a related portfolio of projects that does  lead to the desired outcome; or when there is a significant period of  time between the end of a project and the deployment of any resulting  technology.
Project managers at DOE and NNSA national laboratories know the  outcomes of projects and can provide information to NNSA officials  when needed.
Information about some nonproliferation technologies is classified,  with the result that NNSA is not informed about their uses and users.
These factors notwithstanding, by not consistently tracking and  documenting all relevant outcomes of projects across DNN’s research  and technology development program areas, NNSA is unable to  demonstrate to Congress and the public the full extent to which its  projects result in technologies that are transitioned, deployed, or  otherwise used toward fulfilling the nation’s nuclear nonproliferation goals.  Moreover, not maintaining documentary information about project results  may affect control activities by limiting the quality of information that  NNSA has available to evaluate project performance; it also may reduce  opportunities for sharing knowledge, and may make DNN vulnerable to  the loss of institutional memory when personnel involved with a project  retire or leave the agency.
section_title: Selected Projects Resulted in Advanced, Transitioned, and Deployed Technologies
paragraphs: By gathering information from various sources about our sample of 91  selected DNN research and technology development projects, we were  able to determine that some projects resulted in technologies being  advanced, transitioned, or deployed, or that they resulted in more than  one of these outcomes. More specifically, we found that nearly all of the  projects we selected (88 of 91) advanced the technology itself or the  scientific knowledge behind the technology. Of those 88 projects, we  found that 33 also resulted in the advanced technology being transitioned  to users outside of the project team for further development or  deployment. Among the 33 projects that resulted in a transitioned  technology, 17 also resulted in active technology deployment—that is,  field use by a federal agency or foreign partner. Figure 2 below illustrates  these results.
Almost all of the selected projects in our sample (88 of 91) advanced  technologies in some way. Examples of projects that advanced  technologies include projects in which NNSA scientists built instrument  hardware, made progress in developing new technology combinations or  applications, developed an algorithm or model to further data analysis, or  made scientific advances that improved nonproliferation analysis and  operations.
Some of the projects in our sample that have resulted in technologies  being advanced have not resulted in technologies being transitioned to  end users (55 of 88 such projects). According to NNSA officials and  scientists we interviewed, technology transition has not occurred on these  projects for a variety of reasons, including:
The technology is not developed enough to be transitioned. For  example, a DNN R&D Proliferation Detection project developed a  technology to provide new capabilities in imaging gamma rays. The  imager is a laboratory prototype and needs further development,  according to a scientist we interviewed at Oak Ridge National  Laboratory.
The project has not connected with an end user who wants to receive  the technology. For example, a DNN R&D Proliferation Detection  project developed a hand-held fast-neutron generator, but NNSA has  yet to find an end user interested in the technology, according to the  manager of the Nuclear Threat Reduction Program at Lawrence  Livermore National Laboratory.
The project developed a technology to meet a future nonproliferation  need rather than an existing requirement. For example, a project  within the NPAC Verification program developed a fast-neutron  imaging technology that could be used in stockpile warhead  verification, should a future arms control agreement call for warhead  inspections. The technology currently is not being used in a  verification context, but NNSA officials said it is in use at NNSA sites  to compile benchmark data on nuclear weapons; these data may later  be used in verification and other national security missions.
subsections: section_title: Thirty-three of the 91 Projects in Our Sample Resulted in Transitioned Technologies
paragraphs: We determined that 33 of the 91 projects in our sample resulted in  technologies that were transitioned to end users. These 33 projects  include 17 projects that resulted in deployed technologies, since a  technology is transitioned to an end user before it is deployed. For the  remaining 16 projects, we found that either the technologies had not been  deployed by an end user (13 of the 16) or we could not confirm with end  users that the technologies had been deployed (3 of the 16). Specifically:
We counted 13 of the 16 projects and their resulting technologies as  transitioned but not deployed because the end users need to take  certain steps before the technologies can be deployed, according to  NNSA and end user agency officials. For example, one DNN R&D  Nuclear Detonation Detection project transitioned an instrument to a  DOD end user. However, the instrument has not been deployed  because DOD needs to find funds to further develop the technology,  which, according to DOD officials, was at a low level of technology  readiness when transitioned. In another case, a DNN R&D Ground- based Nuclear Detonation Detection project transitioned software to a  DOD end user, but the user is still evaluating the software before  integrating it into its systems.
For the remaining 3 of the 16 projects, we could not confirm whether  technologies that resulted from the projects and were transitioned to  end users are being used.
section_title: Seventeen of the 91 Projects in Our Sample Resulted in Deployed Technologies
paragraphs: We were able to confirm that 17 of the 91 projects in our sample resulted  in actively deployed technologies. More specifically, we found that 13 of  the 63 DNN R&D projects and 4 of the 28 NPAC projects in our sample  resulted in deployed technologies.
Examples of the projects that resulted in deployed technologies include:
An NPAC Safeguards project that developed an online enrichment  monitor (OLEM) for use in uranium enrichment monitoring. As we  reported in June 2016, the IAEA  is using the OLEM in the Natanz Fuel  Enrichment Plant in Iran to confirm that enrichment levels are at or  below 3.67 percent, per Iran’s commitment under the agreement  known as the Joint Comprehensive Plan of Action (JCPOA). IAEA  has previously used enrichment monitors, but the OLEM is a newer  technology that improves upon older monitoring systems, as we found  in our June 2016 report.
DNN R&D Space-based Nuclear Detonation Detection projects that  resulted in global burst detection sensors that are in use by the Air  Force on satellites to detect, identify, and precisely locate nuclear  explosions.
DNN R&D Ground-based Nuclear Detonation Detection projects that  resulted in enhanced understanding of waves produced by  underground nuclear explosions. The Air Force, the CTBTO, and  international scientists analyze these waves to locate underground  nuclear explosions and establish their yield.
Of the 17 projects that resulted in deployed technology, 9 were active or  ongoing projects and 8 were retired or in closeout, meaning that all of the  research work was completed. Similarly, of the 33 projects that resulted  in transitioned technology, 15 were active or ongoing projects and 18  were retired or in closeout. Projects resulting in deployed and transitioned  technologies can continue to be active for the following reasons:  In some cases, project officials provide maintenance and support to  the technology after it has been deployed. For example, the OLEM  unit has been deployed, but the project is still active because NNSA is  continuing to address end user maintenance needs, according to  agency officials.
In other cases, the project has deployed or transitioned a technology  but has yet to accomplish research goals. For example, a Nuclear  Detonation Detection project has deployed a technology to a DOD  end user, but research continues because the project has not met all  of the research objectives.
See appendix II for a list of the projects in our sample that resulted in  deployed technologies. Appendix III  provides information on the number  of advanced, transitioned, and deployed technologies that resulted from  the 91 projects we selected for our sample, subdivided by DNN program  and functional areas.
section_title: NNSA’s Publicly Reported Program Measures Lack Clarity, and Improved Project-level Documentation Could Enhance Performance Assessment
paragraphs: NNSA’s DNN R&D and NPAC programs use publicly reported measures  to assess program-level performance, and both programs review project  documentation and communicate with project managers to assess  project-level performance. Clarity limitations with the DNN R&D program’s  publicly reported measures, however, make them difficult to interpret, and  improvements in NNSA’s final project documentation could enhance  assessment of project performance.
subsections: section_title: Clarity Limitations May Hinder Users’ Ability to Interpret DNN’s Publicly Reported Performance Measures
paragraphs: NNSA reports publicly on the DNN R&D and NPAC programs’  performance using measures published in DOE’s annual budget  requests, in response to requirements in GPRAMA;  however, limitations  with the clarity of the DNN R&D program’s measures may make it difficult  for users, such as Congress, to understand the targets that NNSA has  established for its research and technology development programs, as  well as how NNSA has measured performance against these targets.  The NNSA budget materials we reviewed present a total of six GPRAMA  measures associated with the DNN R&D and NPAC programs.  Specifically, the DNN R&D program reports on five GPRAMA measures  covering both the Nuclear Detonation Detection and Proliferation  Detection program areas. The NPAC program reports on one measure for  its Safeguards program area, but does not have a GPRAMA  measure for  its Verification program area. The six GPRAMA measures identify  specific performance timeframes and endpoint targets, and the publicly  reported information summarizes NNSA’s assessment of the programs’  performance against the targets. Appendix  IV shows the NPAC and DNN  R&D performance measures as presented in NNSA’s budget request for  fiscal year 2017, including endpoint targets and NNSA’s assessment of  the programs’ performance against these targets.
We found that the DNN R&D program’s performance measures are  unclear because the program does not define targets or explain its  assessments of performance against the targets in sufficient context to  allow users to interpret the measures or performance assessments. For  example, regarding three of the DNN R&D program’s measures:
The Nuclear Detonation Detection measure tracks percentage of  progress against an annual index summarizing the status of all NNSA  Nuclear Detonation Detection R&D deliveries that improve the  nation’s ability to detect nuclear detonations.
The Nuclear Weapons and Material Security measure tracks  cumulative percentage of progress toward demonstrating  improvements in special nuclear material detection, warhead  monitoring, chain-of-custody monitoring, safeguards, and  characterization capabilities.
The Nuclear Weaponization and Material Production Detection  measure tracks cumulative percentage of progress toward  demonstrating improvements in detection and characterization of  nuclear weapons production activities.
For these three measures, the baseline criteria for assessing  performance—the annual index and the cumulative percentages of  progress—are not defined, and no justification is presented to clarify how  NNSA concluded that it had met its performance targets. Without such  context, the measures provide statements of NNSA’s assessment of their  own performance, but do not provide users with information they can use  to evaluate whether NNSA’s assessments are valid and justified.
Previous GAO reports have identified key attributes of successful  performance measures, including that they be clearly stated to enable  entities to assess their own performance and to enable stakeholders to  determine whether a program is achieving its goals. In addition, our  December 2011 report on NNSA program management and coordination  challenges in the area of nuclear nonproliferation identified concerns with  the clarity of performance measures used by the DNN programs we  review in this report. Specifically, we found that several of the DNN R&D  program’s performance measures were linked to secondary criteria that  are classified, official use only, or otherwise not publicly stated, making it  difficult for third-party users to interpret the measures and discern  whether the criteria are appropriate, sufficient, and up to date for tracking  purposes. We recommended in our 2011 report that NNSA clarify these  measures. NNSA neither agreed nor disagreed with this  recommendation.
NNSA officials told us, when we interviewed them for this report, that they  do not agree that the DNN R&D program’s performance measures are  unclear, for two reasons.
First, according to NNSA officials, NNSA provides supplemental  information for Office of Management and Budget (OMB) users of the  performance information that explains the targets and how  performance is measured. Specifically, NNSA officials told us that the  DNN R&D program provides additional documentation to OMB— some of which the officials also provided to us for review—that  includes information explaining the progress against annual targets  under each of the program’s performance measures. However, most  of the information that NNSA provided on the DNN R&D program is  classified or official use only.
Second, an NNSA official told us that the DNN R&D program  performance measures are not unclear because the measures are  used primarily to inform internal program managers who are familiar  with classified requirements against which progress is assessed,  rather than to provide information to third-party users. However, there  may be potential external users of the performance information—such  as congressional decision makers and other external parties—who  are not familiar with the baseline requirements and may have difficulty  interpreting the level of performance achieved by the DNN R&D  program.
In the current review, the clarity issues we found with the DNN R&D  program’s performance measures were similar to those we identified in  December 2011. We continue to believe that NNSA should clarify these  publicly reported measures, so that users of the performance information—including Congress and the public—have a sufficient basis  to judge the programs’ performance in meeting the United States’ nuclear  nonproliferation technology needs.
section_title: Better Documentation Could Enhance NNSA’s Assessment of Project Performance
paragraphs: NNSA assesses the performance of the DNN R&D and NPAC programs’  projects by reviewing project documentation and communicating with  project managers, but improvements in NNSA’s final project  documentation could enhance assessment of project performance. To  assess project performance, NNSA officials said that both programs track  progress during the project by communicating with project managers at  the national laboratories and by reviewing documents, such as quarterly  project reports, that provide information on project performance.  According to NNSA documentation we reviewed and NNSA officials we  interviewed, the DNN R&D and NPAC programs establish baseline  targets for projects’ scope of work and completion date in project plans. In  some cases, initial project plans are updated—for example, to incorporate  new or modified project objectives. At the end of the project, project  managers at the national laboratories produce final project reports for  program managers to review.
NNSA officials stated that these final project reports describe a project’s  technical results. However, the officials said that the final reports do not  document their assessment of project performance against the baseline  targets established in the initial project plans. They also said that there is  no common template for final project reports. We confirmed this in our  review of several final project reports.
NNSA officials provided several reasons why they do not document their  assessment of project performance against baseline targets or use a  common template for final project reports. For example, according to  NNSA officials:
Program managers communicate with project managers at the  laboratories to ensure that projects are proceeding as planned.
Restating the baseline targets in final project reports would be  duplicative of information contained in other sources, such as the  WebPMIS database.
The varying formats of the laboratories’ final reports do not affect  program managers’ ability to understand the technical outcomes of  the projects.
According to NNSA officials, initial baselines may change through  “progressive baselining”—which entails changing the initial project  plans when discoveries warrant or course corrections are needed,  subject to a defined change control process and the approval of the  federal program manager.
Federal internal control standards specify that managers should use  quality information to make informed decisions and evaluate  performance; that managers need to compare actual performance to  planned or expected results; and that entities should record and  communicate information to managers who need it to carry out their  internal control responsibilities. Moreover,  the standards specify that  program managers need data to determine whether their programs are  meeting their goals for accountability for effective and efficient use of  resources. Documenting assessments that compare final project  performance results against baseline targets for scope of work and  completion date—whether through a common template for final project  reports or by other means—could enhance NNSA’s ability to manage its  programs in accordance with these standards.
section_title: DNN Programs Use Similar Five-step Processes to Select Research and Technology Development Projects
paragraphs: Both the DNN R&D and NPAC programs use similar five-step processes  to decide which research and technology development projects to pursue.  According to NNSA officials and stakeholders we interviewed at  technology end user agencies, NNSA collaborates throughout the  decision-making and project selection process with the end users. In our  review of program documents and interviews with agency officials, we  found that the DNN R&D and NPAC programs generally take the  following steps in their project selection processes:  Step 1: DNN establishes project requirements. According to NNSA  officials, DNN R&D and NPAC project requirements flow from national  policy documents, such as the Nuclear Posture Review; additionally,  NPAC project requirements flow from international sources such as the  Treaty on the Non-Proliferation of Nuclear Weapons (NPT),  Comprehensive Safeguards Agreements; Additional  Protocols; and the  IAEA  Department of Safeguards Long Term R&D Plan. To identify the  technical capabilities that are required to meet the requirements  established in these documents and sources, NNSA officials develop  more detailed plans, such as the DNN R&D program’s goals, objectives,  and requirements (GOR) documents. The GOR documents specify  requirements for five of the DNN R&D program’s six functional areas—all  but the cross-cutting Enabling Capabilities functional area. Additionally,  each GOR document has a corresponding technology roadmap, which  establishes detailed technical requirements to meet the needs identified  in the GOR document. Similarly, the NPAC program develops needs  documents that identify specific needs and corresponding capabilities to  develop; NPAC uses these documents to guide the proposal selection  process. NNSA consults with interagency stakeholders to inform the  requirements specified in the GOR documents and needs documents.
Step 2: The DNN programs issue annual calls for proposals. Both  programs issue annual calls for proposals, which specify the technical  requirements proposals should address. The programs also require that  those who submit proposals clearly explain how the proposed project  relates to identified project requirements. According to NNSA officials, the  DNN R&D program typically issues its calls for proposals in mid-October and NPAC issues its calls for proposals during the spring. The DNN  R&D program’s calls for proposals specify weighted criteria against which  proposals are reviewed, including mission relevance, scientific and  technical merit, and budget. The NPAC program’s calls for proposals  differ between its two program areas. The Safeguards program area  provides explicit evaluation criteria including cost effectiveness, technical  maturity, and deployment potential. Calls for proposals in the Verification  program area do not provide explicit evaluation criteria, but they provide  information on program goals and areas of technical consideration for the  current budget year. They also require applicants to describe the  proposed work, including activities, milestones, and deliverables.
Step 3: National laboratories submit informal project proposals. Scientists  at the national laboratories respond to the call for proposals by submitting  informal project proposals, known as white papers, to the NNSA program  offices. The white papers describe proposed projects to demonstrate new  capabilities or technologies sought in the calls for proposals. As specified  in the DNN R&D program’s calls for proposals, the program uses a three- tiered system to rank proposals—specifically, by designating proposals as  hot, warm, or cold—and provides comments to improve the proposals or  future white paper submissions. According to an NNSA official, the DNN  R&D program returns the informal project proposals to the original  submitters. All hot and warm proposals may be amended in response to  program officials’ comments and then resubmitted in the next step of the  process as formal project proposals. NNSA officials stated that cold  proposals receive no further consideration. NNSA officials within the  NPAC program stated that they use a ranking system that is similar to the  DNN R&D program’s system; the officials also provided documentation  that shows their rationale for selecting or not selecting projects for further  consideration. Both programs seek to provide feedback within a general  time frame. The DNN R&D program aims to provide feedback within 2  weeks of receiving the white papers. According to NNSA officials, NPAC’s  Safeguards program area provides feedback on the white papers about 3  to 3 1/2 months from the date of submission, and the Verification program  area provides feedback about 2 1/2 months from the date of submission.  According to NNSA officials and interagency stakeholders we  interviewed, interagency stakeholders are consulted during this step of  the process, as appropriate.
Step 4: Laboratories submit formal project proposals. Scientists at the  national laboratories amend hot and warm white papers and resubmit  them as formal proposals for NNSA’s consideration.
Step 5: DNN selects projects in consultation with laboratories and end  users. NNSA officials identify the formal proposals that best meet the  criteria and requirements specified in the calls for proposals, considering  input provided by laboratory officials, interagency stakeholders, and  potential end users on each proposal. According to NNSA officials, after  discussing budgetary considerations and current, prioritized needs with  interagency stakeholders and potential end users, program officials  develop a final, ranked list of projects selected to receive funding and  support. Program officials then circulate this final list to interagency  stakeholders and potential end users.
Figure 3 shows the general five-step selection process used by DNN’s  programs.
section_title: Conclusions
paragraphs: NNSA’s research and technology development programs make vital  contributions to national security, and the projects they support address a  range of important proliferation detection and monitoring needs, including  improving abilities to detect covert nuclear material production, nuclear  explosions, and potential violations of IAEA  safeguards agreements. It is  not clear, however, how often NNSA’s research and technology  development projects—especially those supported by the DNN R&D  program—result in transitioned or deployed technologies because NNSA  does not consistently track and document such information. We  acknowledge that DNN’s research and technology development programs  pursue a broad range of scientific tools and capabilities, with the goal of  ensuring that the United States remains prepared to deal with  unanticipated nonproliferation  challenges as they emerge. Therefore, we  recognize that not all R&D projects, such as projects conducted at low  levels of technology readiness, are intended to result in a deployed  technology, and that a higher deployment rate does not necessarily  indicate a more successful R&D program, or vice versa. Moreover, we  recognize that in cases where technology transition and deployment are  project goals, certain challenges complicate DNN’s efforts to track the  deployed technologies that result from its projects. Nevertheless, we were  able, as part of our review of selected projects, to track projects’ end  results, including end users’ deployment of technologies that result from  the projects. Moreover, not tracking such end results may affect control  activities by limiting the quality of information that NNSA has available to  evaluate project performance. More consistently tracking and  documenting the transitioned and deployed technologies that result from  its projects would help ensure that NNSA maintains and uses quality  information to evaluate its performance and achieve its objectives, in  keeping with federal internal control standards. Doing so would also  facilitate knowledge sharing within DNN, and it would provide a means by  which to present valuable information to Congress and other decision  makers about the programs’ results and their overall value.
In addition, better tracking of project results by the DNN R&D program  may have the salutary benefit of providing information to the program that  could allow it to develop clearer program performance measures, as we  recommended in December 2011. The DNN R&D program’s  performance measures continue to have clarity limitations similar to the  ones we identified in the December 2011 report, because understanding  the publicly reported performance information depends on users of the  information having access to or being familiar with criteria in official use  only or classified documents. We continue to believe that it is important  for NNSA to implement our December 2011 recommendation that NNSA  clarify the DNN R&D program’s publicly reported performance measures,  so that Congress and other users of these measures have a sufficient  understanding of DNN programs’ status and progress.
By documenting assessments that compare the final results of a project  against the baseline targets for scope of work and completion date that  are established in initial plans for each project, NNSA could improve the  quality of the information it needs to evaluate performance at the end of  the project. Specifically, taking these steps could enhance managers’  ability to use quality data to assess actual project performance against  planned or expected results, consistent with federal internal control  standards. Such assessments are essential to ensure that the DNN R&D  and NPAC programs’ research and technology development projects are  effectively and efficiently using resources, another key aspect of federal  internal control standards. If changes to the initial baselines are  necessary—which may be the case in some instances—we believe that  they should be made in such a way that initial project targets are still  documented and available for review in evaluating the full progression of  a project.
section_title: Recommendations for Executive Action
paragraphs: We recommend that the NNSA Administrator take these two actions:
Direct the DNN R&D program to track and document the transitioned  and deployed technologies that result from its research and  technology development projects, to the extent practicable.
Direct the DNN R&D and NPAC programs to document, using a  common template or other means, their assessment that compares  the final results of each project against the baseline targets  established in each project’s initial project plan.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided drafts of this report to NNSA, DOD, DHS, and State for  review and comment. In emails, DOD, DHS, and State stated that they  had no comments on the report. In NNSA’s written comments, which are  summarized below and reproduced in appendix V, NNSA neither agreed  nor disagreed with our first recommendation and partially agreed with our  second recommendation; however, NNSA stated that it plans to take  actions in response to both recommendations. NNSA also provided  technical comments, which we incorporated as appropriate.
In its written comments, NNSA did not state whether it agreed or  disagreed with the recommendation that, to the extent practicable, the  DNN R&D program track and document transitioned and deployed  technologies resulting from the program’s projects, but it described  planned actions consistent with the recommendation. NNSA stated that  our report did not clearly or fully disclose reasons that the DNN R&D  program does not track and document transition and deployment of  technologies resulting from its projects. Specifically, NNSA commented  that the DNN R&D program’s space-based activities—unlike the other  DNN R&D program areas—are production and deployment oriented, and  that it would therefore be misleading to track individual projects in this  area that are transitioned and deployed. Regarding the other DNN R&D  program areas, NNSA commented that it does not have ongoing insight  into transition and deployment outcomes for various reasons—for  example, due to classification barriers regarding the transition and  deployment of certain technologies. NNSA also expressed concern that  information on such outcomes may not be readily available or may not be  cost-effective to obtain. Our report discusses a number of reasons why  the agency may not consistently track the deployment– and transition- related outcomes of the DNN R&D program’s projects and notes the  production and deployment orientation of the space-based project  portfolio. However, we disagree that tracking the outcomes of these  projects would be misleading. To the contrary, it is vital to track these  outcomes, as the space-based program area receives the largest share  of the DNN R&D program’s funding. Moreover, we believe that  information on project outcomes should be readily available to NNSA  program officials in the course of their regular interactions with the project  managers at the national laboratories and the interagency end users of  nonproliferation technologies—as was the case with projects in our  sample. NNSA agreed that, where such information is readily available,  reporting of transitioned and deployed technologies can provide  information on program successes, and it identified actions it will take  consistent with the recommendation. Specifically, NNSA stated it will  complete an assessment by June 2017 of the DNN R&D program’s  portfolio to determine to what extent transition and deployment data are  readily available for specific projects and include additional information on  those projects as part of its performance reporting. If implemented as  planned, such an assessment would be a step toward ensuring that  NNSA maintains and uses quality information to evaluate its performance  and achieve its objectives, in keeping with federal internal control  standards.
NNSA stated that it agrees in part with our second recommendation—that  NNSA document, using a common template or other means, its  assessment that compares the final results of each DNN R&D and NPAC  project against the baseline targets established in each project’s initial  project plan. NNSA stated that it supports implementation of a common  template for documenting project closeout to provide consistency in  information reporting, but it noted that R&D projects are dissimilar to  acquisition activities and that comparing final project results to an initial  project baseline would be misleading. Specifically, for exploratory R&D  projects, NNSA commented that project outcomes can be “significantly  indeterminate” at the beginning of a project and stated that it uses an  approach known as “progressive baselining” to adjust plans and  benchmarks as research matures, where appropriate. NNSA stated that  final project results should be compared to the final approved plans to  ensure consistency in project documentation and to provide a meaningful  assessment of project performance. NNSA agreed that information on the  project’s initial baseline is important to understand the progression of the  project. We acknowledge that adjustments to initial project plans may be  appropriate in some cases, provided that the initial plans remain available  and transparent and that decisions behind such revisions are well- documented, consistent with federal internal control standards. NNSA  stated it will establish a common template by June 2017 and use it to  compare projects’ final results to the final approved plans. It also stated  that the template will require a brief description of significant changes  between the initial and final approved project plans. If implemented as  planned, the template could help NNSA to ensure that the DNN R&D and  NPAC programs’ research and technology development projects are  effectively and efficiently using resources, in keeping with federal internal  control standards.
We are sending copies of this report to the appropriate congressional  committees; the Administrator of NNSA; the Secretaries of Defense,  Energy, Homeland Security, and State; and other interested parties. In  addition, the report is available at no charge on the GAO website at  http://www.gao.gov.
If you or your staff members have questions about this report, please  contact me at (202) 512-3841 or oakleys@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. GAO staff who made key contributions to this  report are listed in appendix VI.
section_title: Appendix I: Objectives, Scope and Methodology
paragraphs: This report (1) evaluates the extent to which projects managed by the  National Nuclear Security Administration’s (NNSA) research and  technology development programs have resulted in advanced,  transitioned, or deployed nonproliferation technologies, (2) evaluates how  NNSA measures the performance of its research and technology  development programs and projects, and (3) describes how programs in  NNSA’s Office of Defense Nuclear Nonproliferation (DNN) decide which  research and technology development projects to pursue.
To determine the extent to which NNSA’s research and technology  development programs have resulted in advanced, transitioned, or  deployed nonproliferation technologies, we selected a nongeneralizable  sample from research and technology development projects undertaken  between fiscal years 2012 and 2015 by the two DNN programs  responsible for such projects: the Office of DNN Research and  Development (DNN R&D) and the Office of Nonproliferation  and Arms  Control (NPAC). As detailed below, we selected the projects in the  sample based on funding level and project technical area of focus and  status in order to cover a broad range of program activities.
In total, the DNN R&D and NPAC programs provided information on 511  projects. Specifically, the DNN R&D program provided us with a list of  420 projects executed between fiscal years 2012 and 2015, and the two  NPAC program areas that manage technology development projects— International  Nuclear Safeguards (NPAC Safeguards) and Nuclear  Verification (NPAC Verification)—provided  us with a list of 91 projects that  were executed between fiscal years 2012 and 2015.
From this universe, we developed selection criteria to include projects  from five of the major Department of Energy (DOE) and NNSA  laboratories—Los Alamos National Laboratory, Oak Ridge National  Laboratory, Sandia National Laboratories, Lawrence Livermore National  Laboratory, and Pacific Northwest National Laboratory. Our selection  criteria ensured variations in program areas and project outcomes. We  generally limited our selection to the top 25 percent of projects in terms of  total funding between fiscal years 2012 and 2015, and excluded all  projects that were not led by one of the five national laboratories listed  above. We also excluded projects in certain categories, such as projects  under the program directors and university accounts. After these  exclusions, 134 projects remained in the sample.
The next step in our selection process was to group together similar  projects, based on project status, focus area or type, age, and cost. We  generally selected 2 projects from each group, to ensure that each  combination of these characteristics was represented in our final sample.  In cases in which a group contained only one project, we included only  that project.
For the DNN R&D program, we treated projects with the same  laboratory, focus area, and status as a single group, resulting in 40  groups. For each group, we generally chose the 2 projects with the  lowest project number (signifying the earliest start date), resulting in  the inclusion of 63 projects.
For the NPAC Safeguards program area, we treated projects with the  same technology category (comparable to focus area) and project  status as a single group, resulting in 13 groups. Because we did not  have data to determine earliest start date, we generally chose the 2  projects with the highest total expenditures from each group, resulting  in the inclusion of 17 projects.
For the NPAC Verification program area, we treated projects with the  same R&D category (comparable to focus area) and project status as  a single group, resulting in 8 groups. Because we did not have data to  determine earliest start date, we chose the 2 projects with the highest  total expenditures from each group, resulting in the inclusion of 11  projects.
This resulted in a final sample of 91 projects covering the DNN R&D and  NPAC programs and all of their program and functional areas. Table 2,  below, shows the number of projects we selected from each program and  functional area.
To determine the extent to which the 91 projects in our sample resulted in  advanced, transitioned, or deployed technologies, we gathered project  information from interviews and documentary sources. Specifically, we  interviewed project officials, including the project manager, at the national  laboratory that was the lead on the project, as well as interagency officials  that received technologies transitioned from the DNN R&D and NPAC  programs. Our interviews included officials from the Departments of  Defense (DOD), Homeland Security, and State; the International  Atomic  Energy Agency (IAEA); and the Preparatory Commission for the  Comprehensive Test Ban Treaty Organization. We also reviewed  program and project data and documentation, including basic project  data—such as project names and funding information—stored in DNN  R&D’s WebPMIS database; program funding information; project plans;  annual, quarterly, and final reports; and research presentations and  publications. We reviewed the evidence to determine whether each  project in our sample resulted in advanced, transitioned, or deployed  technology or other outcomes. The outcomes that we identified are not  mutually exclusive for any project. For example, one project could  transition a technology to an end user and also result in a publication, or a  project may have resulted in a research outcome that we did not identify  because the outcome was not indicated in the evidence we collected.  Finally, we reviewed documents that establish program goals, including  strategic planning documents, as well as Standards for Internal Control in  the Federal Government, which establishes federal standards for the use  of information in achieving an entity’s objectives.
To evaluate how NNSA measures the performance of its research and  technology development programs and projects, we reviewed externally  reported and internal program– and project-level measures that NNSA  uses to assess performance in DNN’s research and technology  development efforts. For program-level measures, we focused our review  on the publicly reported measures developed and reported on in NNSA’s  budget materials in response to the GPRA Modernization Act of 2010  (GPRAMA). Specifically, we reviewed the GPRAMA  measures for the  DNN R&D and NPAC programs that NNSA presented in its budget  materials for fiscal years 2015 through 2017. We also reviewed classified  and official use only materials that, according to NNSA officials, NNSA  shares with the Office of Management and Budget (OMB) in their  discussions of the GPRAMA measures. The materials we reviewed  included technology roadmaps, which define the technology pathways  that NNSA follows to address portfolio requirements and develop funding  priorities, and final reports that NNSA presents to OMB on the completion  of activities it undertakes in connection with the GPRAMA  measures. We  also reviewed DOE’s Annual Performance Reports for fiscal years 2014  and 2015. In addition, we reviewed documents and interviewed NNSA  officials regarding internal measures that DNN R&D and NPAC use to  monitor program performance, including independent project performance  reviews. To further evaluate DNN’s project-level performance measures,  we reviewed information on the 91 selected projects in our  nongeneralizable sample. Specifically, we reviewed documents that  establish baseline targets for projects’ scope, cost, and completion date  and documents that contain information on project results. We also  interviewed DNN officials to obtain their views on program and project  management and performance. Finally, we reviewed documents that  establish criteria for successful performance measures, including our past  reports on the key attributes of such measures, as well as federal internal  control standards.
To describe how DNN decides which projects to pursue, we reviewed  documents that establish the nonproliferation mission needs of DOE and  interagency and IAEA  stakeholders in nonproliferation-related  research  and technology development. The documents we reviewed include DOD’s 2010 Nuclear Posture Review Report; the Executive Office of the  President’s National Science and Technology Council’s Nuclear Defense  Research and Development Roadmap, Fiscal Years 2013-2017; the  White House’s National Security Strategy of February 2015; DOE and  NNSA strategic plans; the IAEA  Department of Safeguards’ Long-Term  R&D Plan, 2012-2023; and IAEA’s  Development and Implementation  Support Programme for Nuclear Verification 2016-2017. In addition, we  reviewed documents that establish program requirements, such as the  DNN R&D program’s Goals, Objectives, and Requirements documents;  the DNN R&D and NPAC programs’ calls for project proposals, including  the project selection criteria provided in the proposal calls; and project  proposals. We also interviewed NNSA and other agency officials and  project managers at the national laboratories to obtain their views on  program priorities and the decision-making processes used to define  broad program areas and select projects for funding.
To assess the reliability  of the data we analyzed on the projects in our  nongeneralizable sample, we interviewed NNSA officials and reviewed  documentation, and we determined that the data were sufficiently reliable  for presenting the information contained in this report on NNSA project  characteristics and funding.
We conducted this performance audit from November 2015 to February  2017 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
section_title: Appendix II: Selected Defense Nuclear Nonproliferation Research and Technology Development Projects that Resulted in Deployed Technologies
section_title: Appendix III: Selected Research and Technology Development Projects that Resulted in Advanced, Transitioned, and Deployed Technologies
paragraphs: Nuclear Detonation Detection (NDD) program  area/Forensics-based Nuclear Detonation  Detection functional area (5 projects)  NDD program area/Ground-based Nuclear  Detonation Detection functional area (8 projects)  NDD program area/Space-based Nuclear  Detonation Detection functional area (10 projects)  DNN R&D NDD program area subtotal   (23 projects)  Proliferation Detection program area/Nuclear  Weaponization and Material Production Detection  functional area (17 projects)  Proliferation Detection program area/Enabling  Capabilities functional area (10 projects)  Proliferation Detection program area/Nuclear  Weapons and Material Security functional area  (13 projects)  DNN R&D Proliferation Detection program area  subtotal (40 projects)  DNN R&D subtotal (63 projects)  Total (91 projects)
section_title: Appendix IV: Publicly Reported National Nuclear Security Administration Performance Goals for Research and Technology Development Projects
section_title: Appendix V: Comments from the National Nuclear Security Administration
section_title: Appendix VI: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, William Hoehn (Assistant  Director), Danny Baez, Antoinette  C. Capaccio, Penney Harwell Caramia,  Tara Congdon, John Delicath, Rob Grace, Ben Licht, Alexandria  Palmer,  Timothy M. Persons, Steven Putansu, Ron Schwenn, Kiki  Theodoropoulos, Jack Wang, and Tonya Woodbury made key  contributions to this report.