Nuclear Nonproliferation: Better Information Needed on Results of National Nuclear Security Administration's Research and Technology Development Projects
Released: 2017-02-03T00:00:00

Letter

Background

Within DNN, two programs manage research and technology development projects: the DNN Research and Development (DNN R&D) program and the Nonproliferation and Arms Control (NPAC) program. These two programs have established program areas to focus their research and technology development activities.

DNN R&D program areas. Within the DNN R&D program, two program areas pursue research and technology development projects: Nuclear Detonation Detection and Proliferation Detection. The Nuclear Detonation Detection program area develops and provides global monitoring capabilities for detecting foreign nuclear weapon detonations, in order to meet both treaty monitoring and military needs. The Proliferation Detection program area develops capabilities to detect special nuclear materials (which include plutonium and highly enriched uranium), to detect weapons production and movement, and to enhance transparent nuclear reductions and monitoring. These two program areas each are divided into three functional areas.

The Nuclear Detonation Detection program area’s three functional

Space-based Nuclear Detonation Detection, which develops and builds space-based sensors for nuclear explosion detection;

Ground-based Nuclear Detonation Detection, which produces and updates modeling and analysis capabilities—such as software codes and algorithms that help distinguish underground nuclear detonations from natural seismic events—and provides other technical support for the nation’s ground-based nuclear explosion monitoring networks; and

Forensics-based Nuclear Detonation Detection, which conducts R&D to advance capabilities in the field of nuclear forensics analysis.

The Proliferation Detection program area’s three functional areas

Nuclear Weaponization and Material Production Detection, which supports the development of technology to detect and characterize the production of nuclear weapons and related materials by foreign entities;

Nuclear Weapons and Material Security, which, among other things, develops tools for nuclear security, treaty monitoring and verification, and operational interdiction and nuclear security efforts across NNSA; and

Enabling Capabilities, which develops cross-cutting technologies applicable to multiple NNSA and interagency missions.

NPAC program areas. Within the NPAC program, two program areas pursue technology development projects: International Nuclear Safeguards (Safeguards), among other things, develops technologies to detect and deter undeclared nuclear materials and activities. Specifically, Safeguards’ activities include developing and transferring tools, technologies, and approaches to improve U.S., IAEA, and IAEA member states’ capabilities in undertaking IAEA safeguards activities, such as monitoring uranium enrichment levels in nuclear facilities.

Nuclear Verification (Verification), among other things, develops and deploys technologies to maintain the United States’ ability to monitor and verify nuclear reduction agreements, detect treaty violations, and verify other nuclear nonproliferation commitments. The goals of the Verification program area include developing technologies to verifiably and irreversibly disable nuclear facilities in countries of concern.

Figure 1 illustrates the structure of the DNN programs and functional areas that manage research and technology development projects.

From fiscal year 2012 through fiscal year 2015, DNN R&D and NPAC obligated a total of more than $1.1 billion on 511 research and technology development projects. The DNN R&D program obligated more than $1 billion on 420 projects, and the NPAC program obligated about $73 million on 91 projects. According to NNSA officials we interviewed, the Space-based Nuclear Detonation Detection functional area expenditures were mostly for production activities that resulted in technologies deployed to end users, rather than for basic and applied R&D, in contrast with the other DNN R&D program functional areas. DNN R&D and NPAC projects are undertaken at DOE and NNSA national laboratories and sites; these projects generally last about 3 years from beginning to end, according to program officials. Table 1 shows the number of projects undertaken and funding obligated in each program and functional area from fiscal years 2012 through 2015.

According to NNSA officials, the end users of technologies resulting from DNN’s research and technology development programs include DOD, including its Defense Threat Reduction Agency, U.S. Strategic Command, and Air Force Technical Applications Center; DHS; State; IAEA and other international organizations, including the CTBTO; international governments; universities; and private industry, including small business. NNSA itself also uses some of the technologies that result from DNN’s research and technology development programs.

NNSA Does Not Consistently Track and Document the Full Extent of Nonproliferation Technology Advancement, Transition, and Deployment

The full extent to which research and technology development projects managed by NNSA’s DNN R&D and NPAC programs have resulted in advanced, transitioned, or deployed technologies is unclear because NNSA does not consistently track and document all of these project outcomes. However, by reviewing a range of information on a nongeneralizable sample of 91 DNN research and technology development projects, as well as interviewing end users of technologies that resulted from some of the projects, we were able to determine the outcomes of these projects. More specifically, we found that 88 of the 91 projects in the sample resulted in technologies being advanced—that is, the project progressed the technology itself or the scientific knowledge behind the technology. Additionally, we found that, among these 88 projects, 33 resulted in technologies being transitioned—that is, provided to users outside of the project team for further development or deployment. Finally, we found that 17 of these 33 projects resulted in deployed technologies—that is, we were able to confirm that a technology was being actively used in the field by a federal agency or foreign partner.

NNSA Tracks and Documents Nonproliferation Technology Advancement, but Does Not Do So Consistently for Technology Transition and Deployment

The full extent to which NNSA’s research and technology development projects result in advanced, transitioned, or deployed technologies is unclear because NNSA does not consistently track and document all of these project outcomes. Both the DNN R&D and NPAC programs track and document projects’ results in advancing technologies. For instance, the DNN R&D program uses an online system, WebPMIS, to track project costs and maintain an archive of project-related documentation. This documentation includes project reports that provide information on the scientific and technological advancements that result from a project. Similarly, the NPAC program provided documentation of technology advancement for our review. However, the DNN R&D program, which is by far the larger program, does not consistently track and document projects that result in technologies being transitioned or deployed. NNSA officials acknowledged that the DNN R&D program does not use the WebPMIS system, or otherwise collect and maintain information, to track and document whether its projects result in technologies that have been transitioned to or deployed by end users. As a result, the DNN R&D program was not able to readily provide us with information on the extent to which its projects result in transitioned or deployed technologies. In contrast, we found that the NPAC program tracks and documents the extent to which its safeguards– and verification-related projects have resulted in technology transition or deployment, as well as tracking scientific and technological advancements. For example, NNSA officials provided us with information on all NPAC projects for the past 5 fiscal years that showed whether technologies that projects developed have been deployed; they also provided documentation of technologies that have been transitioned to end users. Both the DNN R&D and NPAC programs list technology transition—which includes providing technologies to end users, who may then deploy them—as a goal in strategic planning documents. In addition, NPAC program officials told us that their program evaluates its success in part based on the extent to which end users deploy technologies that the NPAC program develops. Similarly, DNN R&D program officials said that meeting end users’ technology needs constitutes an important part of their program mission. Moreover, under federal internal control standards, management should design control activities to achieve objectives and respond to risks, and should record, communicate, and use quality information to enable managers to carry out internal control responsibilities and evaluate program performance in achieving key objectives.

Differing program emphases—both between the DNN R&D and NPAC programs and among the DNN R&D program’s functional areas—may account for the inconsistencies we found in NNSA’s tracking and documentation of transition– and deployment-related project outcomes. For instance, the NPAC program generally focuses on developing technologies at higher readiness levels to prepare them for transition and deployment in the field, according to NNSA officials. Conversely, most of the DNN R&D program’s functional areas conduct R&D on technologies that are further from being deployment-ready because they require more basic work at lower technology readiness levels, with some exceptions, NNSA officials told us. Notably, the DNN R&D program’s Space-based Nuclear Detonation Detection functional area has a history, dating to the 1960s, of researching, developing, and producing successive upgrades to technologies used by the U.S. Atomic Energy Detection System and its space-based element, the U.S. Nuclear Detonation Detection System, to monitor and detect nuclear detonations, NNSA officials stated. Officials in the DNN R&D program office said that well-defined DOD technology requirements and delivery schedules drive these research, development, and production activities; as a result, the Space-based Nuclear Detonation Detection functional area has a stronger focus on technology production and deployment than the other DNN R&D program functional areas, which typically do not have similarly specific requirements and schedules. relevant data from reliable internal and external sources, processing the data into quality information, and then considering the data to make informed decisions and evaluate program performance. providing end users with tools or knowledge that are sufficiently ready for the end user to take the final steps necessary for the technology’s use in mission-specific purposes, such as adapting a detection algorithm or producing a field-ready detector array.

It can be difficult to trace a link between a project and a given result— for example, when another agency builds on a project’s work; when a discrete project does not lead directly to an outcome, but results in a “spin-off” project or is part of a related portfolio of projects that does lead to the desired outcome; or when there is a significant period of time between the end of a project and the deployment of any resulting technology.

Project managers at DOE and NNSA national laboratories know the outcomes of projects and can provide information to NNSA officials when needed.

Information about some nonproliferation technologies is classified, with the result that NNSA is not informed about their uses and users.

These factors notwithstanding, by not consistently tracking and documenting all relevant outcomes of projects across DNN’s research and technology development program areas, NNSA is unable to demonstrate to Congress and the public the full extent to which its projects result in technologies that are transitioned, deployed, or otherwise used toward fulfilling the nation’s nuclear nonproliferation goals. Moreover, not maintaining documentary information about project results may affect control activities by limiting the quality of information that NNSA has available to evaluate project performance; it also may reduce opportunities for sharing knowledge, and may make DNN vulnerable to the loss of institutional memory when personnel involved with a project retire or leave the agency.

Selected Projects Resulted in Advanced, Transitioned, and Deployed Technologies

By gathering information from various sources about our sample of 91 selected DNN research and technology development projects, we were able to determine that some projects resulted in technologies being advanced, transitioned, or deployed, or that they resulted in more than one of these outcomes. More specifically, we found that nearly all of the projects we selected (88 of 91) advanced the technology itself or the scientific knowledge behind the technology. Of those 88 projects, we found that 33 also resulted in the advanced technology being transitioned to users outside of the project team for further development or deployment. Among the 33 projects that resulted in a transitioned technology, 17 also resulted in active technology deployment—that is, field use by a federal agency or foreign partner. Figure 2 below illustrates these results.

Almost all of the selected projects in our sample (88 of 91) advanced technologies in some way. Examples of projects that advanced technologies include projects in which NNSA scientists built instrument hardware, made progress in developing new technology combinations or applications, developed an algorithm or model to further data analysis, or made scientific advances that improved nonproliferation analysis and operations.

Some of the projects in our sample that have resulted in technologies being advanced have not resulted in technologies being transitioned to end users (55 of 88 such projects). According to NNSA officials and scientists we interviewed, technology transition has not occurred on these projects for a variety of reasons, including:

The technology is not developed enough to be transitioned. For example, a DNN R&D Proliferation Detection project developed a technology to provide new capabilities in imaging gamma rays. The imager is a laboratory prototype and needs further development, according to a scientist we interviewed at Oak Ridge National Laboratory.

The project has not connected with an end user who wants to receive the technology. For example, a DNN R&D Proliferation Detection project developed a hand-held fast-neutron generator, but NNSA has yet to find an end user interested in the technology, according to the manager of the Nuclear Threat Reduction Program at Lawrence Livermore National Laboratory.

The project developed a technology to meet a future nonproliferation need rather than an existing requirement. For example, a project within the NPAC Verification program developed a fast-neutron imaging technology that could be used in stockpile warhead verification, should a future arms control agreement call for warhead inspections. The technology currently is not being used in a verification context, but NNSA officials said it is in use at NNSA sites to compile benchmark data on nuclear weapons; these data may later be used in verification and other national security missions.

Thirty-three of the 91 Projects in Our Sample Resulted in Transitioned Technologies

We determined that 33 of the 91 projects in our sample resulted in technologies that were transitioned to end users. These 33 projects include 17 projects that resulted in deployed technologies, since a technology is transitioned to an end user before it is deployed. For the remaining 16 projects, we found that either the technologies had not been deployed by an end user (13 of the 16) or we could not confirm with end users that the technologies had been deployed (3 of the 16). Specifically:

We counted 13 of the 16 projects and their resulting technologies as transitioned but not deployed because the end users need to take certain steps before the technologies can be deployed, according to NNSA and end user agency officials. For example, one DNN R&D Nuclear Detonation Detection project transitioned an instrument to a DOD end user. However, the instrument has not been deployed because DOD needs to find funds to further develop the technology, which, according to DOD officials, was at a low level of technology readiness when transitioned. In another case, a DNN R&D Ground- based Nuclear Detonation Detection project transitioned software to a DOD end user, but the user is still evaluating the software before integrating it into its systems.

For the remaining 3 of the 16 projects, we could not confirm whether technologies that resulted from the projects and were transitioned to end users are being used.

Seventeen of the 91 Projects in Our Sample Resulted in Deployed Technologies

We were able to confirm that 17 of the 91 projects in our sample resulted in actively deployed technologies. More specifically, we found that 13 of the 63 DNN R&D projects and 4 of the 28 NPAC projects in our sample resulted in deployed technologies.

Examples of the projects that resulted in deployed technologies include:

An NPAC Safeguards project that developed an online enrichment monitor (OLEM) for use in uranium enrichment monitoring. As we reported in June 2016, the IAEA is using the OLEM in the Natanz Fuel Enrichment Plant in Iran to confirm that enrichment levels are at or below 3.67 percent, per Iran’s commitment under the agreement known as the Joint Comprehensive Plan of Action (JCPOA). IAEA has previously used enrichment monitors, but the OLEM is a newer technology that improves upon older monitoring systems, as we found in our June 2016 report.

DNN R&D Space-based Nuclear Detonation Detection projects that resulted in global burst detection sensors that are in use by the Air Force on satellites to detect, identify, and precisely locate nuclear explosions.

DNN R&D Ground-based Nuclear Detonation Detection projects that resulted in enhanced understanding of waves produced by underground nuclear explosions. The Air Force, the CTBTO, and international scientists analyze these waves to locate underground nuclear explosions and establish their yield.

Of the 17 projects that resulted in deployed technology, 9 were active or ongoing projects and 8 were retired or in closeout, meaning that all of the research work was completed. Similarly, of the 33 projects that resulted in transitioned technology, 15 were active or ongoing projects and 18 were retired or in closeout. Projects resulting in deployed and transitioned technologies can continue to be active for the following reasons: In some cases, project officials provide maintenance and support to the technology after it has been deployed. For example, the OLEM unit has been deployed, but the project is still active because NNSA is continuing to address end user maintenance needs, according to agency officials.

In other cases, the project has deployed or transitioned a technology but has yet to accomplish research goals. For example, a Nuclear Detonation Detection project has deployed a technology to a DOD end user, but research continues because the project has not met all of the research objectives.

See appendix II for a list of the projects in our sample that resulted in deployed technologies. Appendix III provides information on the number of advanced, transitioned, and deployed technologies that resulted from the 91 projects we selected for our sample, subdivided by DNN program and functional areas.

NNSA’s Publicly Reported Program Measures Lack Clarity, and Improved Project-level Documentation Could Enhance Performance Assessment

NNSA’s DNN R&D and NPAC programs use publicly reported measures to assess program-level performance, and both programs review project documentation and communicate with project managers to assess project-level performance. Clarity limitations with the DNN R&D program’s publicly reported measures, however, make them difficult to interpret, and improvements in NNSA’s final project documentation could enhance assessment of project performance.

Clarity Limitations May Hinder Users’ Ability to Interpret DNN’s Publicly Reported Performance Measures

NNSA reports publicly on the DNN R&D and NPAC programs’ performance using measures published in DOE’s annual budget requests, in response to requirements in GPRAMA; however, limitations with the clarity of the DNN R&D program’s measures may make it difficult for users, such as Congress, to understand the targets that NNSA has established for its research and technology development programs, as well as how NNSA has measured performance against these targets. The NNSA budget materials we reviewed present a total of six GPRAMA measures associated with the DNN R&D and NPAC programs. Specifically, the DNN R&D program reports on five GPRAMA measures covering both the Nuclear Detonation Detection and Proliferation Detection program areas. The NPAC program reports on one measure for its Safeguards program area, but does not have a GPRAMA measure for its Verification program area. The six GPRAMA measures identify specific performance timeframes and endpoint targets, and the publicly reported information summarizes NNSA’s assessment of the programs’ performance against the targets. Appendix IV shows the NPAC and DNN R&D performance measures as presented in NNSA’s budget request for fiscal year 2017, including endpoint targets and NNSA’s assessment of the programs’ performance against these targets.

We found that the DNN R&D program’s performance measures are unclear because the program does not define targets or explain its assessments of performance against the targets in sufficient context to allow users to interpret the measures or performance assessments. For example, regarding three of the DNN R&D program’s measures:

The Nuclear Detonation Detection measure tracks percentage of progress against an annual index summarizing the status of all NNSA Nuclear Detonation Detection R&D deliveries that improve the nation’s ability to detect nuclear detonations.

The Nuclear Weapons and Material Security measure tracks cumulative percentage of progress toward demonstrating improvements in special nuclear material detection, warhead monitoring, chain-of-custody monitoring, safeguards, and characterization capabilities.

The Nuclear Weaponization and Material Production Detection measure tracks cumulative percentage of progress toward demonstrating improvements in detection and characterization of nuclear weapons production activities.

For these three measures, the baseline criteria for assessing performance—the annual index and the cumulative percentages of progress—are not defined, and no justification is presented to clarify how NNSA concluded that it had met its performance targets. Without such context, the measures provide statements of NNSA’s assessment of their own performance, but do not provide users with information they can use to evaluate whether NNSA’s assessments are valid and justified.

Previous GAO reports have identified key attributes of successful performance measures, including that they be clearly stated to enable entities to assess their own performance and to enable stakeholders to determine whether a program is achieving its goals. In addition, our December 2011 report on NNSA program management and coordination challenges in the area of nuclear nonproliferation identified concerns with the clarity of performance measures used by the DNN programs we review in this report. Specifically, we found that several of the DNN R&D program’s performance measures were linked to secondary criteria that are classified, official use only, or otherwise not publicly stated, making it difficult for third-party users to interpret the measures and discern whether the criteria are appropriate, sufficient, and up to date for tracking purposes. We recommended in our 2011 report that NNSA clarify these measures. NNSA neither agreed nor disagreed with this recommendation.

NNSA officials told us, when we interviewed them for this report, that they do not agree that the DNN R&D program’s performance measures are unclear, for two reasons.

First, according to NNSA officials, NNSA provides supplemental information for Office of Management and Budget (OMB) users of the performance information that explains the targets and how performance is measured. Specifically, NNSA officials told us that the DNN R&D program provides additional documentation to OMB— some of which the officials also provided to us for review—that includes information explaining the progress against annual targets under each of the program’s performance measures. However, most of the information that NNSA provided on the DNN R&D program is classified or official use only.

Second, an NNSA official told us that the DNN R&D program performance measures are not unclear because the measures are used primarily to inform internal program managers who are familiar with classified requirements against which progress is assessed, rather than to provide information to third-party users. However, there may be potential external users of the performance information—such as congressional decision makers and other external parties—who are not familiar with the baseline requirements and may have difficulty interpreting the level of performance achieved by the DNN R&D program.

In the current review, the clarity issues we found with the DNN R&D program’s performance measures were similar to those we identified in December 2011. We continue to believe that NNSA should clarify these publicly reported measures, so that users of the performance information—including Congress and the public—have a sufficient basis to judge the programs’ performance in meeting the United States’ nuclear nonproliferation technology needs.

Better Documentation Could Enhance NNSA’s Assessment of Project Performance

NNSA assesses the performance of the DNN R&D and NPAC programs’ projects by reviewing project documentation and communicating with project managers, but improvements in NNSA’s final project documentation could enhance assessment of project performance. To assess project performance, NNSA officials said that both programs track progress during the project by communicating with project managers at the national laboratories and by reviewing documents, such as quarterly project reports, that provide information on project performance. According to NNSA documentation we reviewed and NNSA officials we interviewed, the DNN R&D and NPAC programs establish baseline targets for projects’ scope of work and completion date in project plans. In some cases, initial project plans are updated—for example, to incorporate new or modified project objectives. At the end of the project, project managers at the national laboratories produce final project reports for program managers to review.

NNSA officials stated that these final project reports describe a project’s technical results. However, the officials said that the final reports do not document their assessment of project performance against the baseline targets established in the initial project plans. They also said that there is no common template for final project reports. We confirmed this in our review of several final project reports.

NNSA officials provided several reasons why they do not document their assessment of project performance against baseline targets or use a common template for final project reports. For example, according to NNSA officials:

Program managers communicate with project managers at the laboratories to ensure that projects are proceeding as planned.

Restating the baseline targets in final project reports would be duplicative of information contained in other sources, such as the WebPMIS database.

The varying formats of the laboratories’ final reports do not affect program managers’ ability to understand the technical outcomes of the projects.

According to NNSA officials, initial baselines may change through “progressive baselining”—which entails changing the initial project plans when discoveries warrant or course corrections are needed, subject to a defined change control process and the approval of the federal program manager.

Federal internal control standards specify that managers should use quality information to make informed decisions and evaluate performance; that managers need to compare actual performance to planned or expected results; and that entities should record and communicate information to managers who need it to carry out their internal control responsibilities. Moreover, the standards specify that program managers need data to determine whether their programs are meeting their goals for accountability for effective and efficient use of resources. Documenting assessments that compare final project performance results against baseline targets for scope of work and completion date—whether through a common template for final project reports or by other means—could enhance NNSA’s ability to manage its programs in accordance with these standards.

DNN Programs Use Similar Five-step Processes to Select Research and Technology Development Projects

Both the DNN R&D and NPAC programs use similar five-step processes to decide which research and technology development projects to pursue. According to NNSA officials and stakeholders we interviewed at technology end user agencies, NNSA collaborates throughout the decision-making and project selection process with the end users. In our review of program documents and interviews with agency officials, we found that the DNN R&D and NPAC programs generally take the following steps in their project selection processes: Step 1: DNN establishes project requirements. According to NNSA officials, DNN R&D and NPAC project requirements flow from national policy documents, such as the Nuclear Posture Review; additionally, NPAC project requirements flow from international sources such as the Treaty on the Non-Proliferation of Nuclear Weapons (NPT), Comprehensive Safeguards Agreements; Additional Protocols; and the IAEA Department of Safeguards Long Term R&D Plan. To identify the technical capabilities that are required to meet the requirements established in these documents and sources, NNSA officials develop more detailed plans, such as the DNN R&D program’s goals, objectives, and requirements (GOR) documents. The GOR documents specify requirements for five of the DNN R&D program’s six functional areas—all but the cross-cutting Enabling Capabilities functional area. Additionally, each GOR document has a corresponding technology roadmap, which establishes detailed technical requirements to meet the needs identified in the GOR document. Similarly, the NPAC program develops needs documents that identify specific needs and corresponding capabilities to develop; NPAC uses these documents to guide the proposal selection process. NNSA consults with interagency stakeholders to inform the requirements specified in the GOR documents and needs documents.

Step 2: The DNN programs issue annual calls for proposals. Both programs issue annual calls for proposals, which specify the technical requirements proposals should address. The programs also require that those who submit proposals clearly explain how the proposed project relates to identified project requirements. According to NNSA officials, the DNN R&D program typically issues its calls for proposals in mid-October and NPAC issues its calls for proposals during the spring. The DNN R&D program’s calls for proposals specify weighted criteria against which proposals are reviewed, including mission relevance, scientific and technical merit, and budget. The NPAC program’s calls for proposals differ between its two program areas. The Safeguards program area provides explicit evaluation criteria including cost effectiveness, technical maturity, and deployment potential. Calls for proposals in the Verification program area do not provide explicit evaluation criteria, but they provide information on program goals and areas of technical consideration for the current budget year. They also require applicants to describe the proposed work, including activities, milestones, and deliverables.

Step 3: National laboratories submit informal project proposals. Scientists at the national laboratories respond to the call for proposals by submitting informal project proposals, known as white papers, to the NNSA program offices. The white papers describe proposed projects to demonstrate new capabilities or technologies sought in the calls for proposals. As specified in the DNN R&D program’s calls for proposals, the program uses a three- tiered system to rank proposals—specifically, by designating proposals as hot, warm, or cold—and provides comments to improve the proposals or future white paper submissions. According to an NNSA official, the DNN R&D program returns the informal project proposals to the original submitters. All hot and warm proposals may be amended in response to program officials’ comments and then resubmitted in the next step of the process as formal project proposals. NNSA officials stated that cold proposals receive no further consideration. NNSA officials within the NPAC program stated that they use a ranking system that is similar to the DNN R&D program’s system; the officials also provided documentation that shows their rationale for selecting or not selecting projects for further consideration. Both programs seek to provide feedback within a general time frame. The DNN R&D program aims to provide feedback within 2 weeks of receiving the white papers. According to NNSA officials, NPAC’s Safeguards program area provides feedback on the white papers about 3 to 3 1/2 months from the date of submission, and the Verification program area provides feedback about 2 1/2 months from the date of submission. According to NNSA officials and interagency stakeholders we interviewed, interagency stakeholders are consulted during this step of the process, as appropriate.

Step 4: Laboratories submit formal project proposals. Scientists at the national laboratories amend hot and warm white papers and resubmit them as formal proposals for NNSA’s consideration.

Step 5: DNN selects projects in consultation with laboratories and end users. NNSA officials identify the formal proposals that best meet the criteria and requirements specified in the calls for proposals, considering input provided by laboratory officials, interagency stakeholders, and potential end users on each proposal. According to NNSA officials, after discussing budgetary considerations and current, prioritized needs with interagency stakeholders and potential end users, program officials develop a final, ranked list of projects selected to receive funding and support. Program officials then circulate this final list to interagency stakeholders and potential end users.

Figure 3 shows the general five-step selection process used by DNN’s programs.

Conclusions

NNSA’s research and technology development programs make vital contributions to national security, and the projects they support address a range of important proliferation detection and monitoring needs, including improving abilities to detect covert nuclear material production, nuclear explosions, and potential violations of IAEA safeguards agreements. It is not clear, however, how often NNSA’s research and technology development projects—especially those supported by the DNN R&D program—result in transitioned or deployed technologies because NNSA does not consistently track and document such information. We acknowledge that DNN’s research and technology development programs pursue a broad range of scientific tools and capabilities, with the goal of ensuring that the United States remains prepared to deal with unanticipated nonproliferation challenges as they emerge. Therefore, we recognize that not all R&D projects, such as projects conducted at low levels of technology readiness, are intended to result in a deployed technology, and that a higher deployment rate does not necessarily indicate a more successful R&D program, or vice versa. Moreover, we recognize that in cases where technology transition and deployment are project goals, certain challenges complicate DNN’s efforts to track the deployed technologies that result from its projects. Nevertheless, we were able, as part of our review of selected projects, to track projects’ end results, including end users’ deployment of technologies that result from the projects. Moreover, not tracking such end results may affect control activities by limiting the quality of information that NNSA has available to evaluate project performance. More consistently tracking and documenting the transitioned and deployed technologies that result from its projects would help ensure that NNSA maintains and uses quality information to evaluate its performance and achieve its objectives, in keeping with federal internal control standards. Doing so would also facilitate knowledge sharing within DNN, and it would provide a means by which to present valuable information to Congress and other decision makers about the programs’ results and their overall value.

In addition, better tracking of project results by the DNN R&D program may have the salutary benefit of providing information to the program that could allow it to develop clearer program performance measures, as we recommended in December 2011. The DNN R&D program’s performance measures continue to have clarity limitations similar to the ones we identified in the December 2011 report, because understanding the publicly reported performance information depends on users of the information having access to or being familiar with criteria in official use only or classified documents. We continue to believe that it is important for NNSA to implement our December 2011 recommendation that NNSA clarify the DNN R&D program’s publicly reported performance measures, so that Congress and other users of these measures have a sufficient understanding of DNN programs’ status and progress.

By documenting assessments that compare the final results of a project against the baseline targets for scope of work and completion date that are established in initial plans for each project, NNSA could improve the quality of the information it needs to evaluate performance at the end of the project. Specifically, taking these steps could enhance managers’ ability to use quality data to assess actual project performance against planned or expected results, consistent with federal internal control standards. Such assessments are essential to ensure that the DNN R&D and NPAC programs’ research and technology development projects are effectively and efficiently using resources, another key aspect of federal internal control standards. If changes to the initial baselines are necessary—which may be the case in some instances—we believe that they should be made in such a way that initial project targets are still documented and available for review in evaluating the full progression of a project.

Recommendations for Executive Action

We recommend that the NNSA Administrator take these two actions:

Direct the DNN R&D program to track and document the transitioned and deployed technologies that result from its research and technology development projects, to the extent practicable.

Direct the DNN R&D and NPAC programs to document, using a common template or other means, their assessment that compares the final results of each project against the baseline targets established in each project’s initial project plan.

Agency Comments and Our Evaluation

We provided drafts of this report to NNSA, DOD, DHS, and State for review and comment. In emails, DOD, DHS, and State stated that they had no comments on the report. In NNSA’s written comments, which are summarized below and reproduced in appendix V, NNSA neither agreed nor disagreed with our first recommendation and partially agreed with our second recommendation; however, NNSA stated that it plans to take actions in response to both recommendations. NNSA also provided technical comments, which we incorporated as appropriate.

In its written comments, NNSA did not state whether it agreed or disagreed with the recommendation that, to the extent practicable, the DNN R&D program track and document transitioned and deployed technologies resulting from the program’s projects, but it described planned actions consistent with the recommendation. NNSA stated that our report did not clearly or fully disclose reasons that the DNN R&D program does not track and document transition and deployment of technologies resulting from its projects. Specifically, NNSA commented that the DNN R&D program’s space-based activities—unlike the other DNN R&D program areas—are production and deployment oriented, and that it would therefore be misleading to track individual projects in this area that are transitioned and deployed. Regarding the other DNN R&D program areas, NNSA commented that it does not have ongoing insight into transition and deployment outcomes for various reasons—for example, due to classification barriers regarding the transition and deployment of certain technologies. NNSA also expressed concern that information on such outcomes may not be readily available or may not be cost-effective to obtain. Our report discusses a number of reasons why the agency may not consistently track the deployment– and transition- related outcomes of the DNN R&D program’s projects and notes the production and deployment orientation of the space-based project portfolio. However, we disagree that tracking the outcomes of these projects would be misleading. To the contrary, it is vital to track these outcomes, as the space-based program area receives the largest share of the DNN R&D program’s funding. Moreover, we believe that information on project outcomes should be readily available to NNSA program officials in the course of their regular interactions with the project managers at the national laboratories and the interagency end users of nonproliferation technologies—as was the case with projects in our sample. NNSA agreed that, where such information is readily available, reporting of transitioned and deployed technologies can provide information on program successes, and it identified actions it will take consistent with the recommendation. Specifically, NNSA stated it will complete an assessment by June 2017 of the DNN R&D program’s portfolio to determine to what extent transition and deployment data are readily available for specific projects and include additional information on those projects as part of its performance reporting. If implemented as planned, such an assessment would be a step toward ensuring that NNSA maintains and uses quality information to evaluate its performance and achieve its objectives, in keeping with federal internal control standards.

NNSA stated that it agrees in part with our second recommendation—that NNSA document, using a common template or other means, its assessment that compares the final results of each DNN R&D and NPAC project against the baseline targets established in each project’s initial project plan. NNSA stated that it supports implementation of a common template for documenting project closeout to provide consistency in information reporting, but it noted that R&D projects are dissimilar to acquisition activities and that comparing final project results to an initial project baseline would be misleading. Specifically, for exploratory R&D projects, NNSA commented that project outcomes can be “significantly indeterminate” at the beginning of a project and stated that it uses an approach known as “progressive baselining” to adjust plans and benchmarks as research matures, where appropriate. NNSA stated that final project results should be compared to the final approved plans to ensure consistency in project documentation and to provide a meaningful assessment of project performance. NNSA agreed that information on the project’s initial baseline is important to understand the progression of the project. We acknowledge that adjustments to initial project plans may be appropriate in some cases, provided that the initial plans remain available and transparent and that decisions behind such revisions are well- documented, consistent with federal internal control standards. NNSA stated it will establish a common template by June 2017 and use it to compare projects’ final results to the final approved plans. It also stated that the template will require a brief description of significant changes between the initial and final approved project plans. If implemented as planned, the template could help NNSA to ensure that the DNN R&D and NPAC programs’ research and technology development projects are effectively and efficiently using resources, in keeping with federal internal control standards.

We are sending copies of this report to the appropriate congressional committees; the Administrator of NNSA; the Secretaries of Defense, Energy, Homeland Security, and State; and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov.

If you or your staff members have questions about this report, please contact me at (202) 512-3841 or oakleys@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix VI.

Appendix I: Objectives, Scope and Methodology

This report (1) evaluates the extent to which projects managed by the National Nuclear Security Administration’s (NNSA) research and technology development programs have resulted in advanced, transitioned, or deployed nonproliferation technologies, (2) evaluates how NNSA measures the performance of its research and technology development programs and projects, and (3) describes how programs in NNSA’s Office of Defense Nuclear Nonproliferation (DNN) decide which research and technology development projects to pursue.

To determine the extent to which NNSA’s research and technology development programs have resulted in advanced, transitioned, or deployed nonproliferation technologies, we selected a nongeneralizable sample from research and technology development projects undertaken between fiscal years 2012 and 2015 by the two DNN programs responsible for such projects: the Office of DNN Research and Development (DNN R&D) and the Office of Nonproliferation and Arms Control (NPAC). As detailed below, we selected the projects in the sample based on funding level and project technical area of focus and status in order to cover a broad range of program activities.

In total, the DNN R&D and NPAC programs provided information on 511 projects. Specifically, the DNN R&D program provided us with a list of 420 projects executed between fiscal years 2012 and 2015, and the two NPAC program areas that manage technology development projects— International Nuclear Safeguards (NPAC Safeguards) and Nuclear Verification (NPAC Verification)—provided us with a list of 91 projects that were executed between fiscal years 2012 and 2015.

From this universe, we developed selection criteria to include projects from five of the major Department of Energy (DOE) and NNSA laboratories—Los Alamos National Laboratory, Oak Ridge National Laboratory, Sandia National Laboratories, Lawrence Livermore National Laboratory, and Pacific Northwest National Laboratory. Our selection criteria ensured variations in program areas and project outcomes. We generally limited our selection to the top 25 percent of projects in terms of total funding between fiscal years 2012 and 2015, and excluded all projects that were not led by one of the five national laboratories listed above. We also excluded projects in certain categories, such as projects under the program directors and university accounts. After these exclusions, 134 projects remained in the sample.

The next step in our selection process was to group together similar projects, based on project status, focus area or type, age, and cost. We generally selected 2 projects from each group, to ensure that each combination of these characteristics was represented in our final sample. In cases in which a group contained only one project, we included only that project.

For the DNN R&D program, we treated projects with the same laboratory, focus area, and status as a single group, resulting in 40 groups. For each group, we generally chose the 2 projects with the lowest project number (signifying the earliest start date), resulting in the inclusion of 63 projects.

For the NPAC Safeguards program area, we treated projects with the same technology category (comparable to focus area) and project status as a single group, resulting in 13 groups. Because we did not have data to determine earliest start date, we generally chose the 2 projects with the highest total expenditures from each group, resulting in the inclusion of 17 projects.

For the NPAC Verification program area, we treated projects with the same R&D category (comparable to focus area) and project status as a single group, resulting in 8 groups. Because we did not have data to determine earliest start date, we chose the 2 projects with the highest total expenditures from each group, resulting in the inclusion of 11 projects.

This resulted in a final sample of 91 projects covering the DNN R&D and NPAC programs and all of their program and functional areas. Table 2, below, shows the number of projects we selected from each program and functional area.

To determine the extent to which the 91 projects in our sample resulted in advanced, transitioned, or deployed technologies, we gathered project information from interviews and documentary sources. Specifically, we interviewed project officials, including the project manager, at the national laboratory that was the lead on the project, as well as interagency officials that received technologies transitioned from the DNN R&D and NPAC programs. Our interviews included officials from the Departments of Defense (DOD), Homeland Security, and State; the International Atomic Energy Agency (IAEA); and the Preparatory Commission for the Comprehensive Test Ban Treaty Organization. We also reviewed program and project data and documentation, including basic project data—such as project names and funding information—stored in DNN R&D’s WebPMIS database; program funding information; project plans; annual, quarterly, and final reports; and research presentations and publications. We reviewed the evidence to determine whether each project in our sample resulted in advanced, transitioned, or deployed technology or other outcomes. The outcomes that we identified are not mutually exclusive for any project. For example, one project could transition a technology to an end user and also result in a publication, or a project may have resulted in a research outcome that we did not identify because the outcome was not indicated in the evidence we collected. Finally, we reviewed documents that establish program goals, including strategic planning documents, as well as Standards for Internal Control in the Federal Government, which establishes federal standards for the use of information in achieving an entity’s objectives.

To evaluate how NNSA measures the performance of its research and technology development programs and projects, we reviewed externally reported and internal program– and project-level measures that NNSA uses to assess performance in DNN’s research and technology development efforts. For program-level measures, we focused our review on the publicly reported measures developed and reported on in NNSA’s budget materials in response to the GPRA Modernization Act of 2010 (GPRAMA). Specifically, we reviewed the GPRAMA measures for the DNN R&D and NPAC programs that NNSA presented in its budget materials for fiscal years 2015 through 2017. We also reviewed classified and official use only materials that, according to NNSA officials, NNSA shares with the Office of Management and Budget (OMB) in their discussions of the GPRAMA measures. The materials we reviewed included technology roadmaps, which define the technology pathways that NNSA follows to address portfolio requirements and develop funding priorities, and final reports that NNSA presents to OMB on the completion of activities it undertakes in connection with the GPRAMA measures. We also reviewed DOE’s Annual Performance Reports for fiscal years 2014 and 2015. In addition, we reviewed documents and interviewed NNSA officials regarding internal measures that DNN R&D and NPAC use to monitor program performance, including independent project performance reviews. To further evaluate DNN’s project-level performance measures, we reviewed information on the 91 selected projects in our nongeneralizable sample. Specifically, we reviewed documents that establish baseline targets for projects’ scope, cost, and completion date and documents that contain information on project results. We also interviewed DNN officials to obtain their views on program and project management and performance. Finally, we reviewed documents that establish criteria for successful performance measures, including our past reports on the key attributes of such measures, as well as federal internal control standards.

To describe how DNN decides which projects to pursue, we reviewed documents that establish the nonproliferation mission needs of DOE and interagency and IAEA stakeholders in nonproliferation-related research and technology development. The documents we reviewed include DOD’s 2010 Nuclear Posture Review Report; the Executive Office of the President’s National Science and Technology Council’s Nuclear Defense Research and Development Roadmap, Fiscal Years 2013-2017; the White House’s National Security Strategy of February 2015; DOE and NNSA strategic plans; the IAEA Department of Safeguards’ Long-Term R&D Plan, 2012-2023; and IAEA’s Development and Implementation Support Programme for Nuclear Verification 2016-2017. In addition, we reviewed documents that establish program requirements, such as the DNN R&D program’s Goals, Objectives, and Requirements documents; the DNN R&D and NPAC programs’ calls for project proposals, including the project selection criteria provided in the proposal calls; and project proposals. We also interviewed NNSA and other agency officials and project managers at the national laboratories to obtain their views on program priorities and the decision-making processes used to define broad program areas and select projects for funding.

To assess the reliability of the data we analyzed on the projects in our nongeneralizable sample, we interviewed NNSA officials and reviewed documentation, and we determined that the data were sufficiently reliable for presenting the information contained in this report on NNSA project characteristics and funding.

We conducted this performance audit from November 2015 to February 2017 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Selected Defense Nuclear Nonproliferation Research and Technology Development Projects that Resulted in Deployed Technologies

Appendix III: Selected Research and Technology Development Projects that Resulted in Advanced, Transitioned, and Deployed Technologies

Nuclear Detonation Detection (NDD) program area/Forensics-based Nuclear Detonation Detection functional area (5 projects) NDD program area/Ground-based Nuclear Detonation Detection functional area (8 projects) NDD program area/Space-based Nuclear Detonation Detection functional area (10 projects) DNN R&D NDD program area subtotal (23 projects) Proliferation Detection program area/Nuclear Weaponization and Material Production Detection functional area (17 projects) Proliferation Detection program area/Enabling Capabilities functional area (10 projects) Proliferation Detection program area/Nuclear Weapons and Material Security functional area (13 projects) DNN R&D Proliferation Detection program area subtotal (40 projects) DNN R&D subtotal (63 projects) Total (91 projects)

Appendix IV: Publicly Reported National Nuclear Security Administration Performance Goals for Research and Technology Development Projects

Appendix V: Comments from the National Nuclear Security Administration

Appendix VI: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, William Hoehn (Assistant Director), Danny Baez, Antoinette C. Capaccio, Penney Harwell Caramia, Tara Congdon, John Delicath, Rob Grace, Ben Licht, Alexandria Palmer, Timothy M. Persons, Steven Putansu, Ron Schwenn, Kiki Theodoropoulos, Jack Wang, and Tonya Woodbury made key contributions to this report.