Combating Nuclear Smuggling: Recent Testing Raises Issues About the Potential Effectiveness of Advanced Radiation Detection Portal Monitors
Released: 2009-11-17T00:00:00

Letter

Improved Testing Rigor Discussed in Our May 2009 Report Demonstrates Limitations of ASPs

Our May 2009 report on the then-current round of ASP testing found that DHS increased the rigor of ASP testing over that of previous tests, and that a particular area of improvement was in the performance testing at the Nevada Test Site, where DNDO compared the capability of ASP and current-generation equipment to detect and identify nuclear and radiological materials. For example, unlike in prior tests, the plan for the 2008 performance test stipulated that the contractors who developed the equipment would not be involved in test execution. This improvement addressed concerns we previously raised about the potential for bias and provided increased credibility to the results. Nevertheless, based on the following factors, in our report we questioned whether the benefits of the new portal monitors justify the high cost: The DHS criteria for a significant increase in operational effectiveness.

Our chief concern with the criteria is that they require only a marginal improvement over current-generation portal monitors in the detection of certain weapons-usable nuclear materials during primary screening. DNDO considers detection of such materials to be a key limitation of current-generation portal monitors. The marginal improvement required of ASPs to meet the DHS criteria is problematic because the detection threshold for the current-generation portal monitors does not specify a level of radiation shielding that smugglers could realistically use. Officials from the Department of Energy (DOE), which designed the threat guidance DHS used to set the detection threshold, and national laboratory officials told us that the current threshold is based not on an analysis of the capabilities of potential smugglers to take effective shielding measures but rather on the limited sensitivity of PVTs to detect anything more than certain lightly shielded nuclear materials. DNDO officials acknowledge that both the new and current-generation portal monitors are capable of detecting certain nuclear materials only when unshielded or lightly shielded. The marginal improvement in detection of such materials required of ASPs is particularly notable given that DNDO has not completed efforts to fine-tune PVTs’ software using a technique called “energy windowing” that could improve the PVTs’ sensitivity to nuclear materials. DNDO officials expect they can achieve small improvements in sensitivity through energy windowing, but DNDO has not yet completed efforts to fine-tune the PVTs’ software. In contrast to the marginal improvement required in detection of certain nuclear materials, the primary screening requirement to reduce the rate of innocent alarms by 80 percent could result in hundreds of fewer secondary screenings per day, thereby reducing CBP’s workload. In addition, the secondary screening criteria, which require ASPs to reduce the probability of misidentifying special nuclear material by one-half, address the limitations of relatively small handheld devices in consistently locating and identifying potential threats in large cargo containers.

Results of performance testing and field validation. The results of performance tests that DNDO presented to us were mixed, particularly in the ASPs’ capability to detect certain shielded nuclear materials during primary screening. The results of performance testing at the Nevada Test Site showed that the new portal monitors detected certain nuclear materials better than PVTs when shielding approximated DOE threat guidance, which is based on light shielding. In contrast, differences in system performance were less notable when shielding was slightly increased or decreased: both the PVTs and ASPs were frequently able to detect certain nuclear materials when shielding was below threat guidance, and both systems had difficulty detecting such materials when shielding was somewhat greater than threat guidance. With regard to secondary screening, ASPs performed better than handheld devices in identification of threats when masked by naturally occurring radioactive material. However, the differences in the ability to identify certain shielded nuclear materials depended on the level of shielding, with increasing levels appearing to reduce any ASP advantages over the handheld identification devices. Other phases of testing uncovered multiple problems in meeting requirements for successfully integrating the new technology into operations at ports of entry. Of the two ASP contractors participating in the current round of testing, one has fallen behind due to severe problems encountered during testing of ASPs’ readiness to be integrated into operations at ports of entry (“integration testing”); the problems may require that the vendor redo previous test phases to be considered for certification. The other vendor’s system completed integration testing, but CBP suspended field validation testing in January 2009 after 2 weeks because of serious performance problems resulting in an overall increase in the number of referrals for secondary screening compared with existing equipment.

DNDO’s plans for computer simulations. As of May 2009, DNDO did not plan to complete injection studies—computer simulations for testing the response of ASPs and PVTs to simulated threat objects concealed in cargo containers—prior to the Secretary of Homeland Security’s decision on certification even though delays to the ASP test schedule have allowed more time to conduct the studies. According to DNDO officials, injection studies address the inability of performance testing to replicate the wide variety of cargo coming into the United States and the inability to place special nuclear material and other threat objects in cargo during field validation. DNDO had earlier indicated that injection studies could provide information comparing the performance of the two systems as part of the certification process for both primary and secondary screening. However, DNDO subsequently decided that performance testing would provide sufficient information to support a decision on ASP certification. DNDO officials said they would instead use injection studies to support effective deployment of the new portal monitors.

Lack of an updated cost-benefit analysis. DNDO had not updated its cost- benefit analysis to take into account the results of ASP testing. An updated analysis that takes into account the testing results, including injection studies, might show that DNDO’s plan to replace existing equipment with ASPs is not justified, particularly given the marginal improvement in detection of certain nuclear materials required of ASPs and the potential to improve the current-generation portal monitors’ sensitivity to nuclear materials, most likely at a lower cost. DNDO officials said they were updating the ASP cost-benefit analysis and planned to complete it prior to a decision on certification by the Secretary of Homeland Security.

Our May report recommended that the Secretary of Homeland Security direct DNDO to (1) assess whether ASPs meet the criteria for a significant increase in operational effectiveness based on a valid comparison with PVTs’ full performance potential and (2) revise the schedule for ASP testing and certification to allow sufficient time for review and analysis of results from the final phases of testing and completion of all tests, including injection studies. We further recommended that, if ASPs are certified, the Secretary direct DNDO to develop an initial deployment plan that allows CBP to uncover and resolve any additional problems not identified through testing before proceeding to full-scale deployment. DHS agreed to a phased deployment that should allow time to uncover ASP problems but disagreed with GAO’s other recommendations, which we continue to believe remain valid.

Results from July 2009 Testing Raise Continuing Issues

The results of DNDO’s most recent round of field validation testing, which it undertook in July 2009, after our May report was released, raise new issues. In July 2009, DNDO resumed the field testing of ASPs at four CBP ports of entry that it initiated in January 2009 but suspended because of serious performance problems. However, the July tests also revealed ASP performance problems, including two critical performance deficiencies. First, the ASP monitors had an unacceptably high number of false positive alarms for the detection of certain high-risk nuclear materials. According to CBP officials, these false alarms are very disruptive in a port environment in that any alarm for this type of nuclear material would cause CBP to take enhanced security precautions because such materials (1) could be used in producing an improvised nuclear device and (2) are rarely part of legitimate or routine cargo. Furthermore, once receiving an alarm for this type of nuclear material, CBP officers are required to conduct a thorough secondary inspection to assure themselves that no nuclear materials are present before permitting the cargo to enter the country. Repeated false alarms for nuclear materials are also causes for concern because such alarms could eventually have the effect of causing CBP officers to doubt the reliability of the ASP and be skeptical about the credibility of future alarms.

Secondly, during the July testing the ASP experienced a “critical failure,” which stemmed from a problem with a key component of the ASP and caused the ASP to shut down. Importantly, during this critical failure, the ASP did not alert the CBP officer that it had shut down and was no longer scanning cargo. As a result, were this not in a controlled testing environment, the CBP officer would have permitted the cargo to enter the country thinking the cargo had been scanned, when it had not. According to CBP officials, resolving this issue is important in order to assure the stability and security of the ASP.

In addition to these key performance problems, the ASP was not able to reduce referrals to secondary inspection by 80 percent as required by the DHS criteria for a significant increase in operational effectiveness. According to the report from the ASP Field Validation Advisory Panel, a panel made up of officials from CBP, DNDO, and a national laboratory, the ASP was able to reduce referrals to secondary inspection by about 69 percent rather than the 80 percent as required by the DHS criteria.

While the performance of the ASP during the July field validation testing raises issues about its potential readiness for deployment, DNDO’s proposed solutions to address these performance problems raise additional questions about whether this equipment will provide any meaningful increase in the ability to detect certain nuclear materials. Specifically, to address the problem of false positive alarms indicating the presence of certain nuclear materials, according to DNDO officials, DNDO has modified the ASP to make this equipment less sensitive to these nuclear materials. While this may address the issue of false positive alarms, it will also diminish the ASP capability of detecting a key high-risk nuclear material. Since the ASP modification, DNDO conducted computer simulations using a vendor-provided system called a “replay tool” to examine the effect of the modification on the ASP’s performance. According to DNDO officials, the replay tool demonstrated that the modified ASP will still be able to detect certain nuclear materials better than the PVT. However, at this point, DNDO does not plan to retest the ASP at the Nevada Test Site where it can examine the effects of these modifications using actual nuclear materials. As we reported earlier this year, the results of the testing at the Nevada Test Site demonstrated that the ASPs represented a marginal improvement in detecting certain nuclear materials. By reducing the sensitivity to these materials and not retesting the modified ASPs against actual nuclear materials, it is uncertain exactly what improvement in detecting certain nuclear materials these costly portal monitors are providing.

While DNDO is reducing the sensitivity of ASPs to certain nuclear materials, it has yet to complete efforts to improve the PVT’s ability to detect these same materials through energy windowing. For several years, CBP officials have repeatedly urged DNDO officials to complete this research. However, it was not apparent from our discussions with DNDO officials if this effort is making meaningful progress with the development of energy windowing or when it will be completed. Furthermore, CBP officials stated that, depending on the outcome of this research, energy windowing could be the more cost effective way to improve detection of certain nuclear materials. In our view, ASPs being modified to diminish their capabilities to detect certain nuclear materials raises questions about whether energy windowing might be able to achieve a similar level of performance against these same materials from the PVTs that are already in place.

Beyond reducing the sensitivity of ASPs to certain nuclear materials, DNDO also plans to address the issue of critical failures by, among other things, installing an indicator light on the ASP that will alert CBP officers that the ASP has experienced a mission-critical failure and is no longer scanning cargo. While this should address the issue of CBP officers not knowing that the ASP has suffered a critical failure, CBP officials stressed to us the need for the ASP to be stable and secure enough to avoid these shutdowns.

In closing, the issues raised by the results of the July 2009 field validation tests provide even greater reason for DNDO to address recommendations from our May 2009 report. In particular, we reiterate the importance of our prior recommendation for DNDO to assess whether ASPs meet the criteria for a significant increase in operational effectiveness based on a valid comparison with PVTs’ full performance potential, given that the ASPs will no longer be as effective in detecting certain nuclear materials.

Mr. Chairman, this completes my prepared statement. I would be happy to respond to any questions that you or other Members of the Subcommittee may have at this time.

GAO Contact and Staff Acknowledgments

For further information about this testimony, please contact me at (202) 512-3841 or aloisee@gao.gov. Dr. Timothy Persons (Chief Scientist), Ned Woodward (Assistant Director), Joseph Cook, and Kevin Tarmann made key contributions to this testimony. Kendall Childers, Karen Keegan, Carol Kolarik, Jonathan Kucskar, Omari Norman, Alison O’Neill, and Rebecca Shea also made important contributions.

This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.