Combating Nuclear Smuggling: Recent Testing Raises Issues About the Potential Effectiveness of Advanced Radiation Detection Portal Monitors
Published: 2009-11-17T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Department of Homeland Security's (DHS) Domestic Nuclear Detection Office (DNDO) is responsible for addressing the threat of nuclear smuggling. Radiation detection portal monitors are key elements in the nation's defenses against such threats. DHS has sponsored testing to develop new monitors, known as advanced spectroscopic portal (ASP) monitors, to replace radiation detection equipment being used at ports of entry. DNDO expects that ASPs may offer improvements over current-generation portal monitors, particularly the potential to identify as well as detect radioactive material and thereby to reduce both the risk of missed threats and the rate of innocent alarms, which DNDO considers to be key limitations of radiation detection equipment currently used by Customs and Border Protection (CBP) at U.S. ports of entry. However, ASPs cost significantly more than current generation portal monitors. Due to concerns about ASPs' cost and performance, Congress has required that the Secretary of Homeland Security certify that ASPs provide a significant increase in operational effectiveness before obligating funds for full-scale ASP procurement. In May 2009, GAO issued a report (GAO-09-655) on the status of the ongoing ASP testing round. This testimony (1) discusses the principal findings and recommendations from GAO's May report on ASP testing and (2) updates those findings based on information from DNDO and CBP officials on the results of testing conducted since the report's issuance. DHS, DNDO, and CBP's oral comments on GAO's new findings were included as appropriate.
section_title: What GAO Found
paragraphs: GAO's May 2009 report on ASP testing found that DHS increased the rigor in comparison with previous tests and thereby added credibility to the test results. However, GAO's report also questioned whether the benefits of the ASPs justify its high cost. In particular, the DHS criteria for a significant increase in operational effectiveness require only a marginal improvement in the detection of certain weapons-usable nuclear materials, which DNDO considers a key limitation of current-generation portal monitors. The marginal improvement required of ASPs is particularly notable given that DNDO has not completed efforts to fine-tune current-generation equipment to provide greater sensitivity. Moreover, the test results showed that ASPs performed better than current-generation portal monitors in detection of such materials concealed by light shielding approximating the threat guidance for setting detection thresholds, but that differences in sensitivity were less notable when shielding was slightly below or above that level. Finally, DNDO had not yet updated its cost-benefit analysis to take into account the results of ASP testing and did not plan to complete computer simulations that could provide additional insight into ASP capabilities and limitations prior to certification even though test delays have allowed more time to conduct the simulations. DNDO officials believed the other tests were sufficient for ASPs to demonstrate a significant increase in operational effectiveness. GAO recommended that DHS assess ASPs against the full potential of current-generation equipment and revise the program schedule to allow time to conduct computer simulations and to uncover and resolve problems with ASPs before full-scale deployment. DHS agreed to a phased deployment that should allow time to uncover ASP problems but disagreed with the other recommendations, which GAO believes remain valid. The results of DNDO's most recent round of field testing raise continuing issues. In July 2009, DNDO resumed the field testing of ASPs that it initiated in January 2009 but suspended because of serious performance problems. However, the July tests also revealed critical performance deficiencies. For example, the ASP had a high number of false positive alarms for the detection of certain nuclear materials. According to CBP, these false alarms are very disruptive in a port environment because any alarm for this type of nuclear material causes CBP to take enhanced security precautions. To address these false alarms, DNDO plans to modify the ASP to make these monitors less sensitive to these nuclear materials and thereby diminishing the ASPs' capability. As GAO reported earlier this year, previous testing results demonstrated that the ASPs represented a marginal improvement in detecting these materials. By reducing the sensitivity to nuclear materials even further, it is uncertain exactly what improvement in detecting these materials the ASPs are providing or whether DNDO might be able to achieve a similar level of performance as the modified ASPs by improving the current-generation portal monitors that are already in place. In addition, the July 2009 testing also identified a critical equipment failure, including an alert malfunction, which DNDO is taking steps to resolve for future testing.

section_title: Letter
paragraphs: I am pleased to be here today to discuss GAO’s work on the Department of  Homeland Security’s (DHS) testing of advanced spectroscopic portal  (ASP) radiation detection monitors. One mission of U.S. Customs and  Border Protection (CBP), an agency within DHS, includes screening cargo  and vehicles coming into this country for smuggled nuclear or radiological  material that could be used in an improvised nuclear device or radiological  dispersal device (a “dirty bomb”). To screen cargo at ports of entry, CBP  conducts primary inspections with radiation detection equipment called  portal monitors—large stationary detectors through which cargo  containers and vehicles pass as they enter the United States. When  radiation is detected, CBP conducts secondary inspections using a second  portal monitor to confirm the original alarm and a handheld radioactive  isotope identification device to identify the radiation’s source and  determine whether it constitutes a threat.
The polyvinyl toluene (PVT) portal monitors CBP currently uses for this  screening can detect radiation but cannot identify the type of material  causing an alarm. As a result, the monitors’ radiation alarms can be set off  even by shipments of bananas, kitty litter, or granite tile because these  materials contain small amounts of benign, naturally occurring radioactive  material. To address the limitations of current-generation portal monitors,  DHS’s Domestic Nuclear Detection Office (DNDO) in 2005 began to  develop and test ASPs, which are designed to both detect radiation and  identify the source. DNDO hopes to use the new portal monitors to  replace at least some PVTs currently used for primary screening, as well as  PVTs and handheld identification devices currently used for secondary  screening.
Since 2006, we have been reporting on issues associated with the cost and  performance of the ASPs and the lack of rigor in testing this equipment.  For example, we found that tests DNDO conducted in early 2007 used  biased test methods that enhanced the apparent performance of ASPs and  did not use critical CBP operating procedures that are fundamental to the  performance of current handheld radiation detectors. In addition, in 2008  we estimated the lifecycle cost of each standard cargo version of the ASP  (including deployment costs) to be about $822,000, compared with about  $308,000 for the PVT standard cargo portal, and the total program cost for  DNDO’s latest plan for deploying radiation portal monitors—which relies  on a combination of ASPs and PVTs and does not deploy radiation portal  monitors at all border crossings—to be about $2 billion.
Concerned about the performance and cost of the ASP monitors, Congress  required the Secretary of Homeland Security to certify that the monitors  will provide a “significant increase in operational effectiveness” before  DNDO obligates funds for full-scale ASP procurement. In response, CBP,  DNDO, and the DHS management directorate jointly issued criteria for  determining whether the new technology provides a significant increase in  operational effectiveness. The primary screening criteria require that the  new portal monitors detect potential threats as well as or better than  PVTs, show improved performance in detection of highly enriched  uranium (HEU), and reduce by 80 percent the number of innocent alarms  that are sent to secondary inspection. To meet the secondary screening  criteria, the new portal monitors must reduce the probability of  misidentifying special nuclear material (e.g., HEU and plutonium) and the  average time to conduct secondary screenings.
DNDO designed and coordinated a new series of tests, originally  scheduled to run from April 2008 through September 2008, to determine  whether the new portal monitors meet the certification criteria and are  ready for deployment. Key phases of this round of testing include  concurrent testing led by DNDO of the new and current equipment’s  ability to detect and identify threats and of ASPs’ readiness to be  integrated into operations for both primary and secondary screening at  ports of entry; field validation testing led by CBP at four northern and  southern border crossings and two seaports; and an independent  evaluation, led by the DHS Science and Technology Directorate at one of  the seaports, of the new portal monitors’ effectiveness and suitability.
In May 2009, we reported on the results of the then-current round of ASP  testing. The findings from that report were based on completed tests and  preliminary results available at the time. Testing on ASPs has continued  since that report was issued. Today my testimony will (1) discuss the  principal findings and recommendations from our May report and (2)  update those findings based on the results of DNDO’s July 2009 ASP field  validation testing. The findings we are presenting today are based on our  previous ASP reports and updated with information collected during  interviews with DNDO and CBP officials. We also reviewed testing results  in a report on the July 2009 tests from the ASP Field Validation Advisory  Panel, a panel made up of officials from CBP, DNDO, and a national  laboratory established to examine testing results and provide  recommendations. On November 12, 2009, we briefed DHS, CBP, and  DNDO officials on the findings of our updated work. During the briefing,  CBP and DNDO officials provided oral comments and offered additional  information and clarifications we included in this testimony as  appropriate. Both our prior work and our updated work were conducted  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to produce a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our statement today.
subsections: section_title: Improved Testing Rigor Discussed in Our May 2009 Report Demonstrates Limitations of ASPs
paragraphs: Our May 2009 report on the then-current round of ASP testing found that  DHS increased the rigor of ASP testing over that of previous tests, and that  a particular area of improvement was in the performance testing at the  Nevada Test Site, where DNDO compared the capability of ASP and  current-generation equipment to detect and identify nuclear and  radiological materials. For example, unlike in prior tests, the plan for the  2008 performance test stipulated that the contractors who developed the  equipment would not be involved in test execution. This improvement  addressed concerns we previously raised about the potential for bias and  provided increased credibility to the results. Nevertheless, based on the  following factors, in our report we questioned whether the benefits of the  new portal monitors justify the high cost:    The DHS criteria for a significant increase in operational effectiveness.
Our chief concern with the criteria is that they require only a marginal  improvement over current-generation portal monitors in the detection of  certain weapons-usable nuclear materials during primary screening.  DNDO considers detection of such materials to be a key limitation of  current-generation portal monitors. The marginal improvement required of  ASPs to meet the DHS criteria is problematic because the detection  threshold for the current-generation portal monitors does not specify a  level of radiation shielding that smugglers could realistically use. Officials  from the Department of Energy (DOE), which designed the threat  guidance DHS used to set the detection threshold, and national laboratory  officials told us that the current threshold is based not on an analysis of  the capabilities of potential smugglers to take effective shielding measures  but rather on the limited sensitivity of PVTs to detect anything more than  certain lightly shielded nuclear materials. DNDO officials acknowledge  that both the new and current-generation portal monitors are capable of  detecting certain nuclear materials only when unshielded or lightly  shielded. The marginal improvement in detection of such materials  required of ASPs is particularly notable given that DNDO has not  completed efforts to fine-tune PVTs’ software using a technique called  “energy windowing” that could improve the PVTs’ sensitivity to nuclear  materials. DNDO officials expect they can achieve small improvements in  sensitivity through energy windowing, but DNDO has not yet completed  efforts to fine-tune the PVTs’ software. In contrast to the marginal  improvement required in detection of certain nuclear materials, the  primary screening requirement to reduce the rate of innocent alarms by 80  percent could result in hundreds of fewer secondary screenings per day,  thereby reducing CBP’s workload. In addition, the secondary screening  criteria, which require ASPs to reduce the probability of misidentifying  special nuclear material by one-half, address the limitations of relatively  small handheld devices in consistently locating and identifying potential  threats in large cargo containers.
Results of performance testing and field validation. The results of  performance tests that DNDO presented to us were mixed, particularly in  the ASPs’ capability to detect certain shielded nuclear materials during  primary screening. The results of performance testing at the Nevada Test  Site showed that the new portal monitors detected certain nuclear  materials better than PVTs when shielding approximated DOE threat  guidance, which is based on light shielding. In contrast, differences in  system performance were less notable when shielding was slightly  increased or decreased: both the PVTs and ASPs were frequently able to  detect certain nuclear materials when shielding was below threat  guidance, and both systems had difficulty detecting such materials when  shielding was somewhat greater than threat guidance. With regard to  secondary screening, ASPs performed better than handheld devices in  identification of threats when masked by naturally occurring radioactive  material. However, the differences in the ability to identify certain shielded  nuclear materials depended on the level of shielding, with increasing levels  appearing to reduce any ASP advantages over the handheld identification  devices. Other phases of testing uncovered multiple problems in meeting  requirements for successfully integrating the new technology into  operations at ports of entry. Of the two ASP contractors participating in  the current round of testing, one has fallen behind due to severe problems  encountered during testing of ASPs’ readiness to be integrated into  operations at ports of entry (“integration testing”); the problems may  require that the vendor redo previous test phases to be considered for  certification. The other vendor’s system completed integration testing, but  CBP suspended field validation testing in January 2009 after 2 weeks  because of serious performance problems resulting in an overall increase  in the number of referrals for secondary screening compared with existing  equipment.
DNDO’s plans for computer simulations. As of May 2009, DNDO did not  plan to complete injection studies—computer simulations for testing the  response of ASPs and PVTs to simulated threat objects concealed in cargo  containers—prior to the Secretary of Homeland Security’s decision on  certification even though delays to the ASP test schedule have allowed  more time to conduct the studies. According to DNDO officials, injection  studies address the inability of performance testing to replicate the wide  variety of cargo coming into the United States and the inability to place  special nuclear material and other threat objects in cargo during field  validation. DNDO had earlier indicated that injection studies could provide  information comparing the performance of the two systems as part of the  certification process for both primary and secondary screening. However,  DNDO subsequently decided that performance testing would provide  sufficient information to support a decision on ASP certification. DNDO  officials said they would instead use injection studies to support effective  deployment of the new portal monitors.
Lack of an updated cost-benefit analysis. DNDO had not updated its cost- benefit analysis to take into account the results of ASP testing. An updated  analysis that takes into account the testing results, including injection  studies, might show that DNDO’s plan to replace existing equipment with  ASPs is not justified, particularly given the marginal improvement in  detection of certain nuclear materials required of ASPs and the potential  to improve the current-generation portal monitors’ sensitivity to nuclear  materials, most likely at a lower cost. DNDO officials said they were  updating the ASP cost-benefit analysis and planned to complete it prior to  a decision on certification by the Secretary of Homeland Security.
Our May report recommended that the Secretary of Homeland Security  direct DNDO to (1) assess whether ASPs meet the criteria for a significant  increase in operational effectiveness based on a valid comparison with  PVTs’ full performance potential and (2) revise the schedule for ASP  testing and certification to allow sufficient time for review and analysis of  results from the final phases of testing and completion of all tests,  including injection studies. We further recommended that, if ASPs are  certified, the Secretary direct DNDO to develop an initial deployment plan  that allows CBP to uncover and resolve any additional problems not  identified through testing before proceeding to full-scale deployment. DHS  agreed to a phased deployment that should allow time to uncover ASP  problems but disagreed with GAO’s other recommendations, which we  continue to believe remain valid.
section_title: Results from July 2009 Testing Raise Continuing Issues
paragraphs: The results of DNDO’s most recent round of field validation testing, which  it undertook in July 2009, after our May report was released, raise new  issues. In July 2009, DNDO resumed the field testing of ASPs at four CBP  ports of entry that it initiated in January 2009 but suspended because of  serious performance problems. However, the July tests also revealed ASP  performance problems, including two critical performance deficiencies.  First, the ASP monitors had an unacceptably high number of false positive  alarms for the detection of certain high-risk nuclear materials. According  to CBP officials, these false alarms are very disruptive in a port  environment in that any alarm for this type of nuclear material would  cause CBP to take enhanced security precautions because such materials  (1) could be used in producing an improvised nuclear device and (2) are  rarely part of legitimate or routine cargo. Furthermore, once receiving an  alarm for this type of nuclear material, CBP officers are required to  conduct a thorough secondary inspection to assure themselves that no  nuclear materials are present before permitting the cargo to enter the  country. Repeated false alarms for nuclear materials are also causes for  concern because such alarms could eventually have the effect of causing  CBP officers to doubt the reliability of the ASP and be skeptical about the  credibility of future alarms.
Secondly, during the July testing the ASP experienced a “critical failure,”  which stemmed from a problem with a key component of the ASP and  caused the ASP to shut down. Importantly, during this critical failure, the  ASP did not alert the CBP officer that it had shut down and was no longer  scanning cargo. As a result, were this not in a controlled testing  environment, the CBP officer would have permitted the cargo to enter the  country thinking the cargo had been scanned, when it had not. According  to CBP officials, resolving this issue is important in order to assure the  stability and security of the ASP.
In addition to these key performance problems, the ASP was not able to  reduce referrals to secondary inspection by 80 percent as required by the  DHS criteria for a significant increase in operational effectiveness.  According to the report from the ASP Field Validation Advisory Panel, a  panel made up of officials from CBP, DNDO, and a national laboratory, the  ASP was able to reduce referrals to secondary inspection by about 69  percent rather than the 80 percent as required by the DHS criteria.
While the performance of the ASP during the July field validation testing  raises issues about its potential readiness for deployment, DNDO’s  proposed solutions to address these performance problems raise  additional questions about whether this equipment will provide any  meaningful increase in the ability to detect certain nuclear materials.  Specifically, to address the problem of false positive alarms indicating the  presence of certain nuclear materials, according to DNDO officials, DNDO  has modified the ASP to make this equipment less sensitive to these  nuclear materials. While this may address the issue of false positive  alarms, it will also diminish the ASP capability of detecting a key high-risk  nuclear material. Since the ASP modification, DNDO conducted computer  simulations using a vendor-provided system called a “replay tool” to  examine the effect of the modification on the ASP’s performance.  According to DNDO officials, the replay tool demonstrated that the  modified ASP will still be able to detect certain nuclear materials better  than the PVT. However, at this point, DNDO does not plan to retest the  ASP at the Nevada Test Site where it can examine the effects of these  modifications using actual nuclear materials. As we reported earlier this  year, the results of the testing at the Nevada Test Site demonstrated that  the ASPs represented a marginal improvement in detecting certain nuclear  materials. By reducing the sensitivity to these materials and not retesting  the modified ASPs against actual nuclear materials, it is uncertain exactly  what improvement in detecting certain nuclear materials these costly  portal monitors are providing.
While DNDO is reducing the sensitivity of ASPs to certain nuclear  materials, it has yet to complete efforts to improve the PVT’s ability to  detect these same materials through energy windowing. For several years,  CBP officials have repeatedly urged DNDO officials to complete this  research. However, it was not apparent from our discussions with DNDO  officials if this effort is making meaningful progress with the development  of energy windowing or when it will be completed. Furthermore, CBP  officials stated that, depending on the outcome of this research, energy  windowing could be the more cost effective way to improve detection of  certain nuclear materials. In our view, ASPs being modified to diminish  their capabilities to detect certain nuclear materials raises questions about  whether energy windowing might be able to achieve a similar level of  performance against these same materials from the PVTs that are already  in place.
Beyond reducing the sensitivity of ASPs to certain nuclear materials,  DNDO also plans to address the issue of critical failures by, among other  things, installing an indicator light on the ASP that will alert CBP officers  that the ASP has experienced a mission-critical failure and is no longer  scanning cargo. While this should address the issue of CBP officers not  knowing that the ASP has suffered a critical failure, CBP officials stressed  to us the need for the ASP to be stable and secure enough to avoid these  shutdowns.
In closing, the issues raised by the results of the July 2009 field validation  tests provide even greater reason for DNDO to address recommendations  from our May 2009 report. In particular, we reiterate the importance of our  prior recommendation for DNDO to assess whether ASPs meet the criteria  for a significant increase in operational effectiveness based on a valid  comparison with PVTs’ full performance potential, given that the ASPs will  no longer be as effective in detecting certain nuclear materials.
Mr. Chairman, this completes my prepared statement. I would be happy to  respond to any questions that you or other Members of the Subcommittee  may have at this time.
section_title: GAO Contact and Staff Acknowledgments
paragraphs: For further information about this testimony, please contact me at (202)  512-3841 or aloisee@gao.gov. Dr. Timothy Persons (Chief Scientist), Ned  Woodward (Assistant Director), Joseph Cook, and Kevin Tarmann made  key contributions to this testimony. Kendall Childers, Karen Keegan, Carol  Kolarik, Jonathan Kucskar, Omari Norman, Alison O’Neill, and Rebecca  Shea also made important contributions.
This is a work of the U.S. government and is not subject to copyright protection in the  United States. The published product may be reproduced and distributed in its entirety  without further permission from GAO. However, because this work may contain  copyrighted images or other material, permission from the copyright holder may be  necessary if you wish to reproduce this material separately.