Combating Nuclear Smuggling: Lessons Learned from DHS Testing of Advanced Radiation Detection Portal Monitors
Released: 2009-06-25T00:00:00

Letter

The Latest Round of Testing Highlights the Limitations of ASPs

Our report on the latest round of ASP testing found that DHS increased the rigor of ASP testing in comparison with previous tests and that a particular area of improvement was in the performance testing at the Nevada Test Site, where DNDO compared the capability of ASP and current-generation equipment to detect and identify nuclear and radiological materials. For example, unlike in prior tests, the plan for the 2008 performance test stipulated that there would be no system contractor involvement in test execution. Such improvements addressed concerns we previously raised about the potential for bias and provided credibility to the results.

Nevertheless, based on the following factors, we continue to question whether the benefits of the new portal monitors justify the high cost: The DHS criteria for a significant increase in operational effectiveness.

Our chief concern with the criteria is that they require a marginal improvement over current-generation portal monitors in the detection of certain weapons-usable nuclear materials when ASPs are deployed for primary screening. DNDO considers detection of such materials to be a key limitation of current-generation portal monitors. We are particularly concerned about the marginal improvement required of ASPs because the detection threshold for the current-generation portal monitors does not specify a level of radiation shielding that smugglers could realistically use. DOE and national laboratory officials told us that DOE’s threat guidance used to set the current detection threshold is based not on an analysis of the capabilities of potential smugglers to take effective shielding measures but rather on the limited sensitivity of PVTs to detect anything more than certain lightly shielded nuclear materials. DNDO officials acknowledge that both the new and current-generation portal monitors are capable of detecting certain nuclear materials only when unshielded or lightly shielded. The marginal improvement in detection of such materials required of ASPs is particularly notable given that DNDO has not completed efforts to fine-tune PVTs’ software and thereby improve sensitivity to nuclear materials. DNDO officials expect they can achieve small improvements in sensitivity, but DNDO has not yet funded efforts to fine-tune PVTs’ software. In contrast to the marginal improvement required in detection of certain nuclear materials, the primary screening requirement to reduce the rate of innocent alarms could result in hundreds of fewer secondary screenings per day, thereby reducing CBP’s workload and delays to commerce. In addition, the secondary screening criteria, which require ASPs to reduce the probability of misidentifying special nuclear material by one-half, address the inability of relatively small handheld devices to consistently locate and identify potential threats in large cargo containers.

Preliminary results of performance testing and field validation. The preliminary results presented to us by DNDO are mixed, particularly in the capability of ASPs used for primary screening to detect certain shielded nuclear materials. Preliminary results show that the new portal monitors detected certain nuclear materials better than PVTs when shielding approximated DOE threat guidance, which is based on light shielding. In contrast, differences in system performance were less notable when shielding was slightly increased or decreased: Both the PVTs and ASPs were frequently able to detect certain nuclear materials when shielding was below threat guidance, and both systems had difficulty detecting such materials when shielding was somewhat greater than threat guidance. With regard to secondary screening, ASPs performed better than handheld devices in identification of threats when masked by naturally occurring radioactive material. However, differences in the ability to identify certain shielded nuclear materials depended on the level of shielding, with increasing levels appearing to reduce any ASP advantages over the handheld identification devices. Other phases of testing uncovered multiple problems in meeting requirements for successfully integrating the new technology into operations at ports of entry. Of the two ASP vendors participating in the 2008 round of testing, one has fallen behind due to severe problems encountered during testing of ASPs’ readiness to be integrated into operations at ports of entry (“integration testing”); the problems may require that the vendor redo previous test phases to be considered for certification. The other vendor’s system completed integration testing, but CBP suspended field validation after 2 weeks because of serious performance problems resulting in an overall increase in the number of referrals for secondary screening compared with existing equipment.

DNDO’s plans for computer simulations. DNDO does not plan to complete injection studies—computer simulations for testing the response of ASPs and PVTs to simulated threat objects concealed in cargo containers—prior to the Secretary of Homeland Security’s decision on certification even though delays to the ASP test schedule have allowed more time to conduct the studies. According to DNDO officials, injection studies address the inability of performance testing to replicate the wide variety of cargo coming into the United States and the inability to place special nuclear material and other threat objects in cargo during field validation. DNDO had earlier indicated that injection studies could provide information comparing the performance of the two systems as part of the certification process for both primary and secondary screening. However, DNDO subsequently decided that performance testing would provide sufficient information to support a decision on ASP certification. DNDO officials said they would instead use injection studies to support effective deployment of the new portal monitors.

Lack of an updated cost-benefit analysis. DNDO has not yet updated its cost-benefit analysis to take into account the results of the latest round of ASP testing. An updated analysis that takes into account the results from the latest round of testing, including injection studies, might show that DNDO’s plan to replace existing equipment with ASPs is not justified, particularly given the marginal improvement in detection of certain nuclear materials required of ASPs and the potential to improve the current-generation portal monitors’ sensitivity to nuclear materials, most likely at a lower cost. DNDO officials said they are currently updating the ASP cost-benefit analysis and plan to complete it prior to a decision on certification by the Secretary of Homeland Security.

Our report recommended that the Secretary of Homeland Security direct DNDO to (1) assess whether ASPs meet the criteria for a significant increase in operational effectiveness based on a valid comparison with PVTs’ full performance potential and (2) revise the schedule for ASP testing and certification to allow sufficient time for review and analysis of results from the final phases of testing and completion of all tests, including injection studies. We further recommended that, if ASPs are certified, the Secretary direct DNDO to develop an initial deployment plan that allows CBP to uncover and resolve any additional problems not identified through testing before proceeding to full-scale deployment. DHS agreed to a phased deployment that should allow time to uncover ASP problems but disagreed with GAO’s other recommendations, which we continue to believe remain valid.

Procurement Decisions for New Technologies Require Rigorous Testing and Thorough Analysis of Results

The challenges DNDO has faced in developing and testing ASPs illustrate the importance of following existing DHS policies as well as best practices for investments in complex homeland security acquisitions and for testing of new technologies. The DHS investment review process calls for executive decision making at key points in an investment’s life cycle and includes many acquisition best practices that, if applied consistently, could help increase the chances for successful outcomes. However, we reported in November 2008 that, for the period from fiscal year 2004 through the second quarter of fiscal year 2008, DHS had not effectively implemented or adhered to its investment review process due to a lack of senior management officials’ involvement as well as limited monitoring and resources. In particular, of DHS’s 48 major investments requiring milestone and annual reviews under the department’s investment review policy, 45 were not assessed in accordance with this policy. In addition, many major investments, including DNDO’s ASP program, had not met the department’s requirements for basic acquisition documents necessary to inform the investment review process. As a result, DHS had not consistently provided the oversight needed to identify and address cost, schedule, and performance problems in its major investments. Among other things, our November 2008 report recommended that the Secretary of Homeland Security direct component heads, such as the Director of DNDO, to ensure that the components have established processes to manage major investments consistent with departmental policies. DHS generally concurred with our recommendations, and we noted that DHS had begun several efforts to address shortcomings in the investment review process identified in our report, including issuing an interim directive requiring DHS components to align their internal policies and procedures by the end of the third quarter of fiscal year 2009. In January 2009, DHS issued a memorandum instructing component heads to create acquisition executives in their organizations to be responsible for management and oversight of component acquisition processes. If fully implemented, these steps should help ensure that DHS components have established processes to manage major investments.

Based on our body of work on ASP testing, one of the primary lessons to be learned is to avoid the pitfalls in testing that stem from a rush to procure new technologies. GAO has previously reported on the negative consequences of pressures imposed by closely linking testing and development programs with decisions to procure and deploy new technologies, including the creation of incentives to postpone difficult tests and limit open communication about test results. We found that testing programs designed to validate a product’s performance against increasing standards for different stages in product development are a best practice for acquisition strategies for new technologies. In the case of ASPs, the push to replace existing equipment with the new portal monitors led to a testing program that until recently lacked the necessary rigor. Even for the most recent round of testing, DNDO’s schedule consistently underestimated the time required to conduct tests, resolve problems uncovered during testing, and complete key documents, including final test reports. In addition, DNDO’s original working schedule did not anticipate the time required to update its cost-benefit analysis to take into account the latest test results. The schedule anticipated completion of testing in mid-September 2008 and the DHS Secretary’s decision on ASP certification between September and November 2008. However, testing is still not completed, and DNDO took months longer than anticipated to complete the final report on performance testing.

As previously mentioned, a number of aspects of the latest round of ASP testing increased the rigor in comparison with earlier rounds and, if properly implemented, could improve the rigor in DHS’s testing of other advanced technologies. Key aspects included the following: Criteria for ensuring test requirements are met. The test and evaluation master plan established criteria requiring that the ASPs meet certain requirements before starting or completing any test phase. For example, the plan required that ASPs have no critical or severe issues rendering them completely unusable or impairing their function. The criteria provided a formal means to ensure that ASPs met certain basic requirements prior to the start of each phase of testing. DNDO and CBP adhered to the criteria even though doing so resulted in integration testing taking longer than anticipated and delaying the start of field validation.

Participation of the technology end user. The participation of CBP (the end user of the new portal monitors) provided an independent check, within DHS, of DNDO’s efforts to develop and test the new portal monitors. For example, CBP added a final requirement to integration testing before proceeding to field validation to demonstrate ASPs’ ability to operate for 40 hours without additional problems and thereby provide for a productive field validation. In addition, the participation of CBP officers in the 2008 round of performance testing allowed DNDO to adhere more closely than in previous tests to CBP’s standard operating procedure for conducting a secondary inspection using the handheld identification devices, thereby providing for an objective test.

Participation of an independent test authority. The DHS Science and Technology Directorate, which is responsible for developing and implementing the department’s test and evaluation policies and standards, will have the lead role in the final phase of ASP testing and thereby provide an additional independent check on testing efforts. The Science and Technology Directorate identified two critical questions, related to ASPs’ operational effectiveness (i.e., detection and identification of threats) and suitability (e.g., reliability, maintainability, and supportability), and drafted its own test plan to address those questions.

Mr. Chairman, this completes my prepared statement. I would be happy to respond to any questions that you or other Members of the Subcommittee may have at this time.

GAO Contact and Staff Acknowledgments

For further information about this testimony, please contact me at (202) 512-3841 or aloisee@gao.gov. Ned Woodward (Assistant Director), Joseph Cook, and Kevin Tarmann made key contributions to this testimony. Dr. Timothy Persons (Chief Scientist), James Ashley, Steve Caldwell, John Hutton, Omari Norman, Alison O’Neill, Amelia Shachoy, and Rebecca Shea also made important contributions.

Appendix I: Key Findings and Recommendations from Related GAO Products on Testing and Development of ASPs

Combating Nuclear Smuggling: DHS Has Made Progress Deploying Radiation Detection Equipment at U.S. Ports-of-Entry, but Concerns Remain. GAO-06-389. Washington, D.C.: March 22, 2006.

Key findings. Prototypes of advanced spectroscopic portals (ASP) were expected to be significantly more expensive than current-generation portal monitors but had not been shown to be more effective. For example, Domestic Nuclear Detection Office (DNDO) officials’ preliminary analysis of 10 ASPs tested at the Nevada Test Site found that the new portal monitors outperformed current-generation equipment in detecting numerous small, medium-size, and threatlike radioactive objects and were able to identify and dismiss most naturally occurring radioactive material. However, the detection capabilities of both types of portal monitors converged as the amount of source material decreased.

Recommendations. We recommended that the Secretary of Homeland Security work with the Director of DNDO to analyze the benefits and costs of deploying ASPs before any of the new equipment is purchased to determine whether any additional detection capability is worth the additional cost. We also recommended that the total program cost estimate for the radiation portal monitor project be revised after completion of the cost-benefit analysis.

Combating Nuclear Smuggling: DHS’s Cost-Benefit Analysis to Support the Purchase of New Radiation Detection Portal Monitors Was Not Based on Available Performance Data and Did Not Fully Evaluate All the Monitors’ Costs and Benefits. GAO-07-133R. Washington, D.C.: October 17, 2006.

Combating Nuclear Smuggling: DHS’s Decision to Procure and Deploy the Next Generation of Radiation Detection Equipment Is Not Supported by Its Cost-Benefit Analysis. GAO-07-581T. Washington, D.C.: March 14, 2007.

Key findings. DNDO’s cost-benefit analysis issued in response to our March 2006 recommendation did not provide a sound analytical basis for DNDO’s decision to purchase and deploy ASPs. We identified a number of problems with the analysis of both the performance of the new portal monitors and the costs. With regard to performance, DNDO did not use the results of its own tests and instead relied on assumptions of the new technology’s anticipated performance level. In addition, the analysis focused on identifying highly enriched uranium (HEU) and did not consider how well the new portal monitors can correctly detect or identify other dangerous radiological or nuclear materials. With regard to costs, DNDO did not follow the DHS guidelines for performing cost-benefit analyses and used questionable assumptions about the procurement costs of portal monitor technology.

Recommendations. We recommended that DHS and DNDO conduct a new cost-benefit analysis using sound analytical methods, including actual performance data and a complete accounting of all major costs and benefits as required by DHS guidelines, and that DNDO conduct realistic testing for both ASPs and current-generation portal monitors.

Combating Nuclear Smuggling: DNDO Has Not Yet Collected Most of the National Laboratories’ Test Results on Radiation Portal Monitors in Support of DNDO’s Testing and Development Program. GAO-07-347R. Washington, D.C.: March 9, 2007.

Key findings. DNDO had not collected a comprehensive inventory of testing information on current-generation portal monitors. Such information, if collected and used, could improve DNDO’s understanding of how well portal monitors detect different radiological and nuclear materials under varying conditions. In turn, this understanding would assist DNDO’s future testing, development, deployment, and purchases of portal monitors.

Recommendations. We recommended that the Secretary of Homeland Security, working with the Director of DNDO, collect reports concerning all of the testing of current-generation portal monitors and review the test reports in order to develop an information database on how the portal monitors perform in both laboratory and field tests on a variety of indicators, such as their ability to detect specific radiological and nuclear materials.

Combating Nuclear Smuggling: Additional Actions Needed to Ensure Adequate Testing of Next Generation Radiation Detection Equipment. GAO-07-1247T. Washington, D.C.: September 18, 2007.

Key findings. We found that tests conducted by DNDO in early 2007 were not an objective and rigorous assessment of the ASPs’ capabilities. Specifically, we raised concerns about DNDO using biased test methods that enhanced the apparent performance of ASPs; not testing the limitations of ASPs’ detection capabilities—for example, by not using a sufficient amount of the type of materials that would mask or hide dangerous sources and that ASPs would likely encounter at ports of entry; and not using a critical Customs and Border Protection (CBP) standard operating procedure that is fundamental to the performance of handheld radiation detectors in the field.

Recommendations. We recommended that the Secretary of Homeland Security delay Secretarial certification and full-scale production decisions on ASPs until all relevant tests and studies had been completed and limitations to tests and studies had been identified and addressed. We further recommended that DHS determine the need for additional testing in cooperation with CBP and other stakeholders and, if additional testing was needed, that the Secretary of DHS appoint an independent group within DHS to conduct objective, comprehensive, and transparent testing that realistically demonstrates the capabilities and limitations of ASPs.

Combating Nuclear Smuggling: DHS’s Program to Procure and Deploy Advanced Radiation Detection Portal Monitors Is Likely to Exceed the Department’s Previous Cost Estimates. GAO-08-1108R. Washington, D.C.: September 22, 2008.

Key findings. Our independent cost estimate suggested that from 2007 through 2017 the total cost of DNDO’s 2006 project execution plan (the most recent official documentation of the program to equip U.S. ports of entry with radiation detection equipment) would likely be about $3.1 billion but could range from $2.6 billion to $3.8 billion. In contrast, we found that DNDO’s cost estimate of $2.1 billion was unreliable because it omitted major project costs, such as elements of the ASPs’ life cycle, and relied on a flawed methodology. DNDO officials told us that the agency was no longer following the 2006 project execution plan and that the scope of the agency’s ASP deployment strategy had been reduced to only the standard cargo portal monitor. Our analysis of DNDO’s summary information outlining its scaled-back plan indicated the total cost to deploy standard cargo portals over the period 2008 through 2017 would be about $2 billion but could range from $1.7 billion to $2.3 billion. Agency officials acknowledged the program requirements that would have been fulfilled by the discontinued ASPs remained valid, including screening rail cars and airport cargo, but the agency had no plans for how such screening would be accomplished.

Recommendations. We recommended that the Secretary of Homeland Security direct the Director of DNDO to work with CBP to update the projection execution plan to guide the entire radiation detection program at U.S. ports of entry, revise the estimate of the program’s cost and ensure that the estimate considers all of the costs associated with its project execution plan, and communicate the revised estimate to Congress so that it is fully apprised of the program’s scope and funding requirements.

Combating Nuclear Smuggling: DHS Needs to Consider the Full Costs and Complete All Tests Prior to Making a Decision on Whether to Purchase Advanced Portal Monitors. GAO-08-1178T. Washington, D.C.: September 25, 2008.

Key findings. In preliminary observations of the 2008 round of ASP testing, we found that DNDO had made progress in addressing a number of problems we identified in previous rounds of ASP testing. However, the DHS criteria for significant increase in operational effectiveness appeared to set a low bar for improvement—for example, by requiring ASPs to perform at least as well as current-generation equipment when nuclear material is present in cargo but not specifying an actual improvement. In addition, the ASP certification schedule did not allow for completion of computer simulations that could provide useful data on ASP capabilities prior to the Secretary’s decision on certification. Finally, we questioned the replacement of current-generation equipment with ASPs until DNDO demonstrates that any additional increase in security would be worth the ASPs’ much higher cost.

Combating Nuclear Smuggling: DHS’s Phase 3 Test Report on Advanced Portal Monitors Does Not Fully Disclose the Limitations of the Test Results. GAO-08-979. Washington, D.C.: September 30, 2008.

Key findings. DNDO’s report on the second group of ASP tests in 2007 (the Phase 3 tests) did not appropriately state test limitations. As a result, the report did not accurately depict the results and could potentially be misleading. The purpose of the Phase 3 tests was to conduct a limited number of test runs in order to identify areas in which the ASP software needed improvement. While aspects of the Phase 3 report addressed this purpose, the preponderance of the report went beyond the test’s original purpose and made comparisons of the performance of the ASPs with one another or with currently deployed portal monitors. We found that it would not be appropriate to use the Phase 3 test report in determining whether the ASPs represent a significant improvement over currently deployed radiation equipment because the limited number of test runs did not support many of the comparisons of ASP performance made in the report.

Recommendations. We recommended that the Secretary of DHS use the results of the Phase 3 tests solely for the purposes for which they were intended—to identify areas needing improvement—and not as a justification for certifying whether the ASPs warrant full-scale production. If the Secretary intends to consider the results of the Phase 3 tests in making a certification decision regarding ASPs, we further recommended that the Secretary direct the Director of DNDO to revise and clarify the Phase 3 test report to more fully disclose and articulate the limitations present in the Phase 3 tests and clearly state which insights from the Phase 3 report are factored into any decision regarding the certification that ASPs demonstrate a significant increase in operational effectiveness. Finally, we recommended that the Secretary direct the Director of DNDO to take steps to ensure that any limitations associated with the 2008 round of testing are properly disclosed when the results are reported.

Combating Nuclear Smuggling: DHS Improved Testing of Advanced Radiation Detection Portal Monitors, but Preliminary Results Show Limits of the New Technology. GAO-09-655. Washington, D.C.: May 21, 2009.

Key findings. We reported that the DHS criteria for a significant increase in operational effectiveness require a large reduction in innocent alarms but a marginal improvement in the detection of certain weapons-usable nuclear materials. In addition, the criteria do not take the current- generation portal monitors’ full potential into account because DNDO has not completed efforts to improve their performance. With regard to ASP testing, we found that DHS increased the rigor in comparison with previous tests, thereby adding credibility to the test results, but that preliminary results were mixed. The results showed that the new portal monitors performed better than current-generation portal monitors in detection of certain nuclear materials concealed by light shielding approximating the threat guidance for setting detection thresholds, but differences in sensitivity were less notable when shielding was slightly below or above that level. Testing also uncovered multiple problems in ASPs meeting the requirements for successful integration into operations at ports of entry. Finally, we found that DNDO did not plan to complete computer simulations that could provide additional insight into ASP capabilities and limitations prior to certification even though delays to testing allowed more time to conduct the simulations.

Recommendations. We recommended that the Secretary of Homeland Security direct the Director of DNDO to assess whether ASPs meet the criteria for a significant increase in operational effectiveness based on a valid comparison with current-generation portal monitors’ full performance potential and revise the schedule for ASP testing and certification to allow sufficient time for review and analysis of results from the final phases of testing and completion of all tests, including computer simulations. If ASPs are certified, we further recommended that the Secretary of Homeland Security direct the Director of DNDO to develop an initial deployment plan that allows CBP to uncover and resolve any additional problems not identified through testing before proceeding to full-scale deployment.

This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.