Combating Nuclear Smuggling: Additional Actions Needed to Ensure Adequate Testing of Next Generation Radiation Detection Equipment
Released: 2007-09-18T00:00:00

Letter

Background

The Department of Homeland Security Appropriations Act for Fiscal Year 2007 states that “none of the funds appropriated…shall be obligated for full scale procurement of monitors until the Secretary of Homeland Security has certified…that a significant increase in operational effectiveness will be achieved.” DNDO noted that certification would meet DHS guidelines for the review and approval of complex acquisitions. Specifically, DNDO stated that the Secretary’s decision would be made in the context of DHS “Key Decision Point 3,” which details the review and approval necessary for DHS acquisition programs to move from the “Capability Development and Demonstration” phase to the “Production and Deployment Phase.”

To meet the statutory requirement to certify the ASPs will provide a “significant increase in operational effectiveness,” and requirements outlined in DHS Management Directive 1400, DNDO, with input from subject matter experts, developed a series of tests intended to demonstrate, among other things, ASP performance and deployment readiness. The tests were conducted at several venues, including the Nevada Test Site, the New York Container Terminal, the Pacific Northwest National Laboratory, and five ports of entry. DNDO stated that its request for full-scale production approval would be based upon completed and documented results of these tests. To meet the Secretary’s goal of deploying 225 ASPs by the end of calendar year 2008, Secretarial Certification was scheduled for June 26, 2007.

To guide the test operations, DNDO defined a set of Critical Operational Issues that outlined the tests’ technical objectives and provided the baseline to measure demonstrated effectiveness. The purpose of the Critical Operational Issue 1 is to “verify operational effectiveness” of ASPs and determine whether “ASP systems significantly increase operational effectiveness relative to the current generation detection and identification system.” DNDO conducted a series of tests at the Nevada Test Site, the single focus of which, according to DNDO, was to resolve Critical Operational Issue 1. According to DNDO, these tests began in February 2007 and concluded in March 2007. DNDO’s Nevada Test Site test plan, dated January 12, 2007, identified three primary test objectives comparing the operational effectiveness of the ASP systems with existing detection and identification systems at current high-volume operational thresholds. Specifically, DNDO sought to determine the ASPs’ probability to (1) detect and identify nuclear and radiological threats (2) discriminate threat and non-threat radionuclides in primary , and (3) detect and identify threat radionuclides in the presence of non-threat radionuclides.

How the Tests at the Nevada Test Site Were Conducted

The Nevada Test Site test plan had two key components. First, DNDO developed guidelines for basic test operations and procedures, including test goals and expectations, test tasks and requirements, and roles and responsibilities of personnel involved in the testing, including the ASP contractors. The second component involved the National Institute of Standards and Technology developing test protocols that defined, among other things, how many times a container carrying test materials would need to be driven through portal monitors in order to obtain statistically relevant results.

DNDO’s tests at the Nevada Test Site were designed to compare the current system—using PVTs in primary inspections and a PVT and RIID combination in secondary inspections—to other configurations including PVTs in primary and ASPs in secondary, and ASPs in both primary and secondary inspection positions. DNDO tested three ASPs and four PVTs. The ASP vendors included Thermo, Raytheon, and Canberra. The PVT vendors included SAIC, TSA, and Ludlum. According to the test plan, to the greatest extent possible, PVT, ASP, and RIID handheld devices would be operated consistent with approved CBP standard operating procedures.

Prior to “formal” collection of the data that would be used to support the resolution of Critical Operational Issue 1, DNDO conducted a series of tests it referred to as “dry runs” and “dress rehearsals.” The purpose of the dry runs was to, among other things, verify ASP systems’ software performance against representative test materials and allow test teams and system contractors to identify and implement software and hardware improvements to ASP systems. The purpose of the dress rehearsals was to observe the ASPs in operation against representative test scenarios and allow the test team to, among other things: develop confidence in the reliability of the ASP system so that operators and data analysts would know what to expect and what data to collect during the formal test, collect sample test data, and determine what errors were likely to occur in the data collection process and eliminate opportunities for error.

In addition to improving ASP performance through dry runs and dress rehearsals conducted prior to formal data collection, ASP contractors were also significantly involved in the Nevada Test Site test processes. Specifically, the test plan stated that “ contractor involvement was an integral part of the NTS test events to ensure the systems performed as designed for the duration of the test.” Furthermore, ASP contractors were available on site to repair their system at the request of the test director and to provide quality control support of the test data through real time monitoring of available data. DNDO stated that Pacific Northwest National Laboratory representatives were also on site to provide the same services for the PVT systems.

DNDO conducted its formal tests in two phases. The first, called Phase 1, was designed to support resolution of Critical Operational Issue 1 with high statistical confidence. DNDO told us on multiple occasions and in a written response that only data collected during Phase 1 would be included in the final report presented to the Secretary to request ASP certification. According to DNDO, the second, called Phase 3, provided data for algorithm development which targeted specific and known areas in need of work and data to aid in the development of secondary screening operations and procedures. According to DNDO documentation, Phase 3 testing was not in support of the full-scale production decision. Further, DNDO stated that Phase 3 testing consisted of relatively small sample sizes since the data would not support estimating the probability of detection with a high confidence level.

On May 30, 2007, following the formal tests and the scoring of their results, DNDO told GAO that it had conducted additional tests that DNDO termed “Special Testing.” The details of these tests were not outlined in the Nevada Test Site test plan. On June 20, 2007, DNDO provided GAO with a test plan document entitled “ASP Special Testing” which described the test sources used to conduct the tests but did not say when the tests took place. According to DNDO, special testing was conducted throughout the formal Phase 1 testing process and included 12 combinations of threat, masking, and shielding materials that differed from “dry run,” “dress rehearsal,” and formal tests. DNDO also stated that the tests were “blind,” meaning that neither DNDO testing officials nor the ASP vendors knew what sources would be included in the tests. According to DNDO, these special tests were recommended by subject matter experts outside the ASP program to address the limitations of the original NTS test plan, including available time and funding resources, special nuclear material sources, and the number of test configurations that could be incorporated in the test plan, including source isotope and activity, shielding materials and thicknesses, masking materials, vehicle types, and measurement conditions.

Unlike the formal tests, National Institute of Standards and Technology officials were not involved in determining the number of test runs necessary to obtain statistically relevant results for the special tests.

DNDO’s Test Methods Raise Concerns Regarding the Reliability of Test Results

Based on our analysis of DNDO’s test plan, the test results, and discussions with experts from four national laboratories, we are concerned that DNDO used biased test methods that enhanced the performance of the ASPs. In the dry runs and dress rehearsals, DNDO conducted many preliminary runs of radiological, nuclear, masking, and shielding materials so that ASP contractors could collect data on the radiation being emitted, and modify their software accordingly. Specifically, we are concerned because almost all of the materials, and most combinations of materials, DNDO used in the formal tests were identical to those that the ASP contractors had specifically set their ASPs to identify during the dry runs and dress rehearsals. It is highly unlikely that such favorable circumstances would present themselves under real world conditions.

A key component of the NTS tests was to test the ASPs’ ability to detect and identify dangerous materials, specifically when that material was masked or “hidden” by benign radioactive materials. Based on our analysis, the masking materials DNDO used at NTS did not sufficiently test the performance limits of the ASPs. DOE national laboratory officials raised similar concerns to DNDO after reviewing a draft of the test plan in November 2006. These officials stated that the masking materials DNDO planned to use in its tests did not emit enough radiation to mask the presence of nuclear materials in a shipping container and noted that many of the materials that DOE program officials regularly observe passing through international ports emit significantly higher levels of radiation than the masking materials DNDO used for its tests.

DNDO officials told us that the masking materials used at the Nevada Test Site represented the average emissions seen in the stream of commerce at the New York Container Terminal. However, according to data accumulated as part of DOE’s program to secure international ports (the Megaports program), a significant percentage of cargo passing through one European port potentially on its way to the United States has emission levels greater than the average radiation level for cargo that typically sets off radiation detection alarms. Importantly, DNDO officials told us that the masking materials used at the Nevada Test Site were not intended to provide insight into the limits of ASP detection capabilities. Yet, DNDO’s own test plan for “ASP Special Testing” states, “The DNDO ASP NTS Test Plan was designed to… measure capabilities and limitations in current ASP systems.”

In addition, the NTS tests did not objectively test the ASPs against the currently deployed radiation detection system. DNDO’s test plan stated that, to the greatest extent possible, PVT, ASP, and RIID handheld devices would be operated consistent with approved CBP standard operating procedures. However, after analyzing test results and procedures used at the Nevada Test Site, CBP officials determined that DNDO had, in fact, not followed a key CBP procedure. In particular, if a threat is identified during a secondary screening, or if the result of the RIID screening isn’t definitive, CBP procedures require officers to send the data to CBP’s Laboratories and Scientific Services for further guidance. DNDO did not include this critical step in its formal tests. CBP officials also expressed concern with DNDO’s preliminary test results when we met with them in May 2007.

In regards to the special tests DNDO conducted, based on what DNDO has told us and our own evaluation of the special test plan, we note that because DNDO did not consult NIST on the design of the blind tests, we do not know the statistical significance of the results, and the tests were not entirely blind because some of the nuclear materials used in the blind tests were also used to calibrate the ASPs on a daily basis.

During the course of our work, CBP, DOE, and national laboratory officials we spoke to voiced concern about their lack of involvement in the planning and execution of the Nevada Test Site tests. We raised our concerns about this issue and those of DOE and CBP to DNDO’s attention on multiple occasions. In response to these concerns, specifically those posed by DOE, DNDO convened a conference on June 27, 2007, of technical experts to discuss the Nevada test results and the methods DNDO used to test the effects of masking materials on what the ASPs are able to detect. As a result of discussions held during that meeting, subject matter experts agreed that computer-simulated injection studies could help determine the ASPs’ ability to detect threats in the presence of highly radioactive masking material.

According to a Pacific Northwest National Laboratory report submitted to DNDO in December 2006, injection studies are particularly useful for measuring the relative performance of algorithms, but their results should not be construed as a measure of (system) vulnerability. To assess the limits of portal monitors’ capabilities, the Pacific Northwest National Laboratory report states that actual testing should be conducted using threat objects immersed in containers with various masking agents, shielding, and cargo. DNDO officials stated at the meeting that further testing could be scheduled, if necessary, to fully satisfy DOE concerns.

On July 20, 2007, DHS Secretary Chertoff notified certain members of the Congress that he planned to convene an independent expert panel to review DNDO’s test procedures, test results, associated technology assessments, and cost-benefit analyses to support the final decision to deploy ASPs. In making this announcement, Secretary Chertoff noted the national importance of developing highly effective radiation detection and identification capabilities as one of the main reasons for seeking an independent review of DNDO’s actions. On August 30, 2007, the DHS Undersecretary for Management recommended that the Secretary of Homeland Security delay Secretarial Certification of ASPs for an additional two months. According to DHS, the current delay is in order to provide CBP more time to field ASP systems, a concern CBP had raised early in our review.

Conclusions

Effectively detecting and identifying radiological or nuclear threats at U.S. borders and ports of entry is a vital matter of national security, and developing new and advanced technology is critical to U.S. efforts to prevent a potential attack. However, it is also critical to fully understand the strengths and weaknesses of any next generation radiation detection technology before it is deployed in the field and to know, to the greatest extent possible, when or how that equipment may fail.

In our view, the tests conducted by DNDO at the Nevada Test Site between February and March 2007 used biased test methods and were not an objective assessment of the ASPs’ performance capabilities. We believe that DNDO’s test methods—specifically, conducting dry runs and dress rehearsals with contractors prior to formal testing—enhanced the performance of the ASPs beyond what they are likely to achieve in actual use. Furthermore, the tests were not a rigorous evaluation of the ASPs’ capabilities, but rather a developmental demonstration of ASP performance under controlled conditions which did not test the limitations of the ASP systems.

As a result of DNDO’s test methods and the limits of the tests—including a need to meet a secretarial certification deadline and the limited configurations of special nuclear material sources, masking, and shielding materials used—we believe that the results of the tests conducted at the Nevada Test Site do not demonstrate a “significant increase in operational effectiveness” relative to the current detection system, and cannot be relied upon to make a full-scale production decision.

Recommendations

We recommend that the Secretary of Homeland Security take the following actions: Delay Secretarial Certification and full-scale production decisions of the ASPs until all relevant tests and studies have been completed and limitations to these tests and studies have been identified and addressed. Furthermore, results of these tests and studies should be validated and made fully transparent to DOE, CBP, and other relevant parties.

Once the tests and studies have been completed, evaluated, and validated, DHS should determine in cooperation with CBP, DOE, and other stakeholders including independent reviewers, if additional testing is needed.

If additional testing is needed, the Secretary should appoint an independent group within DHS, not aligned with the ASP acquisition process, to conduct objective, comprehensive, and transparent testing that realistically demonstrates the capabilities and limitations of the ASP system. This independent group would be separate from the recently appointed independent review panel.

Finally, the results of the tests and analyses should be reported to the appropriate congressional committees before large scale purchases of ASP’s are made.

Mr. Chairman, this concludes our prepared statement. We would be happy to respond to any questions you or other members of the subcommittee may have.

GAO Contact and Staff Acknowledgements

For further information about this testimony, please contact me, Gene Aloise, at (202) 512-3841 or at aloisee@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. Erika D. Carter, Alison O’Neill, Jim Shafer, Daren Sweeney, and Eugene Wisnoski made key contributions to this statement.

Related GAO Products

Combating Nuclear Smuggling: DHS’s Decision to Procure and Deploy the Next Generation of Radiation Detection Equipment Is Not Supported by Its Cost-Benefit Analysis. GAO-07-581T. Washington, D.C.: March.14, 2007.

Nuclear Nonproliferation: Focusing on the highest Priority Radiological Sources Could Improve DOE’s Efforts to Secure Sources in Foreign Countries. GAO-07-580T. Washington, D.C.: March. 13, 2007.

Combating Nuclear Smuggling: DNDO Has Not Yet Collected Most of the National Laboratories’ Test Results on Radiation Portal Monitors in Support of DNDO’s Testing and Development Program. GAO-07-347R. Washington, D.C.: March 9, 2007.

Technology Assessment: Securing the Transport of Cargo Containers. GAO-06-68SU. Washington, D.C.: January 25, 2006.

Combating Nuclear Smuggling: DHS’s Cost-Benefit Analysis to Support the Purchase of New Radiation Detection Portal Monitors Was Not Based on Available Performance Data and Did Not Fully Evaluate All the Monitors’ Costs and Benefits. GAO-07-133R. Washington, D.C.: October 17, 2006.

Combating Nuclear Terrorism: Federal Efforts to Respond to Nuclear and Radiological Threats and to Protect Emergency Response Capabilities Could Be Strengthened. GAO-06-1015. Washington, D.C.: September 21, 2006.

Border Security: Investigators Transported Radioactive Sources Across Our Nation’s Borders at Two Locations. GAO-06-940T. Washington, D.C.: July 7, 2006.

Combating Nuclear Smuggling: Challenges Facing U.S. Efforts to Deploy Radiation Detection Equipment in Other Countries and in the United States. GAO-06-558T. Washington, D.C.: March 28, 2006.

Combating Nuclear Smuggling: DHS Has Made Progress Deploying Radiation Detection Equipment at U.S. Ports-of-Entry, but Concerns Remain. GAO-06-389. Washington, D.C.: March 22, 2006.

Combating Nuclear Smuggling: Corruption, Maintenance, and Coordination Problems Challenge U.S. Efforts to Provide Radiation Detection Equipment to Other Countries. GAO-06-311. Washington, D.C.: March 14, 2006.

Combating Nuclear Smuggling: Efforts to Deploy Radiation Detection Equipment in the United States and in Other Countries. GAO-05-840T. Washington, D.C.: June 21, 2005.

Preventing Nuclear Smuggling: DOE Has Made Limited Progress in Installing Radiation Detection Equipment at Highest Priority Foreign Seaports. GAO-05-375. Washington, D.C.: March 31, 2005.

Homeland Security: DHS Needs a Strategy to Use DOE’s Laboratories for Research on Nuclear, Biological, and Chemical Detection and Response Technologies. GAO-04-653. Washington, D.C.: May 24, 2004.

Homeland Security: Summary of Challenges Faced in Targeting Oceangoing Cargo Containers for Inspection. GAO-04-557T. Washington, D.C.: March 31, 2004).

Homeland Security: Preliminary Observations on Efforts to Target Security Inspections of Cargo Containers. GAO-04-325T. Washington, D.C.: December 16, 2003.

Homeland Security: Radiation Detection Equipment at U.S. Ports of Entry. GAO-03-1153TNI. Washington, D.C.: September 30, 2003.

Homeland Security: Limited Progress in Deploying Radiation Detection Equipment at U.S. Ports of Entry. GAO-03-963. Washington, D.C.: September 4, 2003).

Container Security: Current Efforts to Detect Nuclear Materials, New Initiatives, and Challenges. GAO-03-297T. Washington, D.C.: November 18, 2002.

Customs Service: Acquisition and Deployment of Radiation Detection Equipment. GAO-03-235T. Washington, D.C.: October 17, 2002.

Nuclear Nonproliferation: U.S. Efforts to Combat Nuclear Smuggling. GAO-02-989T. Washington, D.C.: July 30, 2002.

Nuclear Nonproliferation: U.S. Efforts to Help Other Countries Combat Nuclear Smuggling Need Strengthened Coordination and Planning. GAO-02-426. Washington, D.C.: May 16, 2002.

This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.