Combating Nuclear Smuggling: DHS's Phase 3 Test Report on Advanced Portal Monitors Does Not Fully Disclose the Limitations of the Test Results
Released: 2008-10-30T00:00:00

Letter

Background

In the summer of 2005, DNDO tested ASPs from 10 vendors to evaluate their performance capabilities and to select the ASPs that warranted further development and possible procurement. In July 2006, DNDO awarded contracts totaling $1.2 billion over fiveyears to three vendors— Raytheon, Canberra, and Thermo.

The Department of Homeland Security Appropriations Act for Fiscal Year 2007 states that “none of the funds appropriated … shall be obligated for full scale procurement of monitors until the Secretary of Homeland Security has certified … that a significant increase in operational effectiveness will be achieved.” Congress enacted a similar requirement in DHS’s fiscal year 2008 appropriation. In hopes of obtaining secretarial certification by June 2007, DNDO tested ASPs at several sites, including the Nevada Test Site, the New York Container Terminal, the Pacific Northwest National Laboratory, and five ports of entry. DNDO conducted the tests at NTS in two phases. DNDO stated that the Phase 1 tests, performed in February-March 2007, attempted to estimate the performance capabilities of the ASPs with a high degree of statistical confidence. DNDO intended these tests to support the Secretary’s decision on whether to certify the ASPs for the purposes of a full-scale production decision, while the Phase 3 tests were intended to help improve the computer algorithms that the ASPs use to identify the specific radiological or nuclear source inside a container.

On September 18, 2007, we testified that DNDO’s Phase 1 tests did not constitute an objective and rigorous assessment of the ASPs’ capabilities because, among other things, DNDO conducted preliminary test runs on source materials to be used in the tests, and then allowed the vendors to adjust their ASPs to specifically identify the source materials to be tested. We testified that in our view, DNDO’s approach biased the tests in ways that enhanced the apparent performance of the ASPs. We also noted that the tests did not attempt to estimate the limits of ASPs’ detection abilities—an important concern to those who will use them such as CBP officers. During that hearing, DNDO’s Director stated that, contrary to statements DNDO made in its final Phase 3 test plan, DNDO would use the Phase 3 test results to help support the Secretary’s decision on whether to certify the ASPs for full-scale production. Subsequently, DNDO delayed its anticipated date for secretarial certification to the fall of 2008 in order to conduct additional performance tests and field tests during fiscal year 2008.

DNDO’s Phase 3 Test Report Frequently Overlooks the Limitations Associated with the Tests’ Small Sample Sizes

Because the limitations of the Phase 3 test results are not properly discussed in the Phase 3 test report, the report does not accurately portray the results from the Phase 3 tests and could be misleading. The purpose of the Phase 3 tests was to identify areas in which the ASPs needed improvement. While some of the Phase 3 report addresses this purpose, much of the report compares the performance of the ASPs with each other or with PVTs and RIIDs during the tests. However, because DNDO performed each test a limited number of times, the data it used to make some of these comparisons provide little information about the actual capabilities of the ASPs. The narrative of the report often presents each test result as a single value, although, because of the limited number of test runs, the results would be more thoroughly and appropriately stated as a range of potential values. In addition, the report’s narrative sometimes omits key facts that conflict with DNDO’s analysis of the results.

The Phase 3 Test Report Largely Overlooks the Limiting Effects of Performing a Small Number of Tests

“The primary goals of the testing were to provide information by giving the ASP systems an opportunity to perform against a wider range of radionuclides, shielding, and cargo configurations. To allow for more substantial variation among test cases, the test plan called for a smaller number of trials over a larger range of objects and configurations rather than running sufficient number of repetitions for each individual test case to provide higher statistically significan results.” (p. 2)

In these comparisons , results are small sample sizes induce large uncertainties in the estimates of the probabilities being compared [for example: n ≤ 5].” (p.9) “For at 2 mph, the ASP system performances are statistically indistinguishable.” (p.13) “For shielded , performance for all three systems is statistically indistinguishable with probabilities of correct alarm varying approximately between 0.84 and .0.92.” (p.11)

The statements imply that the performances of the ASPs were similar because the results were “statistically indistinguishable.” However, given the small number of test runs, it is impossible to determine with a high degree of confidence whether or not the performances were actually similar. Yet the report’s text describing specific results rarely qualifies the results by stating that the test was run only a few times or that the results should not be considered conclusive of the ASPs’ capabilities. “For the source configurations tested, the ASP systems have equal performance down to the lowest source activity tested.” (p. iii) “The PVT systems display lower performance than the ASP systems for and sources.” (p. iv) “When comparing the ASP systems _ mph identification metric with the _ RIID measurements…, it is observed that the RIID performance is poor compared to the ASP systems.” (p. iv)

Report Text Often Omits the Range of Values Surrounding Each Test Result

“For bare only, … he probability of correct identification varied between 0.34 and 0.5.” (p.14)

However, because each test involved a small number of test runs, these percentages provide little information about the performance capabilities of the ASPs. In fact, because of the small number of test runs, DNDO can only estimate that each ASP can correctly identify the type of source material within a range of values. The fewer the number of test runs, the larger the range. For example, for the ASP that correctly identified the source material 34 percent of the time during the tests, the report text omits the fact that, as shown on an accompanying graph, DNDO can only estimate that the ASP would be able to correctly identify the source between about 10 percent and 65 percent of the time. By stating that the ASP identified the source 34 percent of the time without clarifying that the results came from only a few test runs, the report’s text makes the test results seem more precise than they really are. Similarly, for the ASP that correctly identified the source material 50 percent of the time during the tests, the small number of test runs means that DNDO can only estimate that the ASP would be able to correctly identify the source material between about 15 percent and 85 percent of the time. This range is too wide to have much value in determining how well the ASP may perform in the real world. Although these ranges are clearly shown on the report’s graphs, they are omitted in the report’s descriptions and interpretations of the test results.

Similarly, DNDO’s analysis comparing the performances of ASPs and RIIDs fails to consider the uncertainties created by the tests’ small sample sizes. The report states that the RIIDs “performance is poor compared to the ASP systems.” For example, during the tests, one vendor’s ASP correctly identified one type of source material about 50 percent of the time, while the RIIDs correctly identified the same type of source material about 10 percent of the time. However, given the small number of test runs, DNDO cannot be confident that these percentages precisely indicate the performance capabilities of the ASPs and RIIDs. On the basis of the tests, DNDO can only infer that the ASPs’ and RIIDs’ performance capabilities lie somewhere within a relatively large range of values. As these ranges are illustrated in the report’s graphs, it appears that the difference in the performance of the ASPs and RIIDs may not be statistically different for three of the five types of source materials DNDO tested. This does not necessarily mean that the ASPs and RIIDs performed equally well; rather, DNDO did not conduct each test enough times to determine that the superior performance of the ASPs over the RIIDs reflects the capabilities of the ASPs rather than mere chance.

DNDO’s Phase 3 Test Report Omits Important Details that Affect the Interpretation of the Test Results

“The ASP systems demonstrated detection limits equal to or better than those of any of the PVT systems as configured during testing.” (p.iii)

However, the report’s executive summary fails to note that because DNDO used only one type of source material, the results are largely specific to that particular source material and would not necessarily apply to other specific source materials. In fact, for other types of source material, the report shows several instances in which the PVTs were apparently able to detect other types of source materials better than the ASPs. Moreover, other Phase 3 tests showed that simply moving the source material referred to in the above quote to another place in the container affected the relative performances of the ASPs and PVTs. “the ASP systems have the ability to identify sources when placed inside almost all but the thickest shielding configuration tested.” (p.iv)

Again, however, DNDO fails to note in its report that, as it explained in its Phase 3 test plan, all the shielding used in the Phase 3 tests represented “light shielding.” The report also fails to state how many specific sources the ASPs could correctly identify or how frequently the ASPs could identify them.

Phase 3 Test Results Provide Little Evidence as To Whether ASPs Represent an Improvement Over Currently Deployed Technology

In our view, it is not appropriate to use the Phase 3 test report in determining whether the ASPs represent a significant improvement over currently deployed radiation equipment because the limited number of test runs does not support many of the comparisons of ASP performance made in the Phase 3 report. As noted, DNDO’s use of a small number of runs for each test means that DNDO can only be certain that the ASP can correctly identify or detect a source material over a broad range of possible values rather than at a specific rate. This is true even if the ASP was successful every time a test was conducted. For example, as noted in the Phase 3 test report, if the ASP correctly identified a source material 100 percent of the time, but the test was run only five times, the most DNDO can estimate is that the ASP should be able to correctly identify the source no worse than about 60 percent of the time.

The Phase 3 test results do not help to determine an ASP’s “true” level of performance because DNDO did not design the tests to assess ASP performance with a high degree of statistical confidence. In the Phase 3 test plan, DNDO was very clear that it had intended the tests to help develop a conduct of operations for secondary screenings and to cover a larger array of source materials and test scenarios than were conducted in the Phase 1 tests. “The Phase 3 test campaign was not originally intended to support the Secretarial Certification of the ASP systems. However, the test results provide relevant insights into important aspects of system performance and should be taken into consideration by the Secretary of Homeland Security in making his (ASP procurement) decision.” (p.iii)

It is important to note that DNDO does not elaborate in the test report as to what the “relevant insights” are or how they relate to Secretarial certification. DNDO also does not explain why those insights would be relevant considering that, as stated in the Phase 3 test plan, the results from the tests lack a high degree of statistical significance. Finally, it should be noted that when the Director of DNDO testified in September 2007 that the Phase 3 test results would help inform the Secretary’s recommendation, he also acknowledged that the Phase 3 test report had not yet been prepared.

SNL’s Special Tests Were Designed to Improve ASP Performance

The special tests were performed by experts from Sandia National Laboratories who were not part of the Phase 1 or Phase 3 tests. The special tests were designed to examine potential vulnerabilities associated with either the ASPs or the Phase 1 or Phase 3 test plan and vulnerabilities in DNDO’s test processes. Conducting this type of test would allow the ASP vendors the opportunity to make improvements to their systems in order to address weaknesses revealed during the special tests. Like the Phase 3 tests, the special tests used a small number of runs for each testing scenario. Because of the small number of runs, the test results do not support estimating the probability of detection or identification with a high confidence level making it difficult to use the results of the special tests to support a certification decision by the Secretary of DHS. On this point, the special test report acknowledges that “the special tests were never intended to demonstrate conformity of the systems against specific performance requirements.”

From the special tests, SNL concluded 1. “Areas for software and hardware improvement have been identified based on system performance issues observed for the versions of the ASP hardware and software tested at the NTS during Winter 2007.” 2. “For the data made available to us, the reported results … are consistent with the underlying collected data—indicating that the DNDO ASP system assessment was not biased.” 3. “Recommendations to improve the testing rigor have been made…(noting that) their implementation must be balanced against other test campaign impacts (such as) cost, schedule, availability of resources, etc.,” and 4. “Based on our limited tests we observed no data suggesting that the ASP system performance was inappropriately manipulated by either the vendors or the test team.”

Overall, the special test report appears to accurately describe the purpose, limitations, and results of the special tests. In our view, DNDO should consider SNL’s views as it proceeds with additional ASP testing in 2008. It is important to note, however, in Sandia’s conclusions that the “ASP system assessment was not biased” and that it “observed no data suggesting that the ASP system performance was inappropriately manipulated,” Sandia is referring to the data derived from ASP tests. However, SNL does not comment on the biased testing methods we identified during the Phase 1 ASP tests at the Nevada Test Site in 2007. Specifically, when we stated in September 2007 that DNDO’s Phase 1 tests were biased, we were referring to DNDO’s test methods which (1) used the same test sources and shielding materials during preliminary runs as were used during the actual tests and (2) did not use standard CBP operating procedures in testing the RIIDs.

Conclusions

Preventing the material for a nuclear weapon or a radiological dispersal device from being smuggled into the United States remains a key national security priority. Testing radiation detection equipment to understand its capabilities and limitations is an important part of preventing nuclear smuggling. The Phase 3 and special tests were part of DNDO’s 2007 effort to test ASPs in order to identify areas for further development to these devices. The Phase 3 test results are relevant to DNDO’s original objective for the Phase 3 tests—to identify areas in which the ASPs needed improvement. However, because of the limitations of the tests, DNDO should not be using the test results as indicators of the overall performance capabilities of the ASPs. Moreover, in the Phase 3 report, DNDO presented and analyzed the test results without fully disclosing key limitations of the tests, which is not consistent with basic principles of statistics and data analysis. Because of this, many of the report’s presentations and comparisons of performance among ASPs and between ASPs and PVTs are not well supported and are potentially misleading. Regarding the special tests, SNL notes in its test report that it designed the tests to identify areas where the ASPs need to improve—not to measure the ASPs performance against requirements. Overall, because of the limitations discussed in this report, it is our view that neither the Phase 3 tests nor the special tests should serve as a basis for the Secretary of DHS whether the ASPs represent “a significant increase in operational effectiveness” over current radiation detection equipment.

Recommendations for Executive Action

To ensure that the limitations of the Phase 3 test results, and future ASP test results, are clearly understood, we are making the following four recommendations.

We recommend that the Secretary of DHS use the results of the Phase 3 tests solely for the purposes for which they were intended—to identify areas needing improvement, not as a justification for certifying whether the ASPs warrant full-scale production.

However, if the Secretary of DHS intends to consider the results of the Phase 3 tests, along with other test data information, in making a certification decision regarding ASPs, then we recommend that the Secretary take the following actions: Direct the Director of DNDO to revise and clarify the Phase 3 test report to more fully disclose and articulate the limitations present in the Phase 3 tests—particularly the limitations associated with making comparisons between detection systems from a small number of test runs.

Clearly state which “relevant insights into important aspects of system performance” from the Phase 3 report are factored into any decision regarding the certification that ASPs demonstrate a significant increase in operational effectiveness.

Finally, we further recommend that since there are several phases of additional ASP testing currently ongoing, the Secretary should direct the Director of DNDO take steps to ensure that any limitations associated with ongoing testing are properly disclosed when the results of the current testing are reported.

Comments from the Department of Homeland Security and Our Evaluation

We provided DHS with a draft of this report for its review and comment. Its written comments are presented in appendix I. The department stated that it strongly disagreed with our draft report and two of our report’s recommendations. DHS agreed to take some action on a third recommendation and offered no comments on a fourth recommendation. The department stated several reasons for its disagreement. First, DHS cites narrative from the Phase 3 report explaining that the Phase 3 tests employed fewer test runs per test so as “to allow for more substantial variation among test cases” rather than “running sufficient number of repetitions … to provide high statistical significance results.” Thus, in DHS’s view, our assertion that the report does not “fully disclose” the Phase 3 tests’ limitations concerning the statistical significance of the results is incorrect. Our draft report recognizes DHS’s description of how the Phase 3 tests were conducted. Our concern is that although DNDO cited the limited statistical significance of the test results at the outset of the Phase 3 report, DNDO’s findings do not reflect this limitation. For example, as we note in our draft report, the Phase 3 report repeatedly states that the performances of the various ASPs were “statistically indistinguishable” even though DNDO did not perform enough test runs to estimate with a high degree of confidence whether the performances were actually similar. DNDO presents many of its findings as conclusive statements about ASP performance despite the fact that the Phase 3 test design cannot support these findings.

Second, the department commented that the Phase 3 test report clearly and succinctly stated another limitation of the test methodology— specifically, that the tests were not designed to be a precise indicator of ASP performance. In the department’s view, noting this limitation throughout the Phase 3 report would have been unwieldy. We did not expect DNDO to repeat this limitation throughout the report. However, as suggested in our report, the Phase 3 report should accurately reflect the test results without portraying the results as being more precise than they really are. Using an example from the Phase 3 report, if DNDO notes that an ASP successfully identified a specific source material 34 percent of the time during the tests, it should also indicate that, given the small number of test runs, DNDO can only estimate that the ASP would be able to correctly identify the specific source material between 10 and 65 percent of the time. However, no such discussion of the wide range of potential results is included in the report’s narrative. In our view, presenting the test results without sufficient narrative about the tests’ limitations is potentially misleading.

Third, the department stated that although the Phase 3 tests were not intended to support the DHS Secretary’s certification decision, DHS decided that it needed to consider all available test results in making this decision. DHS further commented that not doing so would subject it to criticism of “cherry-picking” the results. In response, although we acknowledge the need to consider all available test results, we believe they should be considered in their appropriate context, and that test results do not all carry the same weight. In our view, test results with a high degree of statistical significance (i.e., unlikely to be the result of chance) should be considered a better indicator of ASP performance than those with a lower level of statistical significance. Because the Phase 3 tests involved only 1-10 runs per test, very few of the results can be generalized as reliable estimates of how the ASPs perform and thus potentially provide questionable evidence for the certification process. We also note that, in its comments, DHS did not address what Phase 3 results or important insights it considered to be relevant to Secretarial certification.

Fourth, DHS comments that our draft report failed “to acknowledge the depth and breadth of the ASP test campaign, which is by far the most comprehensive test campaign ever conducted on radiation detection equipment.” However, our report describes previous ASP testing and some of our prior findings about that testing, and notes that ASP testing continues in 2008. More importantly, the extent of testing is not the issue at hand. In our view, regardless of how many tests are performed, the tests must employ sound, unbiased methodologies and DNDO should draw and present conclusions from the test results in ways that accurately and fully reflect the data and disclose their limitations.

DHS stated that it disagreed with our recommendations to (1) use the Phase 3 test’s to identify areas needing improvement and not as a basis for certification and (2) revise and clarify the Phase 3 report to reflect the limitations in the tests’ methodology and results. It did not offer comments on our recommendation that the Secretary clearly state what relevant insights from the Phase 3 report are factored into any certification decision. We continue to believe that the Phase 3 tests should be used only for the intended purpose stated in its test plan—to improve the software of ASPs. We would also note that our draft report recommends that DNDO revise and clarify the Phase 3 test report only if it includes Phase 3 test results among the data that will be presented to the Secretary prior to his decision on certification. If DNDO chooses to use the Phase 3 test results for certification, we believe it is important that DNDO explain what test results are relevant to certification and why the value of those results are not mitigated by the limitations associated with the Phase 3 tests’ small sample sizes.

In response to our last recommendation, the department stated that it has taken and will continue to take steps to ensure that it properly discloses any limitations associated with ongoing testing as it moves toward secretarial certification of the ASPs.

As agreed with your offices, unless you publicly announce the contents of this report, we plan no further distribution until 30 days from the report date. At that time, we will send copies of this report to the Secretary of DHS and interested congressional committees. We will also make copies available to others upon request. In addition, this report will be available at no charge on the GAO Web site at http://www.gao.gov.

If you or your staffs have any questions about this report, please contact me at (202) 512-3841 or aloisee@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff that made major contributions to this report are listed in appendix II.

Appendix I: Comments from the Department of Homeland Security

Appendix II: GAO Contact and Staff Acknowledgments

Staff Acknowledgements

In addition to the contact named above, Ned Woodward, Assistant Director; James Ashley, Nabajyoti Barkakati, Carol Kolarik, Omari Norman, Alison O’Neill, Anna Maria Ortiz, Daren Sweeney, Michelle Treistman, and Gene Wisnoski made significant contributions to this report.