Combating Nuclear Smuggling: DHS Improved Testing of Advanced Radiation Detection Portal Monitors, but Preliminary Results Show Limits of the New Technology
Released: 2009-06-22T00:00:00

Letter

DHS’s Criteria for Significant Increase in Operational Effectiveness Require a Marginal Improvement in the Detection of Certain Nuclear Materials and a Large Reduction in Innocent Alarms

Although the DHS criteria for primary screening require an improved ability to detect certain nuclear materials at operational thresholds, ASPs could meet the criteria for improvement while still failing to detect anything more than lightly shielded material. DNDO officials acknowledge that passive radiation detection equipment, which includes both the new and current-generation portal monitors, is capable of detecting certain nuclear materials only when this material is unshielded or lightly shielded. For this reason, the DOE threat guidance used to set PVTs’ detection threshold is based on the equipment’s limited sensitivity to anything more than lightly shielded nuclear material rather than on the assumption that smugglers would take effective shielding measures. DOE developed the guidance in 2002 and 2003 when CBP began deploying PVTs for primary screening. DOE and national laboratory officials responsible for the guidance told us the assumption of light shielding was based not on an analysis of the capabilities of potential smugglers to take effective shielding measures but rather on the limited sensitivity of PVTs to detect anything more than certain lightly shielded nuclear materials. In contrast, PVTs are more sensitive to the relatively strong radiation signature of other nuclear materials, and the threat guidance assumes a higher level of shielding for setting the operational threshold for detection of such materials. However, even for such materials, the DOE threat guidance assumes that shielding would not exceed a level provided by the contents of an average cargo container.

Moreover, DNDO has not completed efforts to fine-tune PVTs’ software and thereby improve sensitivity to nuclear materials. As a result, the criteria compare ASPs to the current performance of PVTs and do not take potential improvements into account, which affects any assessment of “significant” improvement over current technology. DNDO officials expect they can achieve small improvements to PVTs’ performance through additional development of “energy windowing,” a technique currently being used in PVTs to provide greater sensitivity than otherwise possible. Pacific Northwest National Laboratory officials responsible for developing the technique also told us small improvements may be possible, and CBP officials have repeatedly urged DNDO to investigate the potential of the technique. DNDO collected the data needed to further develop energy windowing during the 2008 performance testing at the Nevada Test Site but has not yet funded Pacific Northwest National Laboratory efforts to analyze the data and further develop the technique.

Other aspects of the criteria for a significant increase in operational effectiveness require that ASPs either provide more than a marginal improvement in addressing other limitations of current-generation equipment or at least maintain the same level of performance in areas in which the current-generation equipment is considered adequate: The primary screening requirement for an 80 percent reduction in the rate of innocent alarms could result in hundreds of fewer secondary screenings per day, thereby reducing CBP’s workload and delays to commerce. The actual reduction in the volume of innocent alarms would vary and would be greatest at the nation’s busiest ports of entry, such as Los Angeles/Long Beach, where CBP officials report that PVTs generate up to about 600 innocent alarms per day. A DNDO official said the requirement for an 80 percent reduction in innocent alarms was developed in conjunction with CBP and was based on a level that would provide meaningful workload relief.

The primary screening criteria requiring that ASPs provide at least the same level of sensitivity to plutonium and medical and industrial isotopes, but not specifying an improvement, were based on DNDO’s assessment that PVTs adequately detect such materials, which have a stronger radiation signature than HEU. In addition, CBP officials said that including medical and industrial isotopes in the criteria addressed a CBP requirement for verifying that those transporting certain quantities of these materials into the United States are properly licensed.

The secondary screening requirement that ASPs reduce the probability of misidentifying special nuclear material by one-half addresses the inability of relatively small handheld devices to consistently locate and identify potential threats in large cargo containers. For example, a handheld device may fail to correctly identify special nuclear material if the material is well-shielded or the device is not placed close enough to a radiation source to obtain a recognizable measurement. According to CBP and DNDO, the requirement for a reduction in the average time to conduct secondary screenings is not more specific because the time varies significantly among ports of entry and types of cargo being screened.

DHS Increased the Rigor of Advanced Portal Monitor Testing

Improvements to the 2008 round of testing addressed concerns we raised about earlier rounds of ASP testing. However, the testing still had limitations, and the preliminary results are mixed.

Improvements to Testing Provided Credibility to Test Results

As we testified in September 2008, DHS’s improvements to the 2008 round of ASP testing addressed concerns we raised about previous tests. A particular area of improvement was in the performance testing at the Nevada Test Site, where DNDO compared the capability of ASP and current-generation equipment to detect and identify nuclear and radiological materials, including those that could be used in a nuclear weapon. The improvements addressed concerns we previously raised about the potential for bias and provided credibility to the results within the limited range of scenarios tested by DNDO. For example, we reported in 2007 that DNDO had allowed ASP contractors to adjust their systems after preliminary runs using the same radiological materials that would be used in the formal tests. In contrast, the plan for the 2008 performance test stipulated that there would be no system contractor involvement in test execution, and no ASP contractors were at the test location on the day we observed performance testing. Furthermore, DNDO officials told us, and we observed, that they did not conduct preliminary runs with threat objects used in the formal tests. In 2007, we reported that DNDO did not objectively test the handheld identification devices because it did not adhere to CBP’s standard operating procedure for using the devices to conduct a secondary inspection, which is fundamental to the equipment’s performance in the field. DNDO addressed this limitation in the 2008 round of performance testing: CBP officers operated the devices and adhered as closely to the standard operating procedure as test conditions allowed. While the test conditions did not allow CBP officers to obtain real-time technical support in interpreting the device’s measurements, as they would in the field to increase the probability of correctly identifying a radiation source, DNDO officials said they addressed this limitation. For example, they treated a decision by a CBP officer to indicate the need for technical support as a correct outcome if the test scenario involved the use of a potential threat, such as HEU.

Other aspects of testing, while not specifically addressing concerns we previously raised, also added credibility to the test results. Based on our analysis of the performance test plan, we concluded that the test design was sufficient to identify statistically significant differences between the new technology and current-generation systems when there were relatively large differences in performance. Specifically, DNDO conducted a sufficient number of runs of each scenario used in the 2008 performance testing to identify such differences.

With regard to the general conduct of the 2008 round of testing, two aspects, in particular, enhanced the overall rigor of the tests: (1) criteria for ensuring that ASPs met the requirements for each phase before advancing to the next, and (2) the participation of CBP and the DHS Science and Technology Directorate. The test and evaluation master plan established criteria requiring that the ASPs have no critical or severe issues rendering them completely unusable or impairing their function before starting or completing any test phase. In addition, the criteria established a cumulative limit of 10 issues requiring a work-around (e.g., a straightforward corrective step, such as a minor change in standard operating procedures) and 15 cosmetic issues not affecting proper functioning. DNDO and CBP adhered to the criteria even though doing so resulted in integration testing conducted at the Pacific Northwest National Laboratory taking longer than anticipated and delaying the start of field validation. For example, DNDO and CBP did not allow a vendor’s ASP system to complete integration testing until all critical or severe issues had been resolved.

The involvement of CBP and the DHS Science and Technology Directorate provided an independent check, within DHS, of DNDO’s efforts to develop and test the new portal monitors. For example, the lead CBP official involved in ASP testing told us that DNDO provided an initial assessment of the severity of issues uncovered during testing, but CBP made the final decision on categorizing them as critical, severe, work-around, or cosmetic issues. CBP also added a final requirement to integration testing before proceeding to field validation to demonstrate ASPs’ ability to operate for 40 hours without additional problems. According to CBP officials, their efforts to resolve issues prior to field validation reflect the importance CBP places on ensuring that ASPs are sufficiently stable and technically mature to operate effectively in a working port of entry and thereby provide for a productive field validation.

The DHS Science and Technology Directorate, which is responsible for developing and implementing the department’s test and evaluation policies and standards, will have the lead role in the final phase of ASP testing; the final phase, consisting of 21 days of continuous operation, is scheduled to begin at one seaport after the completion of field validation. The Science and Technology Directorate identified two critical questions to be addressed through operational testing: (1) Will the ASP system improve operational effectiveness (i.e., detection and identification of threats) relative to the current-generation system, and (2) is the ASP system suitable for use in the operational environment at land and sea ports of entry? The suitability of ASPs includes factors such as reliability, maintainability, and supportability. Because the operational testing conducted at one seaport is not sufficient to fully answer these questions—for example, because the testing will not allow threat objects to be inserted into cargo containers—the directorate plans to also conduct an independent analysis of the results from previous test phases, including performance testing.

The 2008 testing still had limitations, which do not detract from the test results’ credibility but do require that results be appropriately qualified. Limitations included the following: The number of handheld identification device measurements collected during performance testing was sufficient to distinguish only particularly large differences from ASPs’ identification ability. In particular, the standard operating procedure for conducting secondary inspections using ASPs, which requires less time than when using handheld devices, allowed DNDO to collect more than twice as many ASP measurements and to test ASPs’ identification ability against more radiation sources than used to test handheld identification devices.

The performance test results cannot be generalized beyond the limited set of scenarios tested. For example, DNDO used a variety of masking and shielding scenarios designed to include cases where both systems had 100 percent detection, cases where both had zero percent detection, and several configurations in between so as to estimate the point where detection capability ceased. However, the scenarios did not represent the full range of possibilities for concealing smuggled nuclear or radiological material. For example, DNDO only tested shielding and masking scenarios separately, to differentiate between the impacts of shielding and masking on the probabilities of detection and identification. As a result, the performance test results cannot show how well each system would detect and identify nuclear or radiological material that is both shielded and masked, which might be expected in an actual smuggling incident. Similarly, DNDO used a limited number of threat objects to test ASPs’ detection and identification performance, such as weapons-grade plutonium but not reactor-grade plutonium, which has a different isotopic composition. A report on special testing of ASPs conducted by Sandia National Laboratories in 2007 recommended that future tests use plutonium sources having alternative isotopic compositions. Sandia based its recommendations on results showing that the performance of ASP systems varied depending on the isotopic composition of plutonium.

The Science and Technology Directorate’s operational testing is designed to demonstrate that the average time between equipment failures (the measure of ASPs’ reliability) is not less than 1,000 hours. Thus, the testing will not show how reliable the equipment will be over a longer term. DHS Science and Technology Directorate officials recognize this limitation and said they designed operational testing only to demonstrate compliance with the ASP performance specification. Furthermore, to the extent that the Science and Technology Directorate relies on performance test results to evaluate ASPs’ ability to detect and identify threats, its analysis of ASPs’ effectiveness will be subject to the same limitations as the original testing and analysis conducted by DNDO.

Preliminary Test Results Are Mixed

The preliminary results presented to us by DNDO are mixed, particularly in the capability of ASPs used for primary screening to detect certain shielded nuclear materials. However, we did not obtain DNDO’s final report on performance testing conducted at the Nevada Test Site until early April 2009, and thus we had limited opportunity to evaluate the report. In addition, we are not commenting on the degree to which the final report provides a fair representation of ASPs’ performance. Preliminary results from performance testing show that the new portal monitors detected certain nuclear materials better than PVTs when shielding approximated DOE threat guidance, which is based on light shielding. In contrast, differences in system performance were less notable when shielding was slightly increased or decreased: Both the PVTs and ASPs were frequently able to detect certain nuclear materials when shielding was below threat guidance, and both systems had difficulty detecting such materials when shielding was somewhat greater than threat guidance. DNDO did not test ASPs or PVTs against moderate or greater shielding because such scenarios are beyond both systems’ ability. (See fig. 3 for a summary of performance test results for detection of certain nuclear materials.)

With regard to secondary screening, ASPs performed better than handheld devices in identification of threats when masked by naturally occurring radioactive material. However, differences in the ability to identify certain shielded nuclear materials depended on the level of shielding, with increasing levels appearing to reduce any ASP advantages over the handheld identification devices—another indication of the fundamental limitation of passive radiation detection.

Other phases of testing, particularly integration testing, uncovered multiple problems meeting requirements for successfully integrating the new technology into operations at ports of entry. Of the two ASP vendors participating in the 2008 round of testing, one has fallen several months behind in testing due to the severity of the problems it encountered during integration testing; the problems were so severe that it may have to redo previous test phases to be considered for certification. The other vendor’s system completed integration testing, but CBP suspended field validation of the system after 2 weeks because of serious performance problems that may require software revisions. In particular, CBP found that the performance problems resulted in an overall increase in the number of referrals for secondary screening compared to the existing equipment. According to CBP, this problem will require significant corrective actions before testing can resume; such corrective actions could in turn change the ability of the ASP system to detect threats. The problem identified during field validation was in addition to ones identified during integration testing, which required multiple work-arounds and cosmetic changes before proceeding to the next test phase. For example, one problem requiring a work-around related to the amount of time it takes for the ASP to sound an alarm when a potential threat material has been detected. Specifications require that ASPs alarm within two seconds of a vehicle exiting the ASP. However, during testing, the vendor’s ASP took longer to alarm when a particular isotope was detected. The work-around to be implemented during field validation requires that all vehicles be detained until cleared by the ASP; the effect on commerce must ultimately be ascertained during field validation.

CBP officials anticipate that they will continue to uncover problems during the first few years of use if the new technology is deployed in the field. The officials do not necessarily regard such problems to be a sign that testing was not rigorous but rather a result of the complexity and newness of the technology and equipment.

Schedule Delays Have Allowed More Time for Analysis and Review of Test Results, but DNDO’s Latest Schedule Does Not Include Computer Simulations to Provide Additional Insight into ASP Capabilities

Delays to the schedule for the 2008 round of testing have allowed more time for analysis and review of results, particularly from performance testing conducted at the Nevada Test Site. The original schedule, which underestimated the time needed for testing, anticipated completion of testing in mid-September 2008 and the DHS Secretary’s decision on ASP certification between September and November 2008. DHS officials acknowledged that scheduling a certification decision shortly after completion of testing would leave limited time to complete final test reports and said the DHS Secretary could rely instead on preliminary reports if the results were favorable to ASPs. DHS’s most recent schedule anticipated a decision on ASP certification as early as May 2009, but DHS has not updated its schedule for testing and certification since suspending field validation in February 2009 due to ASP performance problems.

Problems uncovered during testing of ASPs’ readiness to be integrated into operations at U.S. ports of entry have caused the greatest delays to date and have allowed more time for DNDO to analyze and review the results of performance testing. Integration testing was originally scheduled to conclude in late July 2008 for both ASP vendors. The one ASP system that successfully passed integration testing did not complete the test until late November 2008—approximately 4 months behind schedule. (The delays to integration testing were due in large part to the adherence of DNDO and CBP to the criteria discussed earlier for ensuring that ASPs met the requirements for each test phase.) In contrast, delays to performance testing, which was scheduled to run concurrently with integration testing, were relatively minor. Both ASP systems completed performance testing in August 2008, about a month later than DNDO originally planned.

The schedule delays have allowed more time to conduct injection studies—computer simulations for testing the response of ASPs and PVTs to the radiation signatures of threat objects randomly “injected” (combined) into portal monitor records of actual cargo containers transported into the United States, including some containers with innocent sources of radiation. However, DNDO does not plan to complete the studies prior to the Secretary of Homeland Security’s decision on certification even though DNDO and other officials have indicated that the studies could provide additional insight into the capabilities and limitations of advanced portal monitors. According to DNDO officials, injection studies address the inability of performance testing conducted at the Nevada Test Site to replicate the wide variety of cargo coming into the United States and the inability to bring special nuclear material and other threat objects to ports of entry and place them in cargo during field validation. Similarly, while they acknowledged that injection studies have limitations, DOE national laboratory officials said the studies can increase the statistical confidence in comparisons of ASPs’ and PVTs’ probability of detecting threats concealed in cargo because of the possibility of supporting larger sample sizes than feasible with actual testing. A February 2008 DHS independent review team report on ASP testing also highlighted the benefits of injection studies, including the ability to explore ASP performance against a large number of threat scenarios at a practical cost and schedule and to permit an estimate of the minimum detectable amount for various threats.

DNDO has the data needed to conduct the studies. It has supported efforts to collect data on the radiation signatures for a variety of threat objects, including special nuclear materials, as recorded by both ASP and PVT systems. It has also collected about 7,000 usable “stream-of-commerce” records from ASP and PVT systems installed at a seaport. Furthermore, DNDO had earlier indicated that injection studies could provide information comparing the performance of the two systems as part of the certification process for both primary and secondary screening. However, addressing deficiencies in the stream-of-commerce data delayed the studies, and DNDO subsequently decided that performance testing would provide sufficient information to support a decision on ASP certification. DNDO officials said they would instead use injection studies to support effective deployment of the new portal monitors.

Conclusions

Given that radiation detection equipment is already being used at ports of entry to screen for smuggled nuclear or radiological materials, the decision whether to replace existing equipment requires that the benefits of the new portal monitors be weighed against the costs. DNDO acknowledges that ASPs are significantly more expensive than PVTs to deploy and maintain, and based on preliminary results from the 2008 testing, it is not yet clear that the $2 billion cost of DNDO’s deployment plan is justified. Even if ASPs are able to reduce the volume of innocent cargo referred for secondary screening, they are not expected to detect certain nuclear materials that are surrounded by a realistic level of shielding better than PVTs could. Preliminary results of DNDO’s performance testing show that ASPs outperformed the PVTs in detection of such materials during runs with light shielding, but ASPs’ performance rapidly deteriorated once shielding was slightly increased. Furthermore, DNDO and DOE officials acknowledged that the performance of both portal monitors in detecting such materials with a moderate amount of shielding would be similarly poor. This was one of the reasons that performance testing did not include runs with a moderate level of shielding.

Two additional aspects of the 2008 round of testing call into question whether ASPs’ ability to provide a marginal improvement in detection of nuclear materials and reduce innocent alarms warrants the cost of the new technology. First, the DHS criteria for a significant increase in operational effectiveness do not take into account recent efforts to improve the current-generation portal monitors’ sensitivity to nuclear materials through the “energy windowing” technique, most likely at a much lower cost. Data on developing this technique were collected during the 2008 round of performance testing but have not been analyzed. Second, while DNDO made improvements to the 2008 round of ASP testing that provided credibility to the test results, its test schedule does not allow for completion of injection studies prior to certification even though the studies could provide additional insight into the performance of the new technology. Without results from injection studies, the Secretary of Homeland Security would have to make a decision on certification based on a limited number of test scenarios conducted at the Nevada Test Site.

Assuming that the Secretary of Homeland Security certifies ASPs, CBP officials anticipate that they will discover problems with the equipment when they start using it in the field. Integration testing uncovered a number of such problems, which delayed testing and resulted in ASP vendors making multiple changes to their systems. Correcting such problems in the field could prove to be more costly and time consuming than correcting problems uncovered through testing, particularly if DNDO proceeds directly from certification to full-scale deployment, as allowed under the congressional certification requirement that ASPs provide a significant increase in operational effectiveness.

Recommendations for Executive Action

We recommend that the Secretary of Homeland Security direct the Director of DNDO to take the following two actions to ensure a sound basis for a decision on ASP certification: Assess whether ASPs meet the criteria for a significant increase in operational effectiveness based on a valid comparison with PVTs’ full performance potential, including the potential to further develop PVTs’ use of energy windowing to provide greater sensitivity to threats. Such a comparison could also be factored into an updated cost-benefit analysis to determine whether it would be more cost-effective to continue to use PVTs or deploy ASPs for primary screening at particular ports of entry.

Revise the schedule for ASP testing and certification to allow sufficient time for review and analysis of results from the final phases of testing and completion of all tests, including injection studies.

If ASPs are certified, we further recommend that the Secretary of Homeland Security direct the Director of DNDO to develop an initial deployment plan that allows CBP to uncover and resolve any additional problems not identified through testing before proceeding to full-scale deployment—for example, by initially deploying ASPs at a limited number of ports of entry.

Agency Comments and Our Evaluation

We provided a draft of this report to DOE and DHS for their review and comment. DOE provided technical comments, which we have incorporated into our report as appropriate. DHS’s written comments are reproduced in appendix II.

DHS agreed in part with our recommendations. Specifically, DHS stated that it believes its plan to deploy ASPs in phases, starting at a small number of low-impact locations, is in accordance with our recommendation to develop an initial deployment plan that allows problems to be uncovered and resolved prior to full-scale deployment. We agree that this deployment plan would address our recommendation and note that DHS’s comments are the first indication provided to us of the department’s intention to pursue such a plan.

In contrast, DHS did not concur with our recommendations to (1) assess whether ASPs meet the criteria for a significant increase in operational effectiveness based on a comparison with PVTs’ full potential, including further developing PVTs’ use of energy windowing; and (2) revise the ASP testing and certification schedule to allow sufficient time for completion of all tests, including injection studies. With regard to energy windowing, DHS stated that using current PVT performance as a baseline for comparison is a valid approach because the majority of increased PVT performance through energy windowing has already been achieved. While DHS may be correct, its assessment is based on expert judgment rather than the results of testing and analysis being considered by the department to optimize the use of energy windowing. Given the marginal increase in sensitivity required of ASPs, we stand by our recommendation to assess ASPs against PVTs’ full potential. DHS can then factor PVTs’ full potential into a cost-benefit analysis prior to acquiring ASPs. On this point, DHS commented that its current cost-benefit analysis is a reasonable basis to guide programmatic decisions. However, upon receiving DHS’s comments, we contacted DNDO to obtain a copy of its cost-benefit analysis and were told the analysis is not yet complete.

With regard to injection studies, DHS agreed that the schedule for ASP certification must allow sufficient time for review and analysis of test results but stated that DHS and DOE experts concluded injection studies were not required for certification. DHS instead stated that the series of ASP test campaigns would provide a technically defensible basis for assessing the new technology against the certification criteria. However, DHS did not rebut the reasons we cited for conducting injection studies prior to certification, including test delays that have allowed more time to conduct the studies and the ability to explore ASP performance against a large number of threat scenarios at a practical cost and schedule. On the contrary, DHS acknowledged the delays to testing and the usefulness of injection studies. Given that each phase of testing has revealed new information about the capabilities and limitations of ASPs, we believe conducting injection studies prior to certification would likely offer similar insights and would therefore be prudent prior to a certification decision.

DHS provided additional comments regarding our assessment of the relative sensitivity of ASPs and PVTs and our characterization of the severity of the ASPs’ software problems uncovered during field validation. With regard to sensitivity, DHS implied that our characterization of the relative ability of ASPs and PVTs is inaccurate and misleading because we did not provide a complete analysis of test results. We disagree. First, in meetings to discuss the preliminary results of performance testing conducted at the Nevada Test Site, DNDO officials agreed with our understanding of the ability of ASPs and PVTs deployed for primary screening to detect shielded nuclear materials. Furthermore, contrary to the assertion that a complete analysis requires a comparison of ASPs to handheld identification devices, our presentation is consistent with DHS’s primary screening criterion for detection of shielded nuclear materials, which only requires that ASPs be compared with PVTs. Finally, while we agree that the performance test results require a more complete analysis, DNDO did not provide us with its final performance test report until early April 2009, after DHS provided its comments on our draft report. In the absence of the final report, which DNDO officials told us took longer than anticipated to complete, we summarized the preliminary results that DNDO presented to us during the course of our review as well as to congressional stakeholders.

With regard to ASP software problems uncovered during field validation, we clarified our report in response to DHS’s comment that the severity of the problems has not yet been determined. DHS stated that its preliminary analysis indicates the problems should be resolved by routine adjustments to threshold settings rather than presumably more significant software “revisions.” However, given the history of lengthy delays during ASP testing, we believe that DHS’s assessment of the severity of problems encountered during field validation may be overly optimistic.

As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the Secretaries of Homeland Security and Energy; the Administrator of NNSA; and interested congressional committees. The report will also be available at no charge on the GAO Web site at http://www.gao.gov.

If you or your staffs have any questions about this report, please contact me at (202) 512-3841 or aloisee@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III.

Appendix I: Scope and Methodology

To evaluate the degree to which Department of Homeland Security’s (DHS) criteria for a significant increase in operational effectiveness address the limitations of the current generation of radiation detection equipment, we clarified the intent of the criteria through the Domestic Nuclear Detection Office’s (DNDO) written answers to our questions and through interviews with U.S. Customs and Border Protection (CBP) officials. We also took steps to gain a fuller understanding of the strengths and limitations of the current-generation equipment, which the criteria use as a baseline for evaluating the effectiveness of advanced spectroscopic portals (ASP). In particular, we obtained copies of the Department of Energy (DOE) threat guidance and related documents used to set polyvinyl toluene (PVT) thresholds for detection of nuclear materials. We interviewed DOE and national laboratory officials responsible for the threat guidance about the process for developing it and the basis for its underlying assumptions, including shielding levels. We also interviewed DNDO and Pacific Northwest National Laboratory officials regarding the extent to which PVTs currently deployed at ports of entry meet the guidance and the development and use of energy windowing to enhance PVTs’ sensitivity to nuclear materials.

To evaluate the rigor of the 2008 round of testing as a basis for determining ASPs’ operational effectiveness, we reviewed the test and evaluation master plan and plans for individual phases of testing, including system qualification testing conducted at vendors’ facilities, performance testing conducted at the Nevada Test Site for evaluating ASP detection and identification capabilities, and integration testing conducted at Pacific Northwest National Laboratory for evaluating the readiness of ASPs to be used in an operational environment at ports of entry. We also reviewed draft plans for field validation conducted at CBP ports of entry and the DHS Science and Technology Directorate’s independent operational test and evaluation. In reviewing these documents, we specifically evaluated the extent to which the performance test design was sufficient to identify statistically significant differences between the ASP and current- generation systems and whether DHS had addressed our concerns about previous rounds of ASP testing. We interviewed DNDO, CBP, and other DHS officials responsible for conducting and monitoring tests, and we observed, for one day each, performance testing at the Nevada Test Site and integration testing at DOE’s Pacific Northwest National Laboratory. We also interviewed representatives of entities that supported testing, including DOE’s National Nuclear Security Administration and Pacific Northwest National Laboratory, the National Institute of Standards and Technology, and the Johns Hopkins University Applied Physics Laboratory. We reviewed the DHS independent review team report of previous ASP testing conducted in 2007, and we interviewed the chair of the review team to clarify the report’s findings. Finally, we examined preliminary or final results for the phases of testing completed during our review, and we interviewed DNDO and CBP officials regarding the results.

To evaluate the test schedule, we analyzed the initial working schedule DNDO provided to us in May 2008 and the schedule presented in the August 2008 test and evaluation master plan, and we tracked changes to the schedule and the reasons for any delays. We interviewed DNDO and other officials with a role in testing to determine the amount of time allowed for analysis and review of results. We interviewed DNDO and Pacific Northwest National Laboratory officials regarding the injection studies, including reasons for delays in the studies and plans for including the results as part of the ASP certification process.

We conducted this performance audit from May 2008 to May 2009 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Comments from the Department of Homeland Security

Appendix III: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, Ned Woodward, Assistant Director; Dr. Timothy Persons, Chief Scientist; James Ashley; Steve Caldwell; Joseph Cook; Omari Norman; Alison O’Neill; Rebecca Shea; Kevin Tarmann; and Eugene Wisnoski made key contributions to this report.