Combating Nuclear Smuggling: DHS Improved Testing of Advanced Radiation Detection Portal Monitors, but Preliminary Results Show Limits of the New Technology
Published: 2009-05-21T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Department of Homeland Security's (DHS) Domestic Nuclear Detection Office (DNDO) is testing new advanced spectroscopic portal (ASP) radiation detection monitors. DNDO expects ASPs to reduce both the risk of missed threats and the rate of innocent alarms, which DNDO considers to be key limitations of radiation detection equipment currently used by Customs and Border Protection (CBP) at U.S. ports of entry. Congress has required that the Secretary of DHS certify that ASPs provide a significant increase in operational effectiveness before obligating funds for full-scale procurement. GAO was asked to review (1) the degree to which DHS's criteria for a significant increase in operational effectiveness address the limitations of existing radiation detection equipment, (2) the rigor of ASP testing and preliminary test results, and (3) the ASP test schedule. GAO reviewed the DHS criteria, analyzed test plans, and interviewed DHS officials.
section_title: What GAO Found
paragraphs: The DHS criteria for a significant increase in operational effectiveness require a minimal improvement in the detection of threats and a large reduction in innocent alarms. Specifically, the criteria require a marginal improvement in the detection of certain weapons-usable nuclear materials, considered to be a key limitation of current-generation portal monitors. The criteria require improved performance over the current detection threshold, which for certain nuclear materials is based on the equipment's limited sensitivity to anything more than lightly shielded materials, but do not specify a level of shielding that smugglers could realistically use. In addition, DNDO has not completed efforts to improve current-generation portal monitors' performance. As a result, the criteria do not take the current equipment's full potential into account. With regard to innocent alarms, the other key limitation of current equipment, meeting the criteria could result in hundreds fewer innocent alarms per day, thereby reducing CBP's workload and delays to commerce. DHS increased the rigor of ASP testing in comparison with previous tests. For example, DNDO mitigated the potential for bias in performance testing (a concern GAO raised about prior testing) by stipulating that there would be no ASP contractor involvement in test execution. Such improvements added credibility to the test results. However, the testing still had limitations, such as a limited set of scenarios used in performance testing to conceal test objects from detection. Moreover, the preliminary results are mixed. The results show that the new portal monitors have a limited ability to detect certain nuclear materials at anything more than light shielding levels: ASPs performed better than current-generation portal monitors in detection of such materials concealed by light shielding approximating the threat guidance for setting detection thresholds, but differences in sensitivity were less notable when shielding was slightly below or above that level. Testing also uncovered multiple problems in ASPs meeting the requirements for successful integration into operations at ports of entry. CBP officials anticipate that, if ASPs are certified, new problems will appear during the first few years of deployment in the field. While DNDO's schedule underestimated the time needed for ASP testing, test delays have allowed more time for review and analysis of results. DNDO's original schedule anticipated completion in September 2008. Problems uncovered during testing of ASPs' readiness to be integrated into operations at U.S. ports of entry caused the greatest delays to this schedule. DHS's most recent schedule anticipated a decision on ASP certification as early as May 2009, but DHS recently suspended field validation due to ASP performance problems and has not updated its schedule for testing and certification. In any case, DNDO does not plan to complete computer simulations that could provide additional insight into ASP capabilities and limitations prior to certification even though delays have allowed more time to conduct the simulations. DNDO officials believe the other tests are sufficient for ASPs to demonstrate a significant increase in operational effectiveness.

section_title: Letter
paragraphs: Preventing radioactive material from being smuggled into the United  States is a key national security objective. In particular, terrorists could  use special nuclear material such as highly enriched uranium (HEU) or  plutonium in a nuclear weapon; other radioactive materials could be used  in a radiological dispersal device (a “dirty bomb”). The national security  mission of U.S. Customs and Border Protection (CBP), an agency within  the Department of Homeland Security (DHS), includes screening for  smuggled nuclear or radiological material while facilitating the flow of  legitimate trade and travel. To screen cargo at ports of entry, CBP  conducts primary inspections with radiation detection equipment called  portal monitors—large stationary detectors through which cargo  containers and vehicles pass as they enter the United States. When  radiation is detected, CBP conducts secondary inspections using a second  portal monitor to confirm the original alarm and a handheld radioactive  isotope identification device to identify the radiation’s source and  determine whether it constitutes a threat. CBP officers must investigate  each alarm until they are convinced that the vehicle, occupants, and any  cargo pose no threat and can be allowed to enter the United States.
According to DHS’s Domestic Nuclear Detection Office (DNDO), the  current generation of radiation detection equipment has limitations.  Specifically, the polyvinyl toluene (PVT) portal monitors currently in use  can detect radiation but cannot identify the source. As a result, the  monitors’ radiation alarms can be set off even by benign, naturally  occurring radioactive material. One way to reduce the rate of such  innocent alarms—and thereby minimize unnecessary secondary  inspections and enhance the flow of commerce—is to adjust the  operational thresholds for the level of radiation required for PVTs to alarm  (i.e., operate the PVTs at a reduced level of sensitivity). However, reducing  the sensitivity may make it more difficult to detect certain nuclear  materials.
Since 2005, DNDO has been developing and testing advanced  spectroscopic portals (ASP), a new type of portal monitor designed to  both detect radiation and identify the source. The new portal monitors use  technology similar to that in handheld identification devices currently  used for secondary screening. Key differences from handheld  identification devices include a larger number of detectors, more  sophisticated software, and a more extensive library of radiation  signatures that may provide more consistent and rapid screening and may  increase the likelihood of correct identification. DNDO hopes to use the  new portal monitors to replace at least some PVTs currently used for  primary screening, as well as PVTs and handheld identification devices  currently used for secondary screening. However, the new portal monitors  cost significantly more than PVTs. We estimated in September 2008 that  the lifecycle cost of each standard cargo version of the ASP (including  deployment costs) is about $822,000, compared with about $308,000 for  the PVT standard cargo portal, and that the total program cost for DNDO’s  latest plan for deploying radiation portal monitors—which relies on a  combination of ASPs and PVTs and does not deploy radiation portal  monitors at all border crossings—would be about $2 billion. Moreover,  CBP officials expect operation and maintenance costs to be significantly  higher for ASPs than for PVTs because of the greater complexity of ASP  equipment.
Concerned about the performance and cost of the new ASP monitors,  Congress required the Secretary of Homeland Security to certify that the  monitors will provide a “significant increase in operational effectiveness”  before DNDO obligates funds for full-scale ASP procurement. The  Secretary must submit separate certifications for primary and secondary  inspection. In response, DNDO, CBP, and the DHS management  directorate jointly issued criteria in July 2008 for determining whether the  new technology provides a significant increase in operational  effectiveness—four criteria for primary screening and two for secondary  screening (see fig. 1). The primary screening criteria require that the new  portal monitors detect potential threats as well as or better than PVTs,  show improved performance in detection of HEU, and reduce innocent  alarms. To meet the secondary screening criteria, the new portal monitors  must reduce the probability of misidentifying special nuclear material  (e.g., HEU and plutonium) and the average time to conduct secondary  screenings.
To demonstrate a significant increase in operational effectiveness for  either primary or secondary screening, ASPs must satisfy all of the criteria  for that deployment option, independent of satisfying the criteria for the  other option. The criteria generally compare the new portal monitors to  current-generation equipment as used under CBP’s standard operating  procedure. For example, the standard operating procedure for secondary  screening calls for inconclusive readings to be sent for additional analysis  to CBP’s Laboratories and Scientific Services, which has access to  additional software and trained experts.
DNDO designed and coordinated a series of tests, originally scheduled to  run from April 2008 through September 2008, to determine whether the  new portal monitors meet the certification criteria for primary and  secondary screening and are ready for deployment. Key phases of testing  completed to date include verifying that ASPs meet DNDO’s performance  specification, which was followed by concurrent testing of the new and  current equipments’ ability to detect and identify threats and of ASPs’  readiness to be integrated into operations for both primary and secondary  screening at ports of entry. Two remaining phases not yet completed  include field validation at four northern and southern border crossings and  two seaports, as well as an independent evaluation, conducted by the DHS  Science and Technology Directorate at one of the seaports, of the new  portal monitors’ effectiveness and suitability (see fig. 2). Two ASP vendors  have contracts with DNDO to develop the new portal monitors and are  participating in the round of testing that began in 2008. DNDO designed  the testing to allow each vendor’s system to complete all test phases and  be certified based on its own performance as providing a significant  increase in operational effectiveness.
We have raised concerns since 2006 regarding DNDO’s previous efforts to  develop and test the new portal monitors. In October 2006, we found that  DNDO’s analysis of the benefits and costs of deploying the new portal  monitors relied on assumptions of their anticipated performance level  instead of actual test data. Among other things, we recommended that  DNDO conduct further testing before spending additional funds to  purchase the new equipment. In September 2007, we testified that DNDO’s  testing at the Department of Energy’s (DOE) Nevada Test Site did not  represent an objective or rigorous assessment because DNDO used biased  test methods that enhanced the apparent performance of the ASPs and did  not test the limitations of their detection capabilities. Most recently, we  found in September 2008 that a DNDO report on testing conducted in 2007  did not accurately depict test results and could potentially be misleading.  We concluded that the results could identify areas for improvement but  should not be used as indicators of ASPs’ overall performance.
In this context, you asked us to review the 2008 round of testing leading up  to the Secretary of Homeland Security’s decision on ASP certification. We  reviewed (1) the degree to which DHS’s criteria for a significant increase  in operational effectiveness address the limitations of the current  generation of radiation detection equipment, (2) the rigor of the testing as  a basis for determining ASPs’ operational effectiveness and preliminary  results of testing completed to date, and (3) the extent to which the test  schedule allows time for DHS to review and analyze results. This report  updates our September 2008 testimony, which included preliminary  observations on the DHS criteria for a significant increase in operational  effectiveness and the 2008 round of testing.
To conduct our review, we analyzed DHS’s criteria for a significant  increase in operational effectiveness and DNDO’s written response to our  detailed questions regarding the criteria. Because the criteria compare the  new portal monitors to existing equipment, we analyzed the threat  guidance used to set detection thresholds for PVTs and interviewed DOE  and national laboratory officials responsible for the guidance. In addition,  we analyzed the test plans for the 2008 round of testing, including the test  schedule and reasons for any delays. We interviewed DNDO, CBP, and  other DHS officials responsible for conducting and monitoring tests, and  we observed 1 day each of performance testing at the Nevada Test Site  and integration testing at DOE’s Pacific Northwest National Laboratory.  We analyzed preliminary or final results for the phases of testing  completed during our review, and we interviewed DNDO and CBP officials  regarding the results. (App. I presents a detailed discussion of the scope  and methodology of our review.)
We conducted this performance audit from May 2008 to May 2009 in  accordance with generally accepted government auditing standards. Those  standards require that we plan and perform the audit to obtain sufficient,  appropriate evidence to provide a reasonable basis for our findings and  conclusions based on our audit objectives. We believe that the evidence  obtained provides a reasonable basis for our findings and conclusions  based on our audit objectives.
subsections: section_title: DHS’s Criteria for Significant Increase in Operational Effectiveness Require a Marginal Improvement in the Detection of Certain Nuclear Materials and a Large Reduction in Innocent Alarms
paragraphs: Although the DHS criteria for primary screening require an improved  ability to detect certain nuclear materials at operational thresholds, ASPs  could meet the criteria for improvement while still failing to detect  anything more than lightly shielded material. DNDO officials acknowledge  that passive radiation detection equipment, which includes both the new  and current-generation portal monitors, is capable of detecting certain  nuclear materials only when this material is unshielded or lightly  shielded. For this reason, the DOE threat guidance used to set PVTs’  detection threshold is based on the equipment’s limited sensitivity to  anything more than lightly shielded nuclear material rather than on the  assumption that smugglers would take effective shielding measures. DOE  developed the guidance in 2002 and 2003 when CBP began deploying PVTs  for primary screening. DOE and national laboratory officials responsible  for the guidance told us the assumption of light shielding was based not on  an analysis of the capabilities of potential smugglers to take effective  shielding measures but rather on the limited sensitivity of PVTs to detect  anything more than certain lightly shielded nuclear materials. In contrast,  PVTs are more sensitive to the relatively strong radiation signature of  other nuclear materials, and the threat guidance assumes a higher level of  shielding for setting the operational threshold for detection of such  materials. However, even for such materials, the DOE threat guidance  assumes that shielding would not exceed a level provided by the contents  of an average cargo container.
Moreover, DNDO has not completed efforts to fine-tune PVTs’ software  and thereby improve sensitivity to nuclear materials. As a result, the  criteria compare ASPs to the current performance of PVTs and do not take  potential improvements into account, which affects any assessment of  “significant” improvement over current technology. DNDO officials expect  they can achieve small improvements to PVTs’ performance through  additional development of “energy windowing,” a technique currently  being used in PVTs to provide greater sensitivity than otherwise possible.  Pacific Northwest National Laboratory officials responsible for developing  the technique also told us small improvements may be possible, and CBP  officials have repeatedly urged DNDO to investigate the potential of the  technique. DNDO collected the data needed to further develop energy  windowing during the 2008 performance testing at the Nevada Test Site  but has not yet funded Pacific Northwest National Laboratory efforts to  analyze the data and further develop the technique.
Other aspects of the criteria for a significant increase in operational  effectiveness require that ASPs either provide more than a marginal  improvement in addressing other limitations of current-generation  equipment or at least maintain the same level of performance in areas in  which the current-generation equipment is considered adequate:    The primary screening requirement for an 80 percent reduction in the rate  of innocent alarms could result in hundreds of fewer secondary screenings  per day, thereby reducing CBP’s workload and delays to commerce. The  actual reduction in the volume of innocent alarms would vary and would  be greatest at the nation’s busiest ports of entry, such as Los Angeles/Long  Beach, where CBP officials report that PVTs generate up to about 600  innocent alarms per day. A DNDO official said the requirement for an 80  percent reduction in innocent alarms was developed in conjunction with  CBP and was based on a level that would provide meaningful workload  relief.
The primary screening criteria requiring that ASPs provide at least the  same level of sensitivity to plutonium and medical and industrial isotopes,  but not specifying an improvement, were based on DNDO’s assessment  that PVTs adequately detect such materials, which have a stronger  radiation signature than HEU. In addition, CBP officials said that  including medical and industrial isotopes in the criteria addressed a CBP  requirement for verifying that those transporting certain quantities of  these materials into the United States are properly licensed.
The secondary screening requirement that ASPs reduce the probability of  misidentifying special nuclear material by one-half addresses the inability  of relatively small handheld devices to consistently locate and identify  potential threats in large cargo containers. For example, a handheld device  may fail to correctly identify special nuclear material if the material is  well-shielded or the device is not placed close enough to a radiation  source to obtain a recognizable measurement. According to CBP and  DNDO, the requirement for a reduction in the average time to conduct  secondary screenings is not more specific because the time varies  significantly among ports of entry and types of cargo being screened.
section_title: DHS Increased the Rigor of Advanced Portal Monitor Testing
paragraphs: Improvements to the 2008 round of testing addressed concerns we raised  about earlier rounds of ASP testing. However, the testing still had  limitations, and the preliminary results are mixed.
subsections: section_title: Improvements to Testing Provided Credibility to Test Results
paragraphs: As we testified in September 2008, DHS’s improvements to the 2008 round  of ASP testing addressed concerns we raised about previous tests. A  particular area of improvement was in the performance testing at the  Nevada Test Site, where DNDO compared the capability of ASP and  current-generation equipment to detect and identify nuclear and  radiological materials, including those that could be used in a nuclear  weapon. The improvements addressed concerns we previously raised  about the potential for bias and provided credibility to the results within  the limited range of scenarios tested by DNDO. For example, we reported  in 2007 that DNDO had allowed ASP contractors to adjust their systems  after preliminary runs using the same radiological materials that would be  used in the formal tests. In contrast, the plan for the 2008 performance test  stipulated that there would be no system contractor involvement in test  execution, and no ASP contractors were at the test location on the day we  observed performance testing. Furthermore, DNDO officials told us, and  we observed, that they did not conduct preliminary runs with threat  objects used in the formal tests. In 2007, we reported that DNDO did not  objectively test the handheld identification devices because it did not  adhere to CBP’s standard operating procedure for using the devices to  conduct a secondary inspection, which is fundamental to the equipment’s  performance in the field. DNDO addressed this limitation in the 2008  round of performance testing: CBP officers operated the devices and  adhered as closely to the standard operating procedure as test conditions  allowed. While the test conditions did not allow CBP officers to obtain  real-time technical support in interpreting the device’s measurements, as  they would in the field to increase the probability of correctly identifying a  radiation source, DNDO officials said they addressed this limitation. For  example, they treated a decision by a CBP officer to indicate the need for  technical support as a correct outcome if the test scenario involved the  use of a potential threat, such as HEU.
Other aspects of testing, while not specifically addressing concerns we  previously raised, also added credibility to the test results. Based on our  analysis of the performance test plan, we concluded that the test design  was sufficient to identify statistically significant differences between the  new technology and current-generation systems when there were  relatively large differences in performance. Specifically, DNDO conducted  a sufficient number of runs of each scenario used in the 2008 performance  testing to identify such differences.
With regard to the general conduct of the 2008 round of testing, two  aspects, in particular, enhanced the overall rigor of the tests: (1) criteria  for ensuring that ASPs met the requirements for each phase before  advancing to the next, and (2) the participation of CBP and the DHS  Science and Technology Directorate. The test and evaluation master plan  established criteria requiring that the ASPs have no critical or severe  issues rendering them completely unusable or impairing their function  before starting or completing any test phase. In addition, the criteria  established a cumulative limit of 10 issues requiring a work-around (e.g., a  straightforward corrective step, such as a minor change in standard  operating procedures) and 15 cosmetic issues not affecting proper  functioning. DNDO and CBP adhered to the criteria even though doing so  resulted in integration testing conducted at the Pacific Northwest National  Laboratory taking longer than anticipated and delaying the start of field  validation. For example, DNDO and CBP did not allow a vendor’s ASP  system to complete integration testing until all critical or severe issues had  been resolved.
The involvement of CBP and the DHS Science and Technology Directorate  provided an independent check, within DHS, of DNDO’s efforts to develop  and test the new portal monitors. For example, the lead CBP official  involved in ASP testing told us that DNDO provided an initial assessment  of the severity of issues uncovered during testing, but CBP made the final  decision on categorizing them as critical, severe, work-around, or  cosmetic issues. CBP also added a final requirement to integration testing  before proceeding to field validation to demonstrate ASPs’ ability to  operate for 40 hours without additional problems. According to CBP  officials, their efforts to resolve issues prior to field validation reflect the  importance CBP places on ensuring that ASPs are sufficiently stable and  technically mature to operate effectively in a working port of entry and  thereby provide for a productive field validation.
The DHS Science and Technology Directorate, which is responsible for  developing and implementing the department’s test and evaluation policies  and standards, will have the lead role in the final phase of ASP testing; the  final phase, consisting of 21 days of continuous operation, is scheduled to  begin at one seaport after the completion of field validation. The Science  and Technology Directorate identified two critical questions to be  addressed through operational testing: (1) Will the ASP system improve  operational effectiveness (i.e., detection and identification of threats)  relative to the current-generation system, and (2) is the ASP system  suitable for use in the operational environment at land and sea ports of  entry? The suitability of ASPs includes factors such as reliability,  maintainability, and supportability. Because the operational testing  conducted at one seaport is not sufficient to fully answer these  questions—for example, because the testing will not allow threat objects  to be inserted into cargo containers—the directorate plans to also conduct  an independent analysis of the results from previous test phases, including  performance testing.
The 2008 testing still had limitations, which do not detract from the test  results’ credibility but do require that results be appropriately qualified.  Limitations included the following:    The number of handheld identification device measurements collected  during performance testing was sufficient to distinguish only particularly  large differences from ASPs’ identification ability. In particular, the  standard operating procedure for conducting secondary inspections using  ASPs, which requires less time than when using handheld devices, allowed  DNDO to collect more than twice as many ASP measurements and to test  ASPs’ identification ability against more radiation sources than used to  test handheld identification devices.
The performance test results cannot be generalized beyond the limited set  of scenarios tested. For example, DNDO used a variety of masking and  shielding scenarios designed to include cases where both systems had 100  percent detection, cases where both had zero percent detection, and  several configurations in between so as to estimate the point where  detection capability ceased. However, the scenarios did not represent the  full range of possibilities for concealing smuggled nuclear or radiological  material. For example, DNDO only tested shielding and masking scenarios  separately, to differentiate between the impacts of shielding and masking  on the probabilities of detection and identification. As a result, the  performance test results cannot show how well each system would detect  and identify nuclear or radiological material that is both shielded and  masked, which might be expected in an actual smuggling incident.  Similarly, DNDO used a limited number of threat objects to test ASPs’  detection and identification performance, such as weapons-grade  plutonium but not reactor-grade plutonium, which has a different isotopic  composition. A report on special testing of ASPs conducted by Sandia  National Laboratories in 2007 recommended that future tests use  plutonium sources having alternative isotopic compositions. Sandia based  its recommendations on results showing that the performance of ASP  systems varied depending on the isotopic composition of plutonium.
The Science and Technology Directorate’s operational testing is designed  to demonstrate that the average time between equipment failures (the  measure of ASPs’ reliability) is not less than 1,000 hours. Thus, the testing  will not show how reliable the equipment will be over a longer term. DHS  Science and Technology Directorate officials recognize this limitation and  said they designed operational testing only to demonstrate compliance  with the ASP performance specification. Furthermore, to the extent that  the Science and Technology Directorate relies on performance test results  to evaluate ASPs’ ability to detect and identify threats, its analysis of ASPs’  effectiveness will be subject to the same limitations as the original testing  and analysis conducted by DNDO.
section_title: Preliminary Test Results Are Mixed
paragraphs: The preliminary results presented to us by DNDO are mixed, particularly  in the capability of ASPs used for primary screening to detect certain  shielded nuclear materials. However, we did not obtain DNDO’s final  report on performance testing conducted at the Nevada Test Site until  early April 2009, and thus we had limited opportunity to evaluate the  report. In addition, we are not commenting on the degree to which the  final report provides a fair representation of ASPs’ performance.  Preliminary results from performance testing show that the new portal  monitors detected certain nuclear materials better than PVTs when  shielding approximated DOE threat guidance, which is based on light  shielding. In contrast, differences in system performance were less notable  when shielding was slightly increased or decreased: Both the PVTs and  ASPs were frequently able to detect certain nuclear materials when  shielding was below threat guidance, and both systems had difficulty  detecting such materials when shielding was somewhat greater than threat  guidance. DNDO did not test ASPs or PVTs against moderate or greater  shielding because such scenarios are beyond both systems’ ability. (See  fig. 3 for a summary of performance test results for detection of certain  nuclear materials.)
With regard to secondary screening, ASPs performed better than handheld  devices in identification of threats when masked by naturally occurring  radioactive material. However, differences in the ability to identify certain  shielded nuclear materials depended on the level of shielding, with  increasing levels appearing to reduce any ASP advantages over the  handheld identification devices—another indication of the fundamental  limitation of passive radiation detection.
Other phases of testing, particularly integration testing, uncovered multiple  problems meeting requirements for successfully integrating the new  technology into operations at ports of entry. Of the two ASP vendors  participating in the 2008 round of testing, one has fallen several months  behind in testing due to the severity of the problems it encountered during  integration testing; the problems were so severe that it may have to redo  previous test phases to be considered for certification. The other vendor’s  system completed integration testing, but CBP suspended field validation of  the system after 2 weeks because of serious performance problems that may  require software revisions. In particular, CBP found that the performance  problems resulted in an overall increase in the number of referrals for  secondary screening compared to the existing equipment. According to CBP,  this problem will require significant corrective actions before testing can  resume; such corrective actions could in turn change the ability of the ASP  system to detect threats. The problem identified during field validation was in  addition to ones identified during integration testing, which required multiple  work-arounds and cosmetic changes before proceeding to the next test phase.  For example, one problem requiring a work-around related to the amount of  time it takes for the ASP to sound an alarm when a potential threat material  has been detected. Specifications require that ASPs alarm within two seconds  of a vehicle exiting the ASP. However, during testing, the vendor’s ASP took  longer to alarm when a particular isotope was detected. The work-around to  be implemented during field validation requires that all vehicles be detained  until cleared by the ASP; the effect on commerce must ultimately be  ascertained during field validation.
CBP officials anticipate that they will continue to uncover problems  during the first few years of use if the new technology is deployed in the  field. The officials do not necessarily regard such problems to be a sign  that testing was not rigorous but rather a result of the complexity and  newness of the technology and equipment.
section_title: Schedule Delays Have Allowed More Time for Analysis and Review of Test Results, but DNDO’s Latest Schedule Does Not Include Computer Simulations to Provide Additional Insight into ASP Capabilities
paragraphs: Delays to the schedule for the 2008 round of testing have allowed more  time for analysis and review of results, particularly from performance  testing conducted at the Nevada Test Site. The original schedule, which  underestimated the time needed for testing, anticipated completion of  testing in mid-September 2008 and the DHS Secretary’s decision on ASP  certification between September and November 2008. DHS officials  acknowledged that scheduling a certification decision shortly after  completion of testing would leave limited time to complete final test  reports and said the DHS Secretary could rely instead on preliminary  reports if the results were favorable to ASPs. DHS’s most recent schedule  anticipated a decision on ASP certification as early as May 2009, but DHS  has not updated its schedule for testing and certification since suspending  field validation in February 2009 due to ASP performance problems.
Problems uncovered during testing of ASPs’ readiness to be integrated  into operations at U.S. ports of entry have caused the greatest delays to  date and have allowed more time for DNDO to analyze and review the  results of performance testing. Integration testing was originally scheduled  to conclude in late July 2008 for both ASP vendors. The one ASP system  that successfully passed integration testing did not complete the test until  late November 2008—approximately 4 months behind schedule. (The  delays to integration testing were due in large part to the adherence of  DNDO and CBP to the criteria discussed earlier for ensuring that ASPs met  the requirements for each test phase.) In contrast, delays to performance  testing, which was scheduled to run concurrently with integration testing,  were relatively minor. Both ASP systems completed performance testing  in August 2008, about a month later than DNDO originally planned.
The schedule delays have allowed more time to conduct injection  studies—computer simulations for testing the response of ASPs and PVTs  to the radiation signatures of threat objects randomly “injected”  (combined) into portal monitor records of actual cargo containers  transported into the United States, including some containers with  innocent sources of radiation. However, DNDO does not plan to complete  the studies prior to the Secretary of Homeland Security’s decision on  certification even though DNDO and other officials have indicated that the  studies could provide additional insight into the capabilities and  limitations of advanced portal monitors. According to DNDO officials,  injection studies address the inability of performance testing conducted at  the Nevada Test Site to replicate the wide variety of cargo coming into the  United States and the inability to bring special nuclear material and other  threat objects to ports of entry and place them in cargo during field  validation. Similarly, while they acknowledged that injection studies have  limitations, DOE national laboratory officials said the studies can increase  the statistical confidence in comparisons of ASPs’ and PVTs’ probability of  detecting threats concealed in cargo because of the possibility of  supporting larger sample sizes than feasible with actual testing. A  February 2008 DHS independent review team report on ASP testing also  highlighted the benefits of injection studies, including the ability to  explore ASP performance against a large number of threat scenarios at a  practical cost and schedule and to permit an estimate of the minimum  detectable amount for various threats.
DNDO has the data needed to conduct the studies. It has supported efforts  to collect data on the radiation signatures for a variety of threat objects,  including special nuclear materials, as recorded by both ASP and PVT  systems. It has also collected about 7,000 usable “stream-of-commerce”  records from ASP and PVT systems installed at a seaport. Furthermore,  DNDO had earlier indicated that injection studies could provide  information comparing the performance of the two systems as part of the  certification process for both primary and secondary screening. However,  addressing deficiencies in the stream-of-commerce data delayed the  studies, and DNDO subsequently decided that performance testing would  provide sufficient information to support a decision on ASP certification.  DNDO officials said they would instead use injection studies to support  effective deployment of the new portal monitors.
section_title: Conclusions
paragraphs: Given that radiation detection equipment is already being used at ports of  entry to screen for smuggled nuclear or radiological materials, the  decision whether to replace existing equipment requires that the benefits  of the new portal monitors be weighed against the costs. DNDO  acknowledges that ASPs are significantly more expensive than PVTs to  deploy and maintain, and based on preliminary results from the 2008  testing, it is not yet clear that the $2 billion cost of DNDO’s deployment  plan is justified. Even if ASPs are able to reduce the volume of innocent  cargo referred for secondary screening, they are not expected to detect  certain nuclear materials that are surrounded by a realistic level of  shielding better than PVTs could. Preliminary results of DNDO’s  performance testing show that ASPs outperformed the PVTs in detection  of such materials during runs with light shielding, but ASPs’ performance  rapidly deteriorated once shielding was slightly increased. Furthermore,  DNDO and DOE officials acknowledged that the performance of both  portal monitors in detecting such materials with a moderate amount of  shielding would be similarly poor. This was one of the reasons that  performance testing did not include runs with a moderate level of  shielding.
Two additional aspects of the 2008 round of testing call into question  whether ASPs’ ability to provide a marginal improvement in detection of  nuclear materials and reduce innocent alarms warrants the cost of the new  technology. First, the DHS criteria for a significant increase in operational  effectiveness do not take into account recent efforts to improve the  current-generation portal monitors’ sensitivity to nuclear materials  through the “energy windowing” technique, most likely at a much lower  cost. Data on developing this technique were collected during the 2008  round of performance testing but have not been analyzed. Second, while  DNDO made improvements to the 2008 round of ASP testing that provided  credibility to the test results, its test schedule does not allow for  completion of injection studies prior to certification even though the  studies could provide additional insight into the performance of the new  technology. Without results from injection studies, the Secretary of  Homeland Security would have to make a decision on certification based  on a limited number of test scenarios conducted at the Nevada Test Site.
Assuming that the Secretary of Homeland Security certifies ASPs, CBP  officials anticipate that they will discover problems with the equipment  when they start using it in the field. Integration testing uncovered a  number of such problems, which delayed testing and resulted in ASP  vendors making multiple changes to their systems. Correcting such  problems in the field could prove to be more costly and time consuming  than correcting problems uncovered through testing, particularly if DNDO  proceeds directly from certification to full-scale deployment, as allowed  under the congressional certification requirement that ASPs provide a  significant increase in operational effectiveness.
section_title: Recommendations for Executive Action
paragraphs: We recommend that the Secretary of Homeland Security direct the  Director of DNDO to take the following two actions to ensure a sound  basis for a decision on ASP certification:    Assess whether ASPs meet the criteria for a significant increase in  operational effectiveness based on a valid comparison with PVTs’ full  performance potential, including the potential to further develop PVTs’  use of energy windowing to provide greater sensitivity to threats. Such a  comparison could also be factored into an updated cost-benefit analysis to  determine whether it would be more cost-effective to continue to use  PVTs or deploy ASPs for primary screening at particular ports of entry.
Revise the schedule for ASP testing and certification to allow sufficient  time for review and analysis of results from the final phases of testing and  completion of all tests, including injection studies.
If ASPs are certified, we further recommend that the Secretary of  Homeland Security direct the Director of DNDO to develop an initial  deployment plan that allows CBP to uncover and resolve any additional  problems not identified through testing before proceeding to full-scale  deployment—for example, by initially deploying ASPs at a limited number  of ports of entry.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to DOE and DHS for their review and  comment. DOE provided technical comments, which we have  incorporated into our report as appropriate. DHS’s written comments are  reproduced in appendix II.
DHS agreed in part with our recommendations. Specifically, DHS stated  that it believes its plan to deploy ASPs in phases, starting at a small  number of low-impact locations, is in accordance with our  recommendation to develop an initial deployment plan that allows  problems to be uncovered and resolved prior to full-scale deployment. We  agree that this deployment plan would address our recommendation and  note that DHS’s comments are the first indication provided to us of the  department’s intention to pursue such a plan.
In contrast, DHS did not concur with our recommendations to (1) assess  whether ASPs meet the criteria for a significant increase in operational  effectiveness based on a comparison with PVTs’ full potential, including  further developing PVTs’ use of energy windowing; and (2) revise the ASP  testing and certification schedule to allow sufficient time for completion of  all tests, including injection studies. With regard to energy windowing,  DHS stated that using current PVT performance as a baseline for  comparison is a valid approach because the majority of increased PVT  performance through energy windowing has already been achieved. While  DHS may be correct, its assessment is based on expert judgment rather  than the results of testing and analysis being considered by the department  to optimize the use of energy windowing. Given the marginal increase in  sensitivity required of ASPs, we stand by our recommendation to assess  ASPs against PVTs’ full potential. DHS can then factor PVTs’ full potential  into a cost-benefit analysis prior to acquiring ASPs. On this point, DHS  commented that its current cost-benefit analysis is a reasonable basis to  guide programmatic decisions. However, upon receiving DHS’s comments,  we contacted DNDO to obtain a copy of its cost-benefit analysis and were  told the analysis is not yet complete.
With regard to injection studies, DHS agreed that the schedule for ASP  certification must allow sufficient time for review and analysis of test  results but stated that DHS and DOE experts concluded injection studies  were not required for certification. DHS instead stated that the series of  ASP test campaigns would provide a technically defensible basis for  assessing the new technology against the certification criteria. However,  DHS did not rebut the reasons we cited for conducting injection studies  prior to certification, including test delays that have allowed more time to  conduct the studies and the ability to explore ASP performance against a  large number of threat scenarios at a practical cost and schedule. On the  contrary, DHS acknowledged the delays to testing and the usefulness of  injection studies. Given that each phase of testing has revealed new  information about the capabilities and limitations of ASPs, we believe  conducting injection studies prior to certification would likely offer similar  insights and would therefore be prudent prior to a certification decision.
DHS provided additional comments regarding our assessment of the  relative sensitivity of ASPs and PVTs and our characterization of the  severity of the ASPs’ software problems uncovered during field validation.  With regard to sensitivity, DHS implied that our characterization of the  relative ability of ASPs and PVTs is inaccurate and misleading because we  did not provide a complete analysis of test results. We disagree. First, in  meetings to discuss the preliminary results of performance testing  conducted at the Nevada Test Site, DNDO officials agreed with our  understanding of the ability of ASPs and PVTs deployed for primary  screening to detect shielded nuclear materials. Furthermore, contrary to  the assertion that a complete analysis requires a comparison of ASPs to  handheld identification devices, our presentation is consistent with DHS’s  primary screening criterion for detection of shielded nuclear materials,  which only requires that ASPs be compared with PVTs. Finally, while we  agree that the performance test results require a more complete analysis,  DNDO did not provide us with its final performance test report until early  April 2009, after DHS provided its comments on our draft report. In the  absence of the final report, which DNDO officials told us took longer than  anticipated to complete, we summarized the preliminary results that  DNDO presented to us during the course of our review as well as to  congressional stakeholders.
With regard to ASP software problems uncovered during field validation,  we clarified our report in response to DHS’s comment that the severity of  the problems has not yet been determined. DHS stated that its preliminary  analysis indicates the problems should be resolved by routine adjustments  to threshold settings rather than presumably more significant software  “revisions.” However, given the history of lengthy delays during ASP  testing, we believe that DHS’s assessment of the severity of problems  encountered during field validation may be overly optimistic.
As agreed with your offices, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date. At that time, we will send copies to the Secretaries of  Homeland Security and Energy; the Administrator of NNSA; and interested  congressional committees. The report will also be available at no charge  on the GAO Web site at http://www.gao.gov.
If you or your staffs have any questions about this report, please contact  me at (202) 512-3841 or aloisee@gao.gov. Contact points for our Offices of  Congressional Relations and Public Affairs may be found on the last page  of this report. GAO staff who made key contributions to this report are  listed in appendix III.
section_title: Appendix I: Scope and Methodology
paragraphs: To evaluate the degree to which Department of Homeland Security’s  (DHS) criteria for a significant increase in operational effectiveness  address the limitations of the current generation of radiation detection  equipment, we clarified the intent of the criteria through the Domestic  Nuclear Detection Office’s (DNDO) written answers to our questions and  through interviews with U.S. Customs and Border Protection (CBP)  officials. We also took steps to gain a fuller understanding of the strengths  and limitations of the current-generation equipment, which the criteria use  as a baseline for evaluating the effectiveness of advanced spectroscopic  portals (ASP). In particular, we obtained copies of the Department of  Energy (DOE) threat guidance and related documents used to set  polyvinyl toluene (PVT) thresholds for detection of nuclear materials. We  interviewed DOE and national laboratory officials responsible for the  threat guidance about the process for developing it and the basis for its  underlying assumptions, including shielding levels. We also interviewed  DNDO and Pacific Northwest National Laboratory officials regarding the  extent to which PVTs currently deployed at ports of entry meet the  guidance and the development and use of energy windowing to enhance  PVTs’ sensitivity to nuclear materials.
To evaluate the rigor of the 2008 round of testing as a basis for  determining ASPs’ operational effectiveness, we reviewed the test and  evaluation master plan and plans for individual phases of testing, including  system qualification testing conducted at vendors’ facilities, performance  testing conducted at the Nevada Test Site for evaluating ASP detection and  identification capabilities, and integration testing conducted at Pacific  Northwest National Laboratory for evaluating the readiness of ASPs to be  used in an operational environment at ports of entry. We also reviewed  draft plans for field validation conducted at CBP ports of entry and the  DHS Science and Technology Directorate’s independent operational test  and evaluation. In reviewing these documents, we specifically evaluated  the extent to which the performance test design was sufficient to identify  statistically significant differences between the ASP and current- generation systems and whether DHS had addressed our concerns about  previous rounds of ASP testing. We interviewed DNDO, CBP, and other  DHS officials responsible for conducting and monitoring tests, and we  observed, for one day each, performance testing at the Nevada Test Site  and integration testing at DOE’s Pacific Northwest National Laboratory.  We also interviewed representatives of entities that supported testing,  including DOE’s National Nuclear Security Administration and Pacific  Northwest National Laboratory, the National Institute of Standards and  Technology, and the Johns Hopkins University Applied Physics  Laboratory. We reviewed the DHS independent review team report of  previous ASP testing conducted in 2007, and we interviewed the chair of  the review team to clarify the report’s findings. Finally, we examined  preliminary or final results for the phases of testing completed during our  review, and we interviewed DNDO and CBP officials regarding the results.
To evaluate the test schedule, we analyzed the initial working schedule  DNDO provided to us in May 2008 and the schedule presented in the  August 2008 test and evaluation master plan, and we tracked changes to  the schedule and the reasons for any delays. We interviewed DNDO and  other officials with a role in testing to determine the amount of time  allowed for analysis and review of results. We interviewed DNDO and  Pacific Northwest National Laboratory officials regarding the injection  studies, including reasons for delays in the studies and plans for including  the results as part of the ASP certification process.
We conducted this performance audit from May 2008 to May 2009 in  accordance with generally accepted government auditing standards. Those  standards require that we plan and perform the audit to obtain sufficient,  appropriate evidence to provide a reasonable basis for our findings and  conclusions based on our audit objectives. We believe that the evidence  obtained provides a reasonable basis for our findings and conclusions  based on our audit objectives.
section_title: Appendix II: Comments from the Department of Homeland Security
section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Ned Woodward, Assistant  Director; Dr. Timothy Persons, Chief Scientist; James Ashley; Steve  Caldwell; Joseph Cook; Omari Norman; Alison O’Neill; Rebecca Shea;  Kevin Tarmann; and Eugene Wisnoski made key contributions to this  report.