Combating Nuclear Smuggling: Lessons Learned from DHS Testing of Advanced Radiation Detection Portal Monitors
Published: 2009-06-25T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Department of Homeland Security's (DHS) Domestic Nuclear Detection Office (DNDO) is responsible for addressing the threat of nuclear smuggling. Radiation detection portal monitors are key elements in the nation's defenses against such threats. DHS has sponsored testing to develop new monitors, known as advanced spectroscopic portal (ASP) monitors, to replace radiation detection equipment being used at ports of entry. DNDO expects that ASPs may offer improvements over current-generation portal monitors, particularly the potential to identify as well as detect radioactive material and thereby to reduce both the risk of missed threats and the rate of innocent alarms, which DNDO considers to be key limitations of radiation detection equipment currently used by Customs and Border Protection (CBP) at U.S. ports of entry. However, ASPs cost significantly more than current generation portal monitors. Due to concerns about ASPs' cost and performance, Congress has required that the Secretary of Homeland Security certify that ASPs provide a significant increase in operational effectiveness before obligating funds for full-scale ASP procurement. This testimony addresses (1) GAO findings on DNDO's latest round of ASP testing, and (2) lessons from ASP testing that can be applied to other DHS technology investments. These findings are based on GAO's May 2009 report GAO-09-655 and other related reports.
section_title: What GAO Found
paragraphs: GAO's report on the latest round of ASP testing found that DHS increased the rigor in comparison with previous tests and thereby added credibility to the test results. However, GAO's report also questioned whether the benefits of the ASPs justify the high cost. In particular, the DHS criteria for a significant increase in operational effectiveness require only a marginal improvement in the detection of certain weapons-usable nuclear materials, which DNDO considers a key limitation of current-generation portal monitors. The marginal improvement required of ASPs is particularly notable given that DNDO has not completed efforts to fine-tune current-generation equipment to provide greater sensitivity. Moreover, the preliminary test results show that ASPs performed better than current-generation portal monitors in detection of such materials concealed by light shielding approximating the threat guidance for setting detection thresholds, but that differences in sensitivity were less notable when shielding was slightly below or above that level. Finally, DNDO has not yet updated its cost-benefit analysis to take into account the results of the latest round of ASP testing and does not plan to complete computer simulations that could provide additional insight into ASP capabilities and limitations prior to certification even though test delays have allowed more time to conduct the simulations. DNDO officials believe the other tests are sufficient for ASPs to demonstrate a significant increase in operational effectiveness. GAO recommended that DHS assess ASPs against the full potential of current-generation equipment and revise the program schedule to allow time to conduct computer simulations and to uncover and resolve problems with ASPs before full-scale deployment. DHS agreed to a phased deployment that should allow time to uncover ASP problems but disagreed with the other recommendations, which GAO believes remain valid. The challenges DNDO has faced in developing and testing ASPs illustrate the importance of following best practices for investments in complex homeland security acquisitions and for testing of new technologies. GAO recently found that many major DHS investments, including DNDO's ASP program, had not met the department's requirements for basic acquisition documents necessary to inform the investment review process, which has adopted many acquisition best practices. As a result, DHS had not consistently provided the oversight needed to identify and address cost, schedule, and performance problems in its major investments. A primary lesson to be learned regarding testing is that the push to replace existing equipment with the new portal monitors led to an ASP testing program that until recently lacked the necessary rigor. Even for the most recent round of testing, DNDO's schedule consistently underestimated the time required to conduct tests and resolve problems uncovered during testing. In contrast, GAO has previously found that testing programs designed to validate a product's performance against increasing standards for different stages in product development are a best practice for acquisition strategies for new technologies. Aspects that improved the latest round of ASP testing could also, if properly implemented, provide rigor to DHS's testing of other advanced technologies.

section_title: Letter
paragraphs: I am pleased to be here today to discuss GAO’s work on the Department of  Homeland Security’s (DHS) testing of advanced spectroscopic portal  (ASP) radiation detection monitors. As you are aware, the national  security mission of U.S. Customs and Border Protection (CBP), an agency  within DHS, includes screening for smuggled nuclear or radiological  material that could be used in a nuclear weapon or radiological dispersal  device (a “dirty bomb”). To screen cargo at ports of entry, CBP conducts  primary inspections with radiation detection equipment called portal  monitors—large stationary detectors through which cargo containers and  vehicles pass as they enter the United States. When radiation is detected,  CBP conducts secondary inspections using a second portal monitor to  confirm the original alarm and a handheld radioactive isotope  identification device to identify the radiation’s source and determine  whether it constitutes a threat.
The polyvinyl toluene (PVT) portal monitors currently in use can detect  radiation but cannot identify the type of material causing an alarm. As a  result, the monitors’ radiation alarms can be set off even by benign,  naturally occurring radioactive material. One way to reduce the rate of  such innocent alarms—and thereby minimize unnecessary secondary  inspections and enhance the flow of commerce—is to adjust the  operational thresholds (i.e., operate the PVTs at a reduced level of  sensitivity). However, reducing the sensitivity may make it more difficult  to detect certain nuclear materials.
To address the limitations of current-generation portal monitors, DHS’s  Domestic Nuclear Detection Office (DNDO) in 2005 began to develop and  test ASPs, which are designed to both detect radiation and identify the  source. DNDO hopes to use the new portal monitors to replace at least  some PVTs currently used for primary screening, as well as PVTs and  handheld identification devices currently used for secondary screening.  However, in September 2008, we estimated the lifecycle cost of each  standard cargo version of the ASP (including deployment costs) to be  about $822,000, compared with about $308,000 for the PVT standard cargo  portal, and the total program cost for DNDO’s latest plan for deploying  radiation portal monitors—which relies on a combination of ASPs and  PVTs and does not deploy radiation portal monitors at all border  crossings—to be about $2 billion.
Concerned about the performance and cost of the ASP monitors, Congress  required the Secretary of Homeland Security to certify that the monitors  will provide a “significant increase in operational effectiveness” before  DNDO obligates funds for full-scale ASP procurement. The Secretary  must submit separate certifications for primary and secondary inspection.  In response, CBP, DNDO, and the DHS management directorate jointly  issued criteria in July 2008 for determining whether the new technology  provides a significant increase in operational effectiveness. The primary  screening criteria require that the new portal monitors detect potential  threats as well as or better than PVTs, show improved performance in  detection of highly enriched uranium (HEU), and reduce innocent alarms.  To meet the secondary screening criteria, the new portal monitors must  reduce the probability of misidentifying special nuclear material (e.g.,  HEU and plutonium) and the average time to conduct secondary  screenings.
DNDO designed and coordinated a new series of tests, originally  scheduled to run from April 2008 through September 2008, to determine  whether the new portal monitors meet the certification criteria for primary  and secondary screening and are ready for deployment. Key phases of this  testing program include concurrent testing led by DNDO of the new and  current equipment’s ability to detect and identify threats and of ASPs’  readiness to be integrated into operations for both primary and secondary  screening at ports of entry; field validation led by CBP at four northern  and southern border crossings and two seaports; and an independent  evaluation, led by the DHS Science and Technology Directorate at one of  the seaports, of the new portal monitors’ effectiveness and suitability.
Since 2006, we have issued six reports and four testimonies on  development of radiation detection portal monitors, including today’s  testimony, and have made 19 recommendations for improving DNDO’s  efforts to develop and test portal monitors. Our concerns have focused on  key areas in which DNDO’s efforts have lacked the necessary rigor given  ASPs’ high cost and the importance of the radiation portal monitor project  to our national security. These areas include objective and realistic testing  of ASPs’ performance in comparison with the performance of current- generation equipment; full disclosure and reporting of the limitations of  tests used to support a decision by the Secretary of Homeland Security on  ASP certification; development of a cost estimate that considers the full  costs of the plan for deploying radiation detection portal monitors; and  development of a cost-benefit analysis based on ASPs’ demonstrated  performance and a complete accounting of the portal monitor project’s  costs. (App. I presents a summary of our key findings and  recommendations related to ASPs.) As I will discuss today, DNDO has  improved the rigor of testing but has not yet updated the cost-benefit  analysis that is critical to a decision on whether to replace radiation  detection equipment already deployed at ports of entry with the  significantly more expensive ASPs.
Specifically, my testimony discusses (1) our key findings on the most  recent round of ASP testing and (2) lessons from ASP testing that can be  applied to other DHS technology investments. These findings are based on  our report released this week and other related GAO reports. We  conducted this performance audit work in June 2009 in accordance with  generally accepted government auditing standards. Those standards  require that we plan and perform the audit to obtain sufficient, appropriate  evidence to produce a reasonable basis for our findings and conclusions  based on our audit objectives. We believe that the evidence obtained  provides a reasonable basis for our statement today.
subsections: section_title: The Latest Round of Testing Highlights the Limitations of ASPs
paragraphs: Our report on the latest round of ASP testing found that DHS increased the  rigor of ASP testing in comparison with previous tests and that a particular  area of improvement was in the performance testing at the Nevada Test  Site, where DNDO compared the capability of ASP and current-generation  equipment to detect and identify nuclear and radiological materials. For  example, unlike in prior tests, the plan for the 2008 performance test  stipulated that there would be no system contractor involvement in test  execution. Such improvements addressed concerns we previously raised  about the potential for bias and provided credibility to the results.
Nevertheless, based on the following factors, we continue to question  whether the benefits of the new portal monitors justify the high cost:    The DHS criteria for a significant increase in operational effectiveness.
Our chief concern with the criteria is that they require a marginal  improvement over current-generation portal monitors in the detection of  certain weapons-usable nuclear materials when ASPs are deployed for  primary screening. DNDO considers detection of such materials to be a  key limitation of current-generation portal monitors. We are particularly  concerned about the marginal improvement required of ASPs because the  detection threshold for the current-generation portal monitors does not  specify a level of radiation shielding that smugglers could realistically use.  DOE and national laboratory officials told us that DOE’s threat guidance  used to set the current detection threshold is based not on an analysis of  the capabilities of potential smugglers to take effective shielding measures  but rather on the limited sensitivity of PVTs to detect anything more than  certain lightly shielded nuclear materials. DNDO officials acknowledge  that both the new and current-generation portal monitors are capable of  detecting certain nuclear materials only when unshielded or lightly  shielded. The marginal improvement in detection of such materials  required of ASPs is particularly notable given that DNDO has not  completed efforts to fine-tune PVTs’ software and thereby improve  sensitivity to nuclear materials. DNDO officials expect they can achieve  small improvements in sensitivity, but DNDO has not yet funded efforts to  fine-tune PVTs’ software. In contrast to the marginal improvement  required in detection of certain nuclear materials, the primary screening  requirement to reduce the rate of innocent alarms could result in hundreds  of fewer secondary screenings per day, thereby reducing CBP’s workload  and delays to commerce. In addition, the secondary screening criteria,  which require ASPs to reduce the probability of misidentifying special  nuclear material by one-half, address the inability of relatively small  handheld devices to consistently locate and identify potential threats in  large cargo containers.
Preliminary results of performance testing and field validation. The  preliminary results presented to us by DNDO are mixed, particularly in the  capability of ASPs used for primary screening to detect certain shielded  nuclear materials. Preliminary results show that the new portal monitors  detected certain nuclear materials better than PVTs when shielding  approximated DOE threat guidance, which is based on light shielding. In  contrast, differences in system performance were less notable when  shielding was slightly increased or decreased: Both the PVTs and ASPs  were frequently able to detect certain nuclear materials when shielding  was below threat guidance, and both systems had difficulty detecting such  materials when shielding was somewhat greater than threat guidance.  With regard to secondary screening, ASPs performed better than handheld  devices in identification of threats when masked by naturally occurring  radioactive material. However, differences in the ability to identify certain  shielded nuclear materials depended on the level of shielding, with  increasing levels appearing to reduce any ASP advantages over the  handheld identification devices. Other phases of testing uncovered  multiple problems in meeting requirements for successfully integrating the  new technology into operations at ports of entry. Of the two ASP vendors  participating in the 2008 round of testing, one has fallen behind due to  severe problems encountered during testing of ASPs’ readiness to be  integrated into operations at ports of entry (“integration testing”); the  problems may require that the vendor redo previous test phases to be  considered for certification. The other vendor’s system completed  integration testing, but CBP suspended field validation after 2 weeks  because of serious performance problems resulting in an overall increase  in the number of referrals for secondary screening compared with existing  equipment.
DNDO’s plans for computer simulations. DNDO does not plan to  complete injection studies—computer simulations for testing the response  of ASPs and PVTs to simulated threat objects concealed in cargo  containers—prior to the Secretary of Homeland Security’s decision on  certification even though delays to the ASP test schedule have allowed  more time to conduct the studies. According to DNDO officials, injection  studies address the inability of performance testing to replicate the wide  variety of cargo coming into the United States and the inability to place  special nuclear material and other threat objects in cargo during field  validation. DNDO had earlier indicated that injection studies could provide  information comparing the performance of the two systems as part of the  certification process for both primary and secondary screening. However,  DNDO subsequently decided that performance testing would provide  sufficient information to support a decision on ASP certification. DNDO  officials said they would instead use injection studies to support effective  deployment of the new portal monitors.
Lack of an updated cost-benefit analysis. DNDO has not yet updated its  cost-benefit analysis to take into account the results of the latest round of  ASP testing. An updated analysis that takes into account the results from  the latest round of testing, including injection studies, might show that  DNDO’s plan to replace existing equipment with ASPs is not justified,  particularly given the marginal improvement in detection of certain  nuclear materials required of ASPs and the potential to improve the  current-generation portal monitors’ sensitivity to nuclear materials, most  likely at a lower cost. DNDO officials said they are currently updating the  ASP cost-benefit analysis and plan to complete it prior to a decision on  certification by the Secretary of Homeland Security.
Our report recommended that the Secretary of Homeland Security direct  DNDO to (1) assess whether ASPs meet the criteria for a significant  increase in operational effectiveness based on a valid comparison with  PVTs’ full performance potential and (2) revise the schedule for ASP  testing and certification to allow sufficient time for review and analysis of  results from the final phases of testing and completion of all tests,  including injection studies. We further recommended that, if ASPs are  certified, the Secretary direct DNDO to develop an initial deployment plan  that allows CBP to uncover and resolve any additional problems not  identified through testing before proceeding to full-scale deployment. DHS  agreed to a phased deployment that should allow time to uncover ASP  problems but disagreed with GAO’s other recommendations, which we  continue to believe remain valid.
section_title: Procurement Decisions for New Technologies Require Rigorous Testing and Thorough Analysis of Results
paragraphs: The challenges DNDO has faced in developing and testing ASPs illustrate  the importance of following existing DHS policies as well as best practices  for investments in complex homeland security acquisitions and for testing  of new technologies. The DHS investment review process calls for  executive decision making at key points in an investment’s life cycle and  includes many acquisition best practices that, if applied consistently, could  help increase the chances for successful outcomes. However, we reported  in November 2008 that, for the period from fiscal year 2004 through the  second quarter of fiscal year 2008, DHS had not effectively implemented or  adhered to its investment review process due to a lack of senior  management officials’ involvement as well as limited monitoring and  resources. In particular, of DHS’s 48 major investments requiring  milestone and annual reviews under the department’s investment review  policy, 45 were not assessed in accordance with this policy. In addition,  many major investments, including DNDO’s ASP program, had not met the  department’s requirements for basic acquisition documents necessary to  inform the investment review process. As a result, DHS had not  consistently provided the oversight needed to identify and address cost,  schedule, and performance problems in its major investments. Among  other things, our November 2008 report recommended that the Secretary  of Homeland Security direct component heads, such as the Director of  DNDO, to ensure that the components have established processes to  manage major investments consistent with departmental policies. DHS  generally concurred with our recommendations, and we noted that DHS  had begun several efforts to address shortcomings in the investment  review process identified in our report, including issuing an interim  directive requiring DHS components to align their internal policies and  procedures by the end of the third quarter of fiscal year 2009. In January  2009, DHS issued a memorandum instructing component heads to create  acquisition executives in their organizations to be responsible for  management and oversight of component acquisition processes. If fully  implemented, these steps should help ensure that DHS components have  established processes to manage major investments.
Based on our body of work on ASP testing, one of the primary lessons to  be learned is to avoid the pitfalls in testing that stem from a rush to  procure new technologies. GAO has previously reported on the negative  consequences of pressures imposed by closely linking testing and  development programs with decisions to procure and deploy new  technologies, including the creation of incentives to postpone difficult  tests and limit open communication about test results. We found that  testing programs designed to validate a product’s performance against  increasing standards for different stages in product development are a best  practice for acquisition strategies for new technologies. In the case of  ASPs, the push to replace existing equipment with the new portal monitors  led to a testing program that until recently lacked the necessary rigor.  Even for the most recent round of testing, DNDO’s schedule consistently  underestimated the time required to conduct tests, resolve problems  uncovered during testing, and complete key documents, including final  test reports. In addition, DNDO’s original working schedule did not  anticipate the time required to update its cost-benefit analysis to take into  account the latest test results. The schedule anticipated completion of  testing in mid-September 2008 and the DHS Secretary’s decision on ASP  certification between September and November 2008. However, testing is  still not completed, and DNDO took months longer than anticipated to  complete the final report on performance testing.
As previously mentioned, a number of aspects of the latest round of ASP  testing increased the rigor in comparison with earlier rounds and, if  properly implemented, could improve the rigor in DHS’s testing of other  advanced technologies. Key aspects included the following:    Criteria for ensuring test requirements are met. The test and evaluation  master plan established criteria requiring that the ASPs meet certain  requirements before starting or completing any test phase. For example,  the plan required that ASPs have no critical or severe issues rendering  them completely unusable or impairing their function. The criteria  provided a formal means to ensure that ASPs met certain basic  requirements prior to the start of each phase of testing. DNDO and CBP  adhered to the criteria even though doing so resulted in integration testing  taking longer than anticipated and delaying the start of field validation.
Participation of the technology end user. The participation of CBP (the  end user of the new portal monitors) provided an independent check,  within DHS, of DNDO’s efforts to develop and test the new portal  monitors. For example, CBP added a final requirement to integration  testing before proceeding to field validation to demonstrate ASPs’ ability  to operate for 40 hours without additional problems and thereby provide  for a productive field validation. In addition, the participation of CBP  officers in the 2008 round of performance testing allowed DNDO to adhere  more closely than in previous tests to CBP’s standard operating procedure  for conducting a secondary inspection using the handheld identification  devices, thereby providing for an objective test.
Participation of an independent test authority. The DHS Science and  Technology Directorate, which is responsible for developing and  implementing the department’s test and evaluation policies and standards,  will have the lead role in the final phase of ASP testing and thereby  provide an additional independent check on testing efforts. The Science  and Technology Directorate identified two critical questions, related to  ASPs’ operational effectiveness (i.e., detection and identification of  threats) and suitability (e.g., reliability, maintainability, and  supportability), and drafted its own test plan to address those questions.
Mr. Chairman, this completes my prepared statement. I would be happy to  respond to any questions that you or other Members of the Subcommittee  may have at this time.
section_title: GAO Contact and Staff Acknowledgments
paragraphs: For further information about this testimony, please contact me at (202)  512-3841 or aloisee@gao.gov. Ned Woodward (Assistant Director), Joseph  Cook, and Kevin Tarmann made key contributions to this testimony. Dr.  Timothy Persons (Chief Scientist), James Ashley, Steve Caldwell, John  Hutton, Omari Norman, Alison O’Neill, Amelia Shachoy, and Rebecca Shea  also made important contributions.
section_title: Appendix I: Key Findings and Recommendations from Related GAO Products on Testing and Development of ASPs
paragraphs: Combating Nuclear Smuggling: DHS Has Made Progress  Deploying Radiation Detection Equipment at U.S. Ports-of-Entry,  but Concerns Remain. GAO-06-389. Washington, D.C.: March 22, 2006.
Key findings. Prototypes of advanced spectroscopic portals (ASP) were  expected to be significantly more expensive than current-generation portal  monitors but had not been shown to be more effective. For example,  Domestic Nuclear Detection Office (DNDO) officials’ preliminary analysis  of 10 ASPs tested at the Nevada Test Site found that the new portal  monitors outperformed current-generation equipment in detecting  numerous small, medium-size, and threatlike radioactive objects and were  able to identify and dismiss most naturally occurring radioactive material.  However, the detection capabilities of both types of portal monitors  converged as the amount of source material decreased.
Recommendations. We recommended that the Secretary of Homeland  Security work with the Director of DNDO to analyze the benefits and costs  of deploying ASPs before any of the new equipment is purchased to  determine whether any additional detection capability is worth the  additional cost. We also recommended that the total program cost  estimate for the radiation portal monitor project be revised after  completion of the cost-benefit analysis.
Combating Nuclear Smuggling: DHS’s Cost-Benefit Analysis to  Support the Purchase of New Radiation Detection Portal Monitors  Was Not Based on Available Performance Data and Did Not Fully  Evaluate All the Monitors’ Costs and Benefits. GAO-07-133R.  Washington, D.C.: October 17, 2006.
Combating Nuclear Smuggling: DHS’s Decision to Procure and  Deploy the Next Generation of Radiation Detection Equipment Is  Not Supported by Its Cost-Benefit Analysis. GAO-07-581T.  Washington, D.C.: March 14, 2007.
Key findings. DNDO’s cost-benefit analysis issued in response to our  March 2006 recommendation did not provide a sound analytical basis for  DNDO’s decision to purchase and deploy ASPs. We identified a number of  problems with the analysis of both the performance of the new portal  monitors and the costs. With regard to performance, DNDO did not use the  results of its own tests and instead relied on assumptions of the new  technology’s anticipated performance level. In addition, the analysis  focused on identifying highly enriched uranium (HEU) and did not  consider how well the new portal monitors can correctly detect or identify  other dangerous radiological or nuclear materials. With regard to costs,  DNDO did not follow the DHS guidelines for performing cost-benefit  analyses and used questionable assumptions about the procurement costs  of portal monitor technology.
Recommendations. We recommended that DHS and DNDO conduct a new  cost-benefit analysis using sound analytical methods, including actual  performance data and a complete accounting of all major costs and  benefits as required by DHS guidelines, and that DNDO conduct realistic  testing for both ASPs and current-generation portal monitors.
Combating Nuclear Smuggling: DNDO Has Not Yet Collected Most  of the National Laboratories’ Test Results on Radiation Portal  Monitors in Support of DNDO’s Testing and Development  Program. GAO-07-347R. Washington, D.C.: March 9, 2007.
Key findings. DNDO had not collected a comprehensive inventory of  testing information on current-generation portal monitors. Such  information, if collected and used, could improve DNDO’s understanding  of how well portal monitors detect different radiological and nuclear  materials under varying conditions. In turn, this understanding would  assist DNDO’s future testing, development, deployment, and purchases of  portal monitors.
Recommendations. We recommended that the Secretary of Homeland  Security, working with the Director of DNDO, collect reports concerning  all of the testing of current-generation portal monitors and review the test  reports in order to develop an information database on how the portal  monitors perform in both laboratory and field tests on a variety of  indicators, such as their ability to detect specific radiological and nuclear  materials.
Combating Nuclear Smuggling: Additional Actions Needed to  Ensure Adequate Testing of Next Generation Radiation Detection  Equipment. GAO-07-1247T. Washington, D.C.: September 18, 2007.
Key findings. We found that tests conducted by DNDO in early 2007 were  not an objective and rigorous assessment of the ASPs’ capabilities.  Specifically, we raised concerns about DNDO using biased test methods  that enhanced the apparent performance of ASPs; not testing the  limitations of ASPs’ detection capabilities—for example, by not using a  sufficient amount of the type of materials that would mask or hide  dangerous sources and that ASPs would likely encounter at ports of entry;  and not using a critical Customs and Border Protection (CBP) standard  operating procedure that is fundamental to the performance of handheld  radiation detectors in the field.
Recommendations. We recommended that the Secretary of Homeland  Security delay Secretarial certification and full-scale production decisions  on ASPs until all relevant tests and studies had been completed and  limitations to tests and studies had been identified and addressed. We  further recommended that DHS determine the need for additional testing  in cooperation with CBP and other stakeholders and, if additional testing  was needed, that the Secretary of DHS appoint an independent group  within DHS to conduct objective, comprehensive, and transparent testing  that realistically demonstrates the capabilities and limitations of ASPs.
Combating Nuclear Smuggling: DHS’s Program to Procure and  Deploy Advanced Radiation Detection Portal Monitors Is Likely to  Exceed the Department’s Previous Cost Estimates. GAO-08-1108R.  Washington, D.C.: September 22, 2008.
Key findings. Our independent cost estimate suggested that from 2007  through 2017 the total cost of DNDO’s 2006 project execution plan (the  most recent official documentation of the program to equip U.S. ports of  entry with radiation detection equipment) would likely be about $3.1  billion but could range from $2.6 billion to $3.8 billion. In contrast, we  found that DNDO’s cost estimate of $2.1 billion was unreliable because it  omitted major project costs, such as elements of the ASPs’ life cycle, and  relied on a flawed methodology. DNDO officials told us that the agency  was no longer following the 2006 project execution plan and that the  scope of the agency’s ASP deployment strategy had been reduced to only  the standard cargo portal monitor. Our analysis of DNDO’s summary  information outlining its scaled-back plan indicated the total cost to  deploy standard cargo portals over the period 2008 through 2017 would be  about $2 billion but could range from $1.7 billion to $2.3 billion. Agency  officials acknowledged the program requirements that would have been  fulfilled by the discontinued ASPs remained valid, including screening rail  cars and airport cargo, but the agency had no plans for how such  screening would be accomplished.
Recommendations. We recommended that the Secretary of Homeland  Security direct the Director of DNDO to work with CBP to update the  projection execution plan to guide the entire radiation detection program  at U.S. ports of entry, revise the estimate of the program’s cost and ensure  that the estimate considers all of the costs associated with its project  execution plan, and communicate the revised estimate to Congress so that  it is fully apprised of the program’s scope and funding requirements.
Combating Nuclear Smuggling: DHS Needs to Consider the Full  Costs and Complete All Tests Prior to Making a Decision on  Whether to Purchase Advanced Portal Monitors. GAO-08-1178T.  Washington, D.C.: September 25, 2008.
Key findings. In preliminary observations of the 2008 round of ASP  testing, we found that DNDO had made progress in addressing a number  of problems we identified in previous rounds of ASP testing. However, the  DHS criteria for significant increase in operational effectiveness appeared  to set a low bar for improvement—for example, by requiring ASPs to  perform at least as well as current-generation equipment when nuclear  material is present in cargo but not specifying an actual improvement. In  addition, the ASP certification schedule did not allow for completion of  computer simulations that could provide useful data on ASP capabilities  prior to the Secretary’s decision on certification. Finally, we questioned  the replacement of current-generation equipment with ASPs until DNDO  demonstrates that any additional increase in security would be worth the  ASPs’ much higher cost.
Combating Nuclear Smuggling: DHS’s Phase 3 Test Report on  Advanced Portal Monitors Does Not Fully Disclose the  Limitations of the Test Results. GAO-08-979. Washington, D.C.:  September 30, 2008.
Key findings. DNDO’s report on the second group of ASP tests in 2007  (the Phase 3 tests) did not appropriately state test limitations. As a result,  the report did not accurately depict the results and could potentially be  misleading. The purpose of the Phase 3 tests was to conduct a limited  number of test runs in order to identify areas in which the ASP software  needed improvement. While aspects of the Phase 3 report addressed this  purpose, the preponderance of the report went beyond the test’s original  purpose and made comparisons of the performance of the ASPs with one  another or with currently deployed portal monitors. We found that it  would not be appropriate to use the Phase 3 test report in determining  whether the ASPs represent a significant improvement over currently  deployed radiation equipment because the limited number of test runs did  not support many of the comparisons of ASP performance made in the  report.
Recommendations. We recommended that the Secretary of DHS use the  results of the Phase 3 tests solely for the purposes for which they were  intended—to identify areas needing improvement—and not as a  justification for certifying whether the ASPs warrant full-scale production.  If the Secretary intends to consider the results of the Phase 3 tests in  making a certification decision regarding ASPs, we further recommended  that the Secretary direct the Director of DNDO to revise and clarify the  Phase 3 test report to more fully disclose and articulate the limitations  present in the Phase 3 tests and clearly state which insights from the  Phase 3 report are factored into any decision regarding the certification  that ASPs demonstrate a significant increase in operational effectiveness.  Finally, we recommended that the Secretary direct the Director of DNDO  to take steps to ensure that any limitations associated with the 2008 round  of testing are properly disclosed when the results are reported.
Combating Nuclear Smuggling: DHS Improved Testing of  Advanced Radiation Detection Portal Monitors, but Preliminary  Results Show Limits of the New Technology. GAO-09-655.  Washington, D.C.: May 21, 2009.
Key findings. We reported that the DHS criteria for a significant increase  in operational effectiveness require a large reduction in innocent alarms  but a marginal improvement in the detection of certain weapons-usable  nuclear materials. In addition, the criteria do not take the current- generation portal monitors’ full potential into account because DNDO has  not completed efforts to improve their performance. With regard to ASP  testing, we found that DHS increased the rigor in comparison with  previous tests, thereby adding credibility to the test results, but that  preliminary results were mixed. The results showed that the new portal  monitors performed better than current-generation portal monitors in  detection of certain nuclear materials concealed by light shielding  approximating the threat guidance for setting detection thresholds, but  differences in sensitivity were less notable when shielding was slightly  below or above that level. Testing also uncovered multiple problems in  ASPs meeting the requirements for successful integration into operations  at ports of entry. Finally, we found that DNDO did not plan to complete  computer simulations that could provide additional insight into ASP  capabilities and limitations prior to certification even though delays to  testing allowed more time to conduct the simulations.
Recommendations. We recommended that the Secretary of Homeland  Security direct the Director of DNDO to assess whether ASPs meet the  criteria for a significant increase in operational effectiveness based on a  valid comparison with current-generation portal monitors’ full  performance potential and revise the schedule for ASP testing and  certification to allow sufficient time for review and analysis of results from  the final phases of testing and completion of all tests, including computer  simulations. If ASPs are certified, we further recommended that the  Secretary of Homeland Security direct the Director of DNDO to develop an  initial deployment plan that allows CBP to uncover and resolve any  additional problems not identified through testing before proceeding to  full-scale deployment.
This is a work of the U.S. government and is not subject to copyright protection in the  United States. The published product may be reproduced and distributed in its entirety  without further permission from GAO. However, because this work may contain  copyrighted images or other material, permission from the copyright holder may be  necessary if you wish to reproduce this material separately.