Combating Nuclear Smuggling: DHS's Decision to Procure and Deploy the Next Generation of Radiation Detection Equipment Is Not Supported by Its Cost-Benefit Analysis
Published: 2007-03-14T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Department of Homeland Security (DHS) is responsible for addressing the threat of nuclear smuggling. Radiation detection portal monitors are key elements in our national defenses against such threats. DHS has sponsored R&D and testing activities to develop a "next generation" portal monitor, known as the advanced spectroscopic portal monitor. However, each one costs 6 times more than a current portal monitor. In March 2006, we recommended that DHS conduct a cost-benefit analysis to determine whether the new portal monitors are worth the additional cost. In June 2006, DHS issued its analysis. In October 2006, we issued our report that assessed the DHS study. GAO's statement, based on our October 2006 report, addresses whether DHS's cost-benefit analysis provides an adequate basis for its decision to purchase and deploy the next generation portal monitors.
section_title: What GAO Found
paragraphs: DHS's cost-benefit analysis does not provide a sound analytical basis for its decision to purchase and deploy the new portal monitor technology. Our review of the analysis determined that it had the following problems. Regarding the performance of the portal monitors: instead of using the results of its own portal monitor tests conducted in 2005, DHS assumed that the new portal monitor technology would correctly detect and identify highly enriched uranium (HEU) 95 percent of the time--a performance level that far exceeds the new technology's current capabilities. To determine the performance of the current generation of portal monitors in detecting HEU, DHS used data from limited tests carried out in 2004 that test officials concluded were unreliable for such purposes. DHS's analysis of the new technology portal monitors was incomplete because the analysis focused on identifying HEU, but did not fully consider how well the new portal monitor technology could correctly detect or identify other dangerous radiological or nuclear materials. Regarding cost estimates: in comparing the costs of the new and current technologies, the procurement costs of the current generation portal monitors were highly inflated because DHS assumed a unit cost of about $131,000. However, the contract price at the time of the analysis was about $55,000. According to officials who manage the contract, it was to expire and they expected portal monitor prices to increase, but not nearly as much as DHS assumed. DHS stated that the primary benefit of deploying the new portal monitors is reducing unnecessary secondary inspections. However, DHS's analysis does not fully estimate today's baseline costs for secondary inspections, which makes it impossible to determine whether the use of the new portal monitors as currently planned, will result in significant cost savings for these inspections. The new portal monitor contract price has exceeded DHS's total cost estimate by about $200 million. The cost-benefit analysis shows the total cost for deploying both current and new portal monitors to be about $1 billion. However, in July 2006, DHS announced that it had awarded contracts to develop and purchase up to $1.2 billion worth of the new portal monitors over 5 years. DHS's cost-benefit analysis omitted many factors that could affect the cost of new portal monitors, such as understating the life-cycle costs for operating and maintaining the equipment over time. For these reasons, DHS's cost-benefit analysis does not meet the intent of our March 2006 report recommendation to fully assess the costs and benefits before purchasing any new equipment.

section_title: Letter
paragraphs: I am pleased to appear here today to discuss our assessment of the  Department of Homeland Security’s (DHS) May 2006 cost-benefit analysis  used to support the purchase and deployment of next generation radiation  portal monitors. This is an important decision because, if procured, these  new portal monitors will be considerably more expensive than the portal  monitors in use today. Combating nuclear smuggling is one of our nation’s  key national security objectives and the deployment of radiation detection  equipment including portal monitors at U.S. ports of entry, including  border crossings and domestic seaports, is an integral part of this system.  DHS, through its Domestic Nuclear Detection Office (DNDO), is  responsible for acquiring and supporting the deployment of radiation  detection equipment, including portal monitors, within the United States.  The Pacific Northwest National Laboratory (PNNL), one of the  Department of Energy’s (DOE) national laboratories, is under contract to  manage the deployment of radiation detection equipment for DHS. U.S.  Customs and Border Protection (CBP) is responsible for screening cargo  as it enters the nation at our borders, which includes operating radiation  detection equipment to interdict dangerous nuclear and radiological  materials.
The radiation portal monitors in use today can detect the presence of  radiation, but they cannot distinguish between types of radiological  material. For example, they cannot tell the difference between harmless  products that emit radiation, such as ceramic tile, and dangerous  materials, such as highly enriched uranium (HEU), that could be used to  construct a nuclear weapon. Generally, CBP’s standard procedures require  incoming cargo to pass through one of these radiation portal monitors to  screen for the presence of radiation. This “primary inspection” serves to  alert CBP officers when a radioactive threat might be present. If there is a  potential threat, CBP procedures require a “secondary inspection.” To  confirm the presence of radiation, this secondary inspection usually  includes a second screening by a radiation portal monitor as well as a  screening by CBP officers using radioactive isotope identification devices.  These handheld devices are used to differentiate between types of  radioactive material to determine if the radiation being detected is  dangerous. Both the radiation portal monitors and handheld devices are  limited in their abilities to detect and identify nuclear material.
DHS would like to improve the capabilities of its portal monitors so that  they can perform the dual roles of detecting radiation and identifying  radiological materials. In this regard, DHS has sponsored research,  development, and testing activities designed to create the next generation  of portal monitors capable of performing both functions. These new,  advanced portals are known as advanced spectroscopic portals (ASPs). In  July 2006, DHS awarded contracts to three vendors for developing the  advanced spectroscopic portals’ capabilities. These awards were based  mainly on performance tests conducted at DHS’s Nevada Test Site in 2005,  where ten competing advanced spectroscopic vendors’ monitors were  evaluated. At the same time, three current technology portal monitors  were also tested.
To ensure that DHS’s substantial investment in radiation detection  technology yields the greatest possible level of detection capability at the  lowest possible cost, in a March 2006 GAO report, we recommended that  once the costs and capabilities of ASPs were well understood, and before  any of the new equipment was purchased for deployment, the Secretary of  DHS work with the Director of DNDO to analyze the costs and benefits of  deploying ASPs. Further, we recommended that this analysis focus on  determining whether any additional detection capability provided by the  ASPs was worth the considerable additional costs. In response to our  recommendation, DNDO issued its cost-benefit analysis in May 2006, and  an updated, revised version in June 2006. According to senior agency  officials, DNDO believes that the basic conclusions of its cost-benefit  analysis show that the new advanced spectroscopic portal monitors are a  sound investment for the U.S. government.
Mr. Chairman, my remarks today focus on the cost-benefit analysis DNDO  used in support of its decision to purchase new ASP portal monitors.  Specifically, I will discuss whether DNDO’s June 2006 cost-benefit analysis  provides an adequate basis for the substantial investment that acquiring  and deploying ASPs will necessitate.
My testimony is based upon our October 2006 report that evaluated  DNDO’s cost-benefit analysis. The work for our report was done in  accordance with generally accepted government auditing standards.
In summary, DNDO’s 2006 cost-benefit analysis does not provide a sound  analytical basis for its decision to purchase and deploy the new advanced  spectroscopic portal monitor technology.
Regarding the performance of the portal monitors:  Instead of using the results of its performance tests conducted in 2005,  DNDO’s analysis simply assumed that ASPs could detect highly enriched  uranium 95 percent of the time, a performance level far exceeding the  capabilities of the new technology’s current demonstrated capabilities.  The 2005 test results showed that the best of the three winning vendor  monitors could only identify masked HEU about 50 percent of the time.
To determine the current generation of portal monitors’ performance in  detecting HEU, DNDO used data from limited tests carried out in 2004 that  test officials concluded was unreliable for such purposes. In their written  report, test officials explicitly stated that the data were not indicative of  how well current technology portal monitors might perform in the field  particularly for detecting HEU.
DNDO’s analysis of the new technology portal monitors’ performance was  deficient because it focused on detecting and identifying HEU, but did not  fully consider other dangerous radiological or nuclear materials. DNDO  should have assessed the ASPs’ abilities to detect several realistic threat  materials.
In comparing the costs of the new and current technologies, the  procurement costs of the current generation portal monitors were highly  inflated because DNDO assumed a unit cost of about $131,000, while the  contract price at the time of the analysis was about $55,000. According to  officials who manage the contract, it was to expire and while they  expected portal monitor prices to increase, they did not believe the cost  would be as much as the price used in DNDO’s analysis.
DNDO stated that the primary benefit of deploying the new portal  monitors is reducing unnecessary secondary inspections. However,  DNDO’s analysis did not fully estimate today’s baseline costs for  secondary inspections, which makes it impossible to determine whether  the use of the new portal monitors as currently planned will result in  significant cost savings for these inspections.
The new portal monitor contract price has exceeded DNDO’s total cost  estimate by about $200 million. The cost-benefit analysis shows the total  cost for deploying both current and new portal monitors to be about $1  billion. However, in July 2006, DHS announced that it had awarded  contracts to develop and purchase up to $1.2 billion worth of the new  portal monitors over 5 years.
DNDO’s cost-benefit analysis omitted many factors that could affect the  cost of new portal monitors, such as understating the life-cycle costs for  operating and maintaining the equipment over time.
subsections: section_title: Background
paragraphs: In general, DHS is responsible for providing radiation detection  capabilities at U.S. ports of entry. Until April 2005, CBP managed this  program. However, on April 15, 2005, the President directed the  establishment of DNDO within DHS. DNDO’s duties include acquiring and  supporting the deployment of radiation detection equipment, including  portal monitors. CBP continues its traditional screening function at ports  of entry to interdict dangerous nuclear and radiological materials through  the use of radiation detection equipment. The SAFE Port Act of 2006  formally authorizes DNDO’s creation and operation. PNNL manages the  deployment of radiation detection equipment for DHS.
DHS’s program to deploy radiation detection equipment at U.S. ports of  entry has two goals. The first is to use this equipment to screen all cargo,  vehicles, and individuals coming into the United States. The second is to  screen this traffic without delaying its movement into the nation. DHS’s  current plans call for completing the deployment of radiation portal  monitors at U.S. ports of entry by September 2013. The current technology  portal monitors, known as plastic scintillators or PVTs, cost about $55,000  per unit, while the advanced spectroscopic portal monitors, known as  ASPs, will cost around $377,000 per unit.
In July 2006, DHS announced that it had awarded contracts to three ASP  vendors to further develop and purchase $1.2 billion worth of ASPs over 5  years. Congress, however, provided that none of DNDO’s appropriated  funds for systems acquisition could be obligated for full procurement of  ASPs until the Secretary of DHS certifies through a report to the  Committees on Appropriations for the Senate and House of  Representatives that ASPs would result in a significant increase in  operational effectiveness.
section_title: DNDO Ignored Its Own Performance Test Results and Instead Relied on the Potential Performance of New Portal Monitors and Unreliable Estimates of Current Equipment Performance
paragraphs: DHS is developing new portal monitors, known as ASPs that, in addition to  detecting nuclear or radiological material, can also identify the type of  material. In 2005, DNDO conducted side-by-side testing at the Nevada Test  Site (NTS) on 10 ASP systems and 3 PVT systems developed by private  sector companies, including the PVT systems currently deployed. DHS  requested that the National Institutes of Standards and Technology (NIST)  provide assistance by conducting an independent analysis of data acquired  during the last phase of developmental testing of ASPs to help DHS  determine the performance of ASP portal monitors being proposed by  private sector companies. NIST compared the 10 ASP systems, and in June  2006 submitted a report to DHS on the results of that testing.
Performance tests of ASPs showed that they did not meet DNDO’s main  performance assumption in the cost-benefit analysis of correctly  identifying HEU 95 percent of the time it passes through portal monitors.  The 95 percent performance assumption included ASPs’ ability to both  detect bare, or unmasked, HEU in a container and HEU masked in a  container with a more benign radiological material. Based on NIST’s  assessment of the performance data, the ASP prototypes (manufactured  by the three companies that won DNDO’s recent ASP procurement  contract) tested at NTS identified bare HEU only 70 to 88 percent of time.  Performance tests also showed that ASPs’ ability to identify masked HEU  fell far short of meeting the 95 percent goal established for the cost-benefit  analysis. According to DNDO, identifying masked HEU is the most difficult  case to address. DOE officials told us that benign radiological materials  could be used to hide the presence of HEU. NIST reported that the best  ASP prototype DNDO tested in Nevada during 2005, and which won a  procurement contract, was able to correctly identify masked HEU and  depleted uranium (DU) only 53 percent of the time. Similarly, the ASP  prototypes submitted by the other two companies that won DNDO ASP  procurement contracts were able to identify masked HEU and DU only 45  percent and 17 percent of the time.
Despite these results, DNDO did not use the information from these tests  in its cost-benefit analysis. Instead, DNDO officials told us that since the  new portal monitors cannot meet the 95 percent detection goal, they relied  on the assumption that they will reach that level of performance sometime  in the future. DNDO officials asserted that the ASPs’ current performance  levels would improve, but they provided no additional information as to  how the 95 percent goal will be achieved or an estimate of when the  technology would attain this level of performance.
Moreover, DNDO’s cost-benefit analysis only considered the benefits of  ASPs’ ability to detect and identify HEU and did not consider the ASPs’  ability to detect and identify other nuclear and radiological materials. The  ability of an ASP to identify specific nuclear or radiological materials  depends on whether the ASP contains software that is specific to those  materials. In our view, a complete cost-benefit analysis would include an  assessment of ASPs’ ability to detect and identify a variety of nuclear and  radiological material, not just HEU. By excluding radiological and nuclear  materials other than HEU, DNDO’s analysis did not consider the number  of secondary inspections that would be related to these materials and  hence it likely underestimated the costs of ASP use. Further, DNDO told  us the assumptions for the ability of ASP systems to detect and identify  HEU 95 percent of the time came from the ASP performance  specifications. However, we examined the performance specifications and  found no specific requirement for detecting or identifying HEU with a 95  percent probability. While there is a requirement in the performance  specification for the identification for HEU and other special nuclear  material, we found no associated probability of success in performing this  function.
DNDO’s cost-benefit analysis also may not accurately reflect the  capabilities of PVTs to detect nuclear or radiological material. DNDO  officials acknowledged that DNDO tested the performance of PVTs along  with the ASPs in 2005, but did not use the results of these tests in its cost- benefit analysis. According to these officials and NIST staff who assisted  in the testing, the PVT performance data were unusable because the PVTs’  background settings were not set properly. Consequently, DNDO officials  told us the analysis was based on the performance of PVT monitors that  PNNL tested during 2004 in New York. However, the results from these  field tests are not definitive because, as PNNL noted in its final report, the  tests did not use HEU and, therefore, the results from the tests did not  indicate how well PVT portal monitors would be able to detect HEU in the  field. Moreover, the PVT portal monitors that PNNL used had only one  radiation detection panel as opposed to the four-panel PVT monitors that  DHS currently deploys at U.S. ports of entry. An expert at a national  laboratory told us that larger surface areas are more likely to detect  radiological or nuclear material. DNDO also stated that due to the nature  of the testing at the Nevada Test Site, the tests did not provide the data  needed for the cost-benefit analysis. According to DNDO officials, this  data would come from analysis of the performance of fielded systems at  U.S. ports-of-entry where the probability to detect threats could be  compared to false alarm rates.
DNDO’s director stated twice in testimony before the House Homeland  Security Committee, Subcommittee on Prevention of Nuclear and  Biological Attack—once on June 21, 2005, and again on May 26, 2006—that  the ASP and PVT portals would be evaluated against one another in  “extensive high-fidelity” tests. In our view, the results of such testing are  critical to any decision by DNDO to employ new technology, such as ASPs,  that might help protect the nation from nuclear smuggling. According to  DNDO officials, new tests now underway at the DHS Nevada Test Site are  comparing the performance of ASPs and PVTs side-by-side.
section_title: DNDO’s Cost-Benefit Analysis was Incomplete and Used Inflated Cost Estimates for PVT Equipment
paragraphs: DNDO officials told us they did not follow the DHS guidelines for  performing cost-benefit analyses in conducting their own cost-benefit  analysis. These guidelines stipulate, among other things, that such studies  should address all of the major costs and benefits that could have a  material effect on DHS programs. However, DNDO’s analysis omitted  many factors that could affect the cost of new radiation portal monitors.  For example, DNDO officials told us that there are over 12 different types  of ASP monitors, yet they only estimated the cost of cargo portal monitors  that would be used at land border crossings. In reality, DNDO and CBP  plan to deploy different types of ASPs that would have varying costs, such  as portal monitors at seaports which would have higher costs.  Additionally, DNDO did not capture all the costs related to developing the  different types of ASP monitors. In our view, developing realistic cost  estimates should not be sacrificed in favor of simplicity.
DNDO also underestimated the life-cycle costs for operations and  maintenance for both PVT and ASP equipment over time. DNDO’s analysis  assumed a 5-year life-cycle for both PVT and ASP equipment. However,  DNDO officials told us that a 10-year life-cycle was a more reasonable  expectation for PVT and ASP equipment. The analysis assumes that the  annual maintenance costs for PVT and ASP monitors will each equal 10  percent of their respective procurement costs. This means that  maintenance costs for PVTs would be about $5,500 per year per unit based  on a $55,000 purchase price and ASP maintenance costs would be about  $38,000 per year per unit based on a $377,000 purchase price. Given the  much higher maintenance costs for ASPs and the doubling of the life-cycle  to 10 years, the long-term implications for these cost differences would be  magnified. Consequently, DNDO’s analysis has not accounted for about  $181 million in potential maintenance costs for ASPs alone.
Furthermore, DNDO did not assess the likelihood that radiation detection  equipment would either misidentify or fail to detect nuclear and  radiological materials. Rather, DNDO’s cost-benefit analysis focuses on the  ability of ASPs to reduce false alarms—alarms that indicate nuclear or  radiological material is in a container when, in fact, the material is actually  non-threatening, such as ceramic tile. Reducing false alarms would reduce  the number of secondary inspections of non-threatening nuclear and  radiological materials and therefore the costs of those inspections.  However, as required by DHS’s guide to performing cost-benefit analyses,  DNDO’s analysis did not include all costs. In particular, the analysis did  not include the potentially much bigger cost of “false negatives.” False  negatives are instances in which a container possesses a threatening  nuclear or radiological material, but the portal monitor either misidentifies  the material as non-threatening or does not detect the material at all, thus  allowing the material to enter the country. During the 2005 Nevada tests,  the incidence of false negatives among the three vendors who received  contracts ranged from about 45 percent to slightly more that 80 percent.  This raises concerns because, as explained to us by a scientist at a national  laboratory, at this level of performance, ASPs could conceivably  misidentify HEU as a benign nuclear or radiological material or not detect  it at all, particularly if the HEU is placed side by side with a non- threatening material such as kitty litter.
In recent testimonies before Congress, DNDO’s Director has cited the  primary benefit of deploying ASP monitors as reducing unnecessary  secondary inspections. DNDO’s cost-benefit analysis focused on  measuring the benefits of ASP’s ability to reduce false alarms—alarms that  indicate nuclear or radiological material is present when, in fact, it is not  or such material is actually non-threatening. Reducing false alarms would  reduce the number of secondary inspections of non-threatening nuclear  and radiological materials and therefore the costs of those inspections.  Even on this point, however, DNDO’s analysis was flawed. For example, it  did not estimate the costs of secondary inspections as they are carried out  today. DNDO’s analysis needs these baseline costs to compare alternatives  because without them, it is impossible to determine whether the use of  ASPs, as planned, will result in cost savings for secondary inspections.  While we agree that facilitating commerce at U.S. ports of entry by  reducing unnecessary secondary inspections is an important goal, we  believe that the primary rationale for deploying portal monitors should be  to protect the nation from nuclear and/or radiological attack. We found  that DNDO’s analysis did not even attempt to measure the level—or  value—of security afforded by portal monitors.
In addition, the ASP contract award has exceeded DNDO’s estimate for  total cost by about $200 million. The cost-benefit analysis shows the total  cost for deploying PVT and ASP monitors to be about $1 billion, which  covers all costs related to acquisition, design, maintenance, and physical  inspection over 5 years (for both PVT and ASP). However, in July 2006,  DHS announced that it had awarded contracts to develop and purchase up  to $1.2 billion worth of ASP portal monitors over 5 years. Furthermore, the  cost-benefit analysis underestimates total deployment costs and does not  account for other major costs, such as physical inspections of cargo  containers, an additional procurement of 442 new PVT monitors,  installation and integration, and maintenance.
Finally, DNDO overstated the purchase price of PVT monitors. Although  DHS is currently paying an average of about $55,000 per monitor, DNDO’s  cost-benefit analysis assumed the PVT would cost $130,959—the highest  published manufacturers’ price for the government. According to DNDO’s  Director, DNDO chose the highest published price because the current  contract for portal monitors at that time was to expire, and the portal  monitors will probably cost more in the future. However, the information  DNDO provided us does not explain why DNDO assumes that the future  price will be more than double what DHS was currently paying, as  assumed in DNDO’s analysis. PNNL officials told us that the future price  will almost certainly be lower than the price used in DNDO’s analysis.
In conclusion, DNDO’s approach to the cost-benefit analysis omitted many  factors that could affect the cost of new radiation portal monitors. For  these reasons, DHS’s cost-benefit analysis does not meet the intent of our  March 2006 report recommendation to fully assess the costs and benefits  before purchasing any new equipment.
Mr. Chairman, this concludes my prepared statement. I would be happy to  respond to any questions that you or other members of the Subcommittee  may have.
section_title: GAO Contact and Staff Acknowledgements
paragraphs: For further information about this testimony, please contact me, Gene  Aloise, at (202) 512-3841 or at aloisee@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on the  last page of this statement. John Delicath, Jim Shafer, and Eugene  Wisnoski made key contributions to this statement.
This is a work of the U.S. government and is not subject to copyright protection in the  United States. It may be reproduced and distributed in its entirety without further  permission from GAO. However, because this work may contain copyrighted images or  other material, permission from the copyright holder may be necessary if you wish to  reproduce this material separately.