Combating Nuclear Smuggling: DHS's Decision to Procure and Deploy the Next Generation of Radiation Detection Equipment Is Not Supported by Its Cost-Benefit Analysis
Released: 2007-03-14T00:00:00

Letter

Background

In general, DHS is responsible for providing radiation detection capabilities at U.S. ports of entry. Until April 2005, CBP managed this program. However, on April 15, 2005, the President directed the establishment of DNDO within DHS. DNDO’s duties include acquiring and supporting the deployment of radiation detection equipment, including portal monitors. CBP continues its traditional screening function at ports of entry to interdict dangerous nuclear and radiological materials through the use of radiation detection equipment. The SAFE Port Act of 2006 formally authorizes DNDO’s creation and operation. PNNL manages the deployment of radiation detection equipment for DHS.

DHS’s program to deploy radiation detection equipment at U.S. ports of entry has two goals. The first is to use this equipment to screen all cargo, vehicles, and individuals coming into the United States. The second is to screen this traffic without delaying its movement into the nation. DHS’s current plans call for completing the deployment of radiation portal monitors at U.S. ports of entry by September 2013. The current technology portal monitors, known as plastic scintillators or PVTs, cost about $55,000 per unit, while the advanced spectroscopic portal monitors, known as ASPs, will cost around $377,000 per unit.

In July 2006, DHS announced that it had awarded contracts to three ASP vendors to further develop and purchase $1.2 billion worth of ASPs over 5 years. Congress, however, provided that none of DNDO’s appropriated funds for systems acquisition could be obligated for full procurement of ASPs until the Secretary of DHS certifies through a report to the Committees on Appropriations for the Senate and House of Representatives that ASPs would result in a significant increase in operational effectiveness.

DNDO Ignored Its Own Performance Test Results and Instead Relied on the Potential Performance of New Portal Monitors and Unreliable Estimates of Current Equipment Performance

DHS is developing new portal monitors, known as ASPs that, in addition to detecting nuclear or radiological material, can also identify the type of material. In 2005, DNDO conducted side-by-side testing at the Nevada Test Site (NTS) on 10 ASP systems and 3 PVT systems developed by private sector companies, including the PVT systems currently deployed. DHS requested that the National Institutes of Standards and Technology (NIST) provide assistance by conducting an independent analysis of data acquired during the last phase of developmental testing of ASPs to help DHS determine the performance of ASP portal monitors being proposed by private sector companies. NIST compared the 10 ASP systems, and in June 2006 submitted a report to DHS on the results of that testing.

Performance tests of ASPs showed that they did not meet DNDO’s main performance assumption in the cost-benefit analysis of correctly identifying HEU 95 percent of the time it passes through portal monitors. The 95 percent performance assumption included ASPs’ ability to both detect bare, or unmasked, HEU in a container and HEU masked in a container with a more benign radiological material. Based on NIST’s assessment of the performance data, the ASP prototypes (manufactured by the three companies that won DNDO’s recent ASP procurement contract) tested at NTS identified bare HEU only 70 to 88 percent of time. Performance tests also showed that ASPs’ ability to identify masked HEU fell far short of meeting the 95 percent goal established for the cost-benefit analysis. According to DNDO, identifying masked HEU is the most difficult case to address. DOE officials told us that benign radiological materials could be used to hide the presence of HEU. NIST reported that the best ASP prototype DNDO tested in Nevada during 2005, and which won a procurement contract, was able to correctly identify masked HEU and depleted uranium (DU) only 53 percent of the time. Similarly, the ASP prototypes submitted by the other two companies that won DNDO ASP procurement contracts were able to identify masked HEU and DU only 45 percent and 17 percent of the time.

Despite these results, DNDO did not use the information from these tests in its cost-benefit analysis. Instead, DNDO officials told us that since the new portal monitors cannot meet the 95 percent detection goal, they relied on the assumption that they will reach that level of performance sometime in the future. DNDO officials asserted that the ASPs’ current performance levels would improve, but they provided no additional information as to how the 95 percent goal will be achieved or an estimate of when the technology would attain this level of performance.

Moreover, DNDO’s cost-benefit analysis only considered the benefits of ASPs’ ability to detect and identify HEU and did not consider the ASPs’ ability to detect and identify other nuclear and radiological materials. The ability of an ASP to identify specific nuclear or radiological materials depends on whether the ASP contains software that is specific to those materials. In our view, a complete cost-benefit analysis would include an assessment of ASPs’ ability to detect and identify a variety of nuclear and radiological material, not just HEU. By excluding radiological and nuclear materials other than HEU, DNDO’s analysis did not consider the number of secondary inspections that would be related to these materials and hence it likely underestimated the costs of ASP use. Further, DNDO told us the assumptions for the ability of ASP systems to detect and identify HEU 95 percent of the time came from the ASP performance specifications. However, we examined the performance specifications and found no specific requirement for detecting or identifying HEU with a 95 percent probability. While there is a requirement in the performance specification for the identification for HEU and other special nuclear material, we found no associated probability of success in performing this function.

DNDO’s cost-benefit analysis also may not accurately reflect the capabilities of PVTs to detect nuclear or radiological material. DNDO officials acknowledged that DNDO tested the performance of PVTs along with the ASPs in 2005, but did not use the results of these tests in its cost- benefit analysis. According to these officials and NIST staff who assisted in the testing, the PVT performance data were unusable because the PVTs’ background settings were not set properly. Consequently, DNDO officials told us the analysis was based on the performance of PVT monitors that PNNL tested during 2004 in New York. However, the results from these field tests are not definitive because, as PNNL noted in its final report, the tests did not use HEU and, therefore, the results from the tests did not indicate how well PVT portal monitors would be able to detect HEU in the field. Moreover, the PVT portal monitors that PNNL used had only one radiation detection panel as opposed to the four-panel PVT monitors that DHS currently deploys at U.S. ports of entry. An expert at a national laboratory told us that larger surface areas are more likely to detect radiological or nuclear material. DNDO also stated that due to the nature of the testing at the Nevada Test Site, the tests did not provide the data needed for the cost-benefit analysis. According to DNDO officials, this data would come from analysis of the performance of fielded systems at U.S. ports-of-entry where the probability to detect threats could be compared to false alarm rates.

DNDO’s director stated twice in testimony before the House Homeland Security Committee, Subcommittee on Prevention of Nuclear and Biological Attack—once on June 21, 2005, and again on May 26, 2006—that the ASP and PVT portals would be evaluated against one another in “extensive high-fidelity” tests. In our view, the results of such testing are critical to any decision by DNDO to employ new technology, such as ASPs, that might help protect the nation from nuclear smuggling. According to DNDO officials, new tests now underway at the DHS Nevada Test Site are comparing the performance of ASPs and PVTs side-by-side.

DNDO’s Cost-Benefit Analysis was Incomplete and Used Inflated Cost Estimates for PVT Equipment

DNDO officials told us they did not follow the DHS guidelines for performing cost-benefit analyses in conducting their own cost-benefit analysis. These guidelines stipulate, among other things, that such studies should address all of the major costs and benefits that could have a material effect on DHS programs. However, DNDO’s analysis omitted many factors that could affect the cost of new radiation portal monitors. For example, DNDO officials told us that there are over 12 different types of ASP monitors, yet they only estimated the cost of cargo portal monitors that would be used at land border crossings. In reality, DNDO and CBP plan to deploy different types of ASPs that would have varying costs, such as portal monitors at seaports which would have higher costs. Additionally, DNDO did not capture all the costs related to developing the different types of ASP monitors. In our view, developing realistic cost estimates should not be sacrificed in favor of simplicity.

DNDO also underestimated the life-cycle costs for operations and maintenance for both PVT and ASP equipment over time. DNDO’s analysis assumed a 5-year life-cycle for both PVT and ASP equipment. However, DNDO officials told us that a 10-year life-cycle was a more reasonable expectation for PVT and ASP equipment. The analysis assumes that the annual maintenance costs for PVT and ASP monitors will each equal 10 percent of their respective procurement costs. This means that maintenance costs for PVTs would be about $5,500 per year per unit based on a $55,000 purchase price and ASP maintenance costs would be about $38,000 per year per unit based on a $377,000 purchase price. Given the much higher maintenance costs for ASPs and the doubling of the life-cycle to 10 years, the long-term implications for these cost differences would be magnified. Consequently, DNDO’s analysis has not accounted for about $181 million in potential maintenance costs for ASPs alone.

Furthermore, DNDO did not assess the likelihood that radiation detection equipment would either misidentify or fail to detect nuclear and radiological materials. Rather, DNDO’s cost-benefit analysis focuses on the ability of ASPs to reduce false alarms—alarms that indicate nuclear or radiological material is in a container when, in fact, the material is actually non-threatening, such as ceramic tile. Reducing false alarms would reduce the number of secondary inspections of non-threatening nuclear and radiological materials and therefore the costs of those inspections. However, as required by DHS’s guide to performing cost-benefit analyses, DNDO’s analysis did not include all costs. In particular, the analysis did not include the potentially much bigger cost of “false negatives.” False negatives are instances in which a container possesses a threatening nuclear or radiological material, but the portal monitor either misidentifies the material as non-threatening or does not detect the material at all, thus allowing the material to enter the country. During the 2005 Nevada tests, the incidence of false negatives among the three vendors who received contracts ranged from about 45 percent to slightly more that 80 percent. This raises concerns because, as explained to us by a scientist at a national laboratory, at this level of performance, ASPs could conceivably misidentify HEU as a benign nuclear or radiological material or not detect it at all, particularly if the HEU is placed side by side with a non- threatening material such as kitty litter.

In recent testimonies before Congress, DNDO’s Director has cited the primary benefit of deploying ASP monitors as reducing unnecessary secondary inspections. DNDO’s cost-benefit analysis focused on measuring the benefits of ASP’s ability to reduce false alarms—alarms that indicate nuclear or radiological material is present when, in fact, it is not or such material is actually non-threatening. Reducing false alarms would reduce the number of secondary inspections of non-threatening nuclear and radiological materials and therefore the costs of those inspections. Even on this point, however, DNDO’s analysis was flawed. For example, it did not estimate the costs of secondary inspections as they are carried out today. DNDO’s analysis needs these baseline costs to compare alternatives because without them, it is impossible to determine whether the use of ASPs, as planned, will result in cost savings for secondary inspections. While we agree that facilitating commerce at U.S. ports of entry by reducing unnecessary secondary inspections is an important goal, we believe that the primary rationale for deploying portal monitors should be to protect the nation from nuclear and/or radiological attack. We found that DNDO’s analysis did not even attempt to measure the level—or value—of security afforded by portal monitors.

In addition, the ASP contract award has exceeded DNDO’s estimate for total cost by about $200 million. The cost-benefit analysis shows the total cost for deploying PVT and ASP monitors to be about $1 billion, which covers all costs related to acquisition, design, maintenance, and physical inspection over 5 years (for both PVT and ASP). However, in July 2006, DHS announced that it had awarded contracts to develop and purchase up to $1.2 billion worth of ASP portal monitors over 5 years. Furthermore, the cost-benefit analysis underestimates total deployment costs and does not account for other major costs, such as physical inspections of cargo containers, an additional procurement of 442 new PVT monitors, installation and integration, and maintenance.

Finally, DNDO overstated the purchase price of PVT monitors. Although DHS is currently paying an average of about $55,000 per monitor, DNDO’s cost-benefit analysis assumed the PVT would cost $130,959—the highest published manufacturers’ price for the government. According to DNDO’s Director, DNDO chose the highest published price because the current contract for portal monitors at that time was to expire, and the portal monitors will probably cost more in the future. However, the information DNDO provided us does not explain why DNDO assumes that the future price will be more than double what DHS was currently paying, as assumed in DNDO’s analysis. PNNL officials told us that the future price will almost certainly be lower than the price used in DNDO’s analysis.

In conclusion, DNDO’s approach to the cost-benefit analysis omitted many factors that could affect the cost of new radiation portal monitors. For these reasons, DHS’s cost-benefit analysis does not meet the intent of our March 2006 report recommendation to fully assess the costs and benefits before purchasing any new equipment.

Mr. Chairman, this concludes my prepared statement. I would be happy to respond to any questions that you or other members of the Subcommittee may have.

GAO Contact and Staff Acknowledgements

For further information about this testimony, please contact me, Gene Aloise, at (202) 512-3841 or at aloisee@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. John Delicath, Jim Shafer, and Eugene Wisnoski made key contributions to this statement.

This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.