Combating Nuclear Smuggling: DHS Needs to Consider the Full Costs and Complete All Tests Prior to Making a Decision on Whether to Purchase Advanced Portal Monitors
Released: 2008-09-25T00:00:00

Letter

DNDO’s Program to Deploy Radiation Detection Portal Monitors at U.S. Ports of Entry Is Likely to Cost About $3 Billion

As we discuss in our report, our independent cost estimate suggested the total cost of DNDO’s program to equip U.S. ports of entry with radiation detection equipment will likely be about $3.1 billion, but could range between $2.6 billion and $3.8 billion. We based our estimate on the anticipated costs of DNDO implementing its 2006 project execution plan, the most recent official documentation of the radiation portal monitor project’s objectives, scope, schedule, costs, and funding requirements. According to this plan, DNDO plans to buy and deploy several types of ASPs, including those designed to screen rail cars, airport cargo, and seaport cargo, as well as mobile ASPs that provide greater flexibility in screening commerce. The plan also provides for the deployment of several types of PVTs. Clearly, the numbers and types of portal monitors deployed will significantly affect the total cost of the radiation portal monitor project. In all, the project execution plan calls for the purchase of 2,754 portal monitors, with a total of 2,582 scheduled for deployment— approximately 1,034 ASPs and 1,548 PVTs—and 172 held in excess at the project’s completion.

In developing our estimate, we categorized radiation portal monitor project costs according to program phases—design and development, procurement, deployment, maintenance, and operational sustainment. We did not incorporate operational costs, in particular the cost of CBP officers operating the radiation detection equipment. DNDO and CBP believe that deploying ASPs will reduce the use of CBP staff resources for radiation detection tasks, but currently there are no usable estimates of how CBP’s staffing would change with the deployment of ASPs. Our analysis included a period of 11 years: actual life-cycle expenses from fiscal year 2007, and estimated life cycle costs from fiscal year 2008 through 2017.

DNDO’s estimate of $2.1 billion to deploy radiation detection equipment (submitted as part of its budget request to OMB in March 2008) is unreliable because it omits major project costs and relies on flawed methodology. As a result, DNDO’s cost estimates and budget requests for the radiation portal monitor project are too low, which could lead to significant cost overruns later in the project. DNDO’s estimate contains the following major deficiencies: It does not appear to include the costs of all variations of ASPs contained in DNDO’s project execution plan. DNDO’s current baseline considers only the standard ASP cargo portal and ignores the costs of other types of ASPs, such as those designed to screen rail cars, airport and seaport cargo, and mobile detectors. According to senior DNDO officials, the current approved project execution plan no longer reflects the agency’s procurement and deployment plans for ASPs. In acknowledging that the project execution plan should be revised and updated, these officials told us that DNDO currently plans to field only the standard cargo ASP portal, mainly at high-volume ports of entry. (We used the 2006 project execution plan because it is the agency’s only official plan for ASP deployment.)

DNDO’s estimate considers only 8 years rather than 10 years, the operational life expected by the manufacturer of sodium iodide crystals, a key ASP component. DHS cost guidance maintains that a project’s life cycle can be estimated over that period of time during which equipment will remain available before it is exhausted, that is, decayed or deteriorated. DNDO officials agreed that a 10-year life cycle cost estimate would have been more appropriate and said that they would have used a 10-year estimate had they not been constrained by OMB budget submission software, which limits the number of years of costs that can be included.

DNDO’s cost estimate does not include all of the elements of the ASPs’ life cycle, such as maintenance or operational sustainment. These costs are approximately $999.2 million and $364.9 million, respectively, under our cost estimate.

DNDO did not have detailed documentation of the costs to support its estimates. According to OMB, DHS, and GAO guidelines, such documentation is necessary to establish the basis of the estimates and to provide assurances that the estimates are credible.

Assuming DNDO attempts to implement its authorized 2006 project execution plan rather than its scaled-back plan, our report estimated a $753 million budget shortfall for the radiation portal monitor project for fiscal years 2008 through 2012 (the years included in both our analysis and the DNDO analysis). Additionally, we estimated that DNDO will require another $833 million from fiscal year 2013 through fiscal year 2017 to complete the entire life cycle of the program.

According to DNDO officials, our cost estimate will result in estimates higher than the program’s current true cost because it is based on the outdated project execution plan. However, in the absence of more recent documentation, we believe our independent cost estimate must be based on the agency’s most recent approved plan (the 2006 project execution plan). Furthermore, several official DNDO documents specify that multiple versions of ASPs will be deployed. For example, the agency’s February 2006 expenditure plan submitted to the Congress foresees “several variants” of ASP systems being deployed, with standard cargo, rail, and automobile versions mentioned specifically. And DNDO’s submissions to OMB for budget years 2008 and 2009 describe a program that includes land crossings, seaports, rail lines, airports, and other ports of entry. Finally, agency officials acknowledged the program requirements that would have been fulfilled by the discontinued ASPs remain valid, including screening rail cars, airport cargo, and cargo at seaport terminals, but the agency has no current plans for how such screening will be accomplished. These officials told us the technology to accomplish these requirements likely will not be ASP monitors and could be a totally new technology. We believe a comprehensive estimate of the cost to provide radiation detection equipment for U.S. ports of entry should account for meeting these objectives, even if DNDO decides that ASP technology is not suited to them.

DNDO agreed in June 2008 to update its project execution plan so that we could better estimate the costs of the agency’s current plans. DNDO also agreed to provide an updated estimate of the standard portal’s life cycle costs and to meet with us to reconcile our two estimates. However, DNDO did not provide a revised project execution plan or cost estimate and instead provided in July 2008 a one-page spreadsheet of summary information outlining its plans to buy and deploy portal monitors for the 7- year period 2008 through 2014. DNDO’s summary data indicate that during this time period the agency plans to deploy 717 ASPs and 1,005 PVTs. The summary data do not provide the breadth and depth of information needed to generate detailed and fully documented cost estimates. Furthermore, according to subsequent discussions with a senior DNDO official, if ongoing tests indicate the ASPs’ performance warrants it, the agency may speed its deployment of ASPs over the next few months. In our view, the frequent changes in deployment plans and the lack of available cost documentation raise concerns about the overall management of the radiation portal monitor project and whether it is guided by a sound and stable strategy. Nonetheless, we used the agency’s summary data to perform a more limited cost estimate for only the standard cargo portal. We found that from 2008 to 2017 the total program cost for buying and deploying standard cargo portals would likely be about $2 billion, but could range from about $1.7 billion to $2.3 billion.

Our report recommended that the Secretary of Homeland Security direct the Director of DNDO to (1) work with the Commissioner of CBP to update the project execution plan to guide the entire radiation detection program at U.S. ports of entry, (2) revise DNDO’s estimate of the program’s cost and ensure that the estimate considers all the costs associated with its project execution plan, and (3) communicate this revised estimate to the Congress so that it is fully apprised of the program’s scope and funding requirements. DHS agreed with our recommendations in its comments responding to our draft report.

Preliminary Observations of Ongoing Testing of ASPs

We are currently reviewing the ongoing 2008 ASP testing program and process leading to a decision by the Secretary of Homeland Security on certification of ASPs, and we plan to issue a final report in November 2008, prior to the Secretary’s decision on certification. Our work to date shows that DNDO has made progress in addressing a number of problems we identified in previous rounds of ASP testing. A particular area of improvement is in the performance testing at the Nevada Test Site, where DNDO conducts trials of the ability of ASPs to detect and identify radioactive materials, particularly those that could be used in a nuclear weapon. In 2007, we reported that DNDO had allowed ASP system contractors to adjust their systems after preliminary runs using the same radiological materials that would be used in the formal tests, potentially biasing the test results. In contrast, in our more recent work, we found that the plan for the 2008 round of performance testing stipulated that there would be no system contractor involvement in test execution, and no ASP system contractors were present at the test location on the day we observed performance testing. Furthermore, DNDO officials told us, and we observed, that they did not conduct preliminary runs with threat objects that were used in the formal tests. A further area of improvement in the performance testing concerns the use of handheld RIIDs. Specifically, DNDO used actual CBP officers who adhered to nearly all of the standard operating procedures in testing the RIIDs. This improvement addresses our concern from last year that DNDO did not objectively test the performance of the RIIDs because it did not use a critical CBP standard operating procedure that is fundamental to the equipment’s performance in the field.

Despite these improvements, we have identified potential areas of concern based on our ongoing review. Our concerns fall into three areas: DHS criteria for “significant increase in operational effectiveness.”

DNDO, CBP and the DHS management directorate agreed on six criteria for the degree of improvement ASPs must provide over current generation equipment—four criteria for primary screening and two for secondary screening. These criteria appear to set a low bar for improvement. Most notably, the criteria for primary screening require ASPs to perform at least as well as current generation equipment when special nuclear material or medical or industrial isotopes are present in cargo, but they do not specify an actual improvement. Similarly, one of the criteria for secondary screening requires ASPs to reduce the average time to release conveyances but does not indicate a specific level of reduction. As such, the criteria leave open the possibility of a Secretarial decision in favor of certification even if ASPs do not provide a significantly higher probability of detection compared with current generation PVTs when deployed for primary screening and only a small reduction in the time required for secondary screening. We recently requested additional information from DNDO about the rationale behind these criteria, particularly in light of seemingly stricter criteria found in other documents such as the ASP performance specification, which requires an 80 percent probability of detection, at a 95 percent confidence, of government-designated representative threat objects. We are awaiting DNDO’s response.

The extent to which the 2008 testing program provides a sound basis for determining ASPs’ operational effectiveness. While the performance testing at Nevada Test Site could provide a useful comparison of the performance of the ASPs and current generation equipment for both primary and secondary screening given the improvements described above, DNDO does not plan to complete “injection studies” that could provide additional data on ASP performance prior to ASP certification. (Injection studies are a type of computer simulation for testing the response of the ASP threat identification algorithms to data on threat objects combined with stream-of-commerce data from a port of entry.) Rather, DNDO officials told us that testing using actual threat objects is sufficient for certification. While we recognize the limitations of injection studies such as the inability to exactly model threat objects hidden in commerce, reliance on performance testing but not injection studies for the Secretary’s decision on certification will result in less data with which to evaluate the capabilities of ASPs, particularly for primary screening. According to officials from Pacific Northwest National Laboratory, which has a lead role in injection studies, computer simulations allow for an increased statistical confidence in the measured probability of detection compared with testing at Nevada Test Site. The ASP test and evaluation master plan similarly emphasizes the importance of injection studies for a complete test of radiation detection system performance, and a February 2008 DNDO presentation to us includes among its fiscal year 2008 objectives the use of preliminary results from injection studies to evaluate performance of ASPs for primary screening. However, injection studies have taken longer than projected due to a number of issues such as deficiencies in the stream-of-commerce data.

The schedule leading up to a decision on ASP certification by the Secretary of Homeland Security. The test schedule leading up to certification is highly compressed and is running at least 8 weeks behind the schedule DNDO provided to us in May 2008, leaving limited time for analysis and review of test results. Key phases of the test schedule include system qualification, followed by testing at Pacific Northwest National Laboratory to demonstrate that ASPs are ready to be integrated into the interdiction systems at U.S. ports of entry and concurrent testing at Nevada Test Site to compare the ASP performance in detecting and identifying threats with that of current generation equipment. According to DNDO, the systems qualification and performance tests have been completed, but only after being delayed due to problems with system qualification. The final two phases, field validation at four ports of entry operated by CBP and operational testing at one port of entry, have not yet started. Field validation was to have started yesterday—September 24— but as with integration and performance testing this phase has also been delayed, and CBP has not yet indicated when it will begin. Even before this latest delay, the time between the scheduled end of operational testing in early November and a Secretarial decision on certification on November 28 allowed less than a month for review and analysis of results. Furthermore, DHS officials have indicated that the Secretary may make a decision on certification on the basis of “quick look” (preliminary) rather than final reports if the quick look reports are favorable to ASPs. The limited time between completion of testing and a decision on certification, combined with the potential reliance on quick look reports, increases the risk of a decision in favor of certification being called into question later by a more thorough analysis and review of results.

Assuming that DHS addresses these concerns—for example, by clarifying the criteria for significant increase in operational effectiveness, performing the injection studies, and delaying a Secretarial decision on certification—the 2008 round of testing could provide an objective basis for comparing ASPs with current generation equipment. However, consistent with our March 2006 recommendation that DHS analyze the benefits and costs of deploying ASPs to determine whether any additional detection capability provided by ASPs is worth the cost, we would still question the replacement of current generation equipment with ASPs, particularly considering the gaps identified by DNDO in the global nuclear detection architecture—essentially, an integrated system of radiation detection equipment and interdiction activities to combat nuclear smuggling in foreign countries, at the U.S. border, and inside the United States. As we testified before this committee earlier this year, such gaps include land border crossings into the United States between formal points of entry, small maritime vessels, and international general aviation. An updated cost-benefit analysis, based on the latest information on ASP costs and test performance, could help policymakers address any trade- offs in addressing these gaps versus replacing current generation radiation detection equipment already deployed at ports of entry. The current generation equipment has known limitations, particularly with regard to the limited ability to detect certain nuclear materials and the inability to distinguish between types of radiological material. However, the existing equipment provides at least some detection capability and may act as a deterrent, and there is no evidence we are aware of that it is impeding the flow of commerce.

Mr. Chairman, this concludes my prepared statement. I would be pleased to respond to any questions that you or other members of the Committee may have.

GAO Contact and Staff Acknowledgments

For further information on this testimony, please contact Gene Aloise at (202) 512-3841 or by e-mail at aloisee@gao.gov. Individuals making key contributions to this testimony include Joseph Cook, Jennifer Echard, Brian Octeau, Alison O’Neill, Tim Persons (GAO’s Chief Scientist), Karen Richey, Benjamin Shouse, Kevin Tarmann, Eugene Wisnoski, and Ned Woodward.

This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.