Combating Nuclear Smuggling: DHS Needs to Consider the Full Costs and Complete All Tests Prior to Making a Decision on Whether to Purchase Advanced Portal Monitors
Published: 2008-09-25T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Department of Homeland Security's (DHS) Domestic Nuclear Detection Office (DNDO) is responsible for addressing the threat of nuclear smuggling. Radiation detection portal monitors are key elements in our national defenses against such threats. DHS has sponsored testing to develop new monitors, known as advanced spectroscopic portal (ASP) monitors, to replace radiation detection equipment currently being used at ports of entry. ASPs may offer improvements over current generation portal monitors, particularly the potential to identify as well as detect radioactive material and thereby minimize both missed threats and false alarms. However, ASPs cost significantly more than current generation portal monitors, and testing of ASPs' capabilities needs to be more objective and rigorous. Due to concerns about ASPs' cost and performance, Congress has required that the Secretary of DHS certify that ASPs will provide a significant increase in operational effectiveness before obligating funds for full-scale ASP procurement. DHS is currently testing ASPs and anticipates a decision on certification in November 2008. This testimony addresses (1) the highlights of GAO's September 2008 report on the life cycle cost estimate to deploy ASPs (GAO-08-1108R), and (2) preliminary observations from ongoing work reviewing the current program of ASP testing.
section_title: What GAO Found
paragraphs: GAO's independent cost estimate suggested that from 2007 through 2017 the cost of DNDO's program to equip U.S. ports of entry with radiation detection equipment will likely be about $3.1 billion, but could range from $2.6 billion to $3.8 billion. GAO's estimate was based on the anticipated costs of DNDO implementing its 2006 project execution plan, the most recent official documentation of the program. DNDO's cost estimate of $2.1 billion to implement its project execution plan is unreliable because it omits major project costs, such as maintenance, and relies on a flawed methodology. For example, although the normal life expectancy of the standard cargo ASP is about 10 years, DNDO's estimate considers only 8 years. According to DNDO officials, the agency is now following a scaled-back ASP deployment strategy rather than the 2006 project execution plan, and a senior DNDO official told GAO the ASP deployment strategy could change dramatically depending on the outcome of ongoing testing. GAO's analysis indicated the cost to implement the scaled-back plans over the period 2008 through 2017 will be about $2 billion, but could range from $1.7 billion to $2.3 billion. However, frequent changes in DNDO's deployment strategy make it difficult to assess ASP program costs. GAO's recent report recommended that the Secretary of Homeland Security direct DNDO to update the project execution plan, revise its cost estimate, and communicate the revised estimate to the Congress so that it is fully apprised of the program's scope and funding requirements. DHS agreed with the recommendations. DNDO has made progress in addressing a number of problems GAO identified in previous rounds of ASP testing. However, GAO's ongoing review of the 2008 ASP testing program identified several potential areas of concern. First, the DHS criteria for "significant increase in operational effectiveness" appear to set a low bar for improvement--for example, by requiring ASPs to perform at least as well as current generation equipment when nuclear material is present in cargo but not specifying an actual improvement. GAO recently requested additional information from DNDO about the rationale behind these criteria, particularly in light of seemingly stricter criteria found in other documents. Second, the ASP certification schedule does not allow for completion of computer simulations that could provide additional data on ASP performance. While these computer simulations may have limitations, they also could provide useful data on ASP capabilities prior to the Secretary's decision on certification. Finally, the test schedule is highly compressed and is running at least 8 weeks behind, leaving limited time for analysis and review of test results. Assuming that DHS addresses these concerns, the 2008 round of testing could provide an objective basis for comparing ASPs with current generation equipment. However, GAO recommended in March 2006 that DHS analyze the benefits and costs of deploying ASPs to determine whether any additional detection capability provided by ASPs is worth the cost, and would still question the replacement of current generation equipment with ASPs until DNDO demonstrates that any additional increase in security would be worth the ASPs' much higher cost.

section_title: Letter
paragraphs: I am pleased to be here today to discuss our work on the plans of the  Department of Homeland Security’s (DHS) Domestic Nuclear Detection  Office (DNDO) to develop and deploy advanced spectroscopic portal  (ASP) radiation detection monitors to the nation’s ports of entry.  Preventing radioactive material from being smuggled into the United  States—perhaps to be used by terrorists in a nuclear weapon or in a  radiological dispersal device (a “dirty bomb”)—is a key national security  objective. Today I will discuss our September report, which focuses on  developing an independent life cycle cost estimate for replacing radiation  detection equipment already deployed at U.S. ports of entry with ASPs,  and our ongoing work reviewing DNDO’s current program of ASP testing  activities, which started in April 2008 and are scheduled to be completed  in November 2008 under DNDO’s latest plan. These tests are critically  important because they will serve as the primary support for a  congressionally mandated DHS Secretarial certification of the  effectiveness of ASPs, also scheduled for November 2008.
Radiation portal monitors—large stationary detectors through which  cargo containers and trucks pass as they enter the United States—are an  important component of the radiation detection system. The portal  monitors in use today, known as polyvinyl toluene (PVT) monitors, are  capable of detecting radiation but not identifying specific radioactive  materials. To address this limitation, U.S. Customs and Border Protection  (CBP) relies on handheld detection technologies, particularly radioactive  isotope identification devices (RIID), to help CBP officers distinguish  between dangerous and benign materials. CBP’s standard operating  procedures for use of radiation detection equipment include conducting  primary inspections with PVTs to detect the presence of radioactivity, and  secondary inspections with PVTs and RIIDs to confirm and identify the  source and determine whether it constitutes a threat. Any vehicle  triggering a PVT alarm is referred to a secondary screening area, where it  is sent through a second PVT to confirm the original alarm. Whether the  second PVT confirms the alarm or not, the vehicle, driver, and any  passengers or cargo are scanned by a CBP officer with a RIID, which can  detect radiation and also identify many of the most commonly used  radioactive materials by name. All PVT alarms must be resolved—that is,  CBP officers must investigate each alarm until they are convinced that the  vehicle, occupants, and any cargo pose no threat and, if radioactive  materials are found, that the vehicle occupants appear to have a legitimate  reason to possess and transport them—before the vehicle, driver, and any  passengers can be allowed to enter the United States.
In contrast to PVTs, ASPs offer the potential to detect radiation and  identify the source, reducing the need for secondary screenings of cargo  containing benign radioactive materials. According to DNDO, the inability  of the current generation PVT systems to identify the type of material  causing an alarm results in the need to balance system sensitivity with the  false and innocent alarm burden to the flow of commerce. DNDO also  believes that CBP may use an inordinate amount of inspection resources  for radiation detection at the expense of other missions, such as drug  interdiction. To address these limitations, DNDO is sponsoring the  development of ASPs with the expectation that they will minimize both  missed threats and false alarms when deployed for primary screening and  improve upon the RIID’s identification performance and time required to  correctly resolve primary alarms when deployed for secondary screening.
ASPs cost significantly more than PVTs. DNDO’s latest unit cost estimate  (including deployment costs) is about $800,000 for the standard cargo  version of the ASP and about $425,000 for the PVT standard cargo portal.  Due to concerns about the performance and higher cost of ASPs relative  to PVTs, the Congress has required that the Secretary of Homeland  Security submit a report certifying that ASPs will provide a significant  increase in operational effectiveness. Prior to primary and secondary  deployment of the ASPs, the Secretary must submit separate and distinct  certifications that address the unique requirements for operational  effectiveness of each type of deployment.
Our prior work on DNDO’s efforts to develop and procure ASPs raised a  number of serious concerns. In March 2006, we reported on, among other  things, DNDO’s efforts to develop ASPs; we recommended that DHS  analyze the benefits and costs of deploying ASPs to determine whether  any additional detection capability provided by ASPs is worth the  additional cost. In October 2006, we concluded that DNDO’s cost-benefit  analysis did not provide a sound basis for its decision to purchase and  deploy ASPs at an estimated cost at that time of $1.2 billion—for example,  because DNDO relied on assumptions of the anticipated performance level  of ASPs instead of actual test data. We also reported that DNDO did not  assess the likelihood that ASPs would either misidentify or fail to detect  nuclear or radiological material; rather, it focused its analysis on reducing  the time necessary to screen traffic at ports of entry and the impact of any  delays on commerce. We recommended, in part, that DNDO conduct  further testing before spending additional funds to purchase ASPs.  Similarly, in September 2007, we testified that DNDO’s testing of ASPs at  the Department of Energy’s (DOE) Nevada Test Site did not represent an  objective or rigorous assessment because DNDO used biased test methods  that enhanced the apparent performance of the ASPs and did not test the  limitations of the ASPs’ detection capabilities. We recommended that DHS  delay ASP certification and full-scale production decisions until all  relevant tests and studies have been completed, limitations to these tests  and studies have been identified and addressed, and results of the tests  and analyses have been reported to the appropriate congressional  committees. For reasons I am about to discuss, these recommendations  are as important today as when we made them last year.
My testimony today addresses (1) the highlights of our September 2008  report on our independent life cycle cost estimate for deploying ASPs at  ports of entry, and (2) preliminary observations from our ongoing work  reviewing the current program of ASP testing. For our September 2008  report, we contracted with a company with expertise in estimating the life  cycle costs of major federal acquisitions. Together we developed an  independent cost estimate of DHS’s 2006 strategy to deploy radiation  detection portal monitors, covering fiscal years 2007 through 2017, to  ensure that the Congress has authoritative information on all the life-cycle  costs associated with a full-scale acquisition of radiation portal monitors.  To review ASP testing, we analyzed ASP test plans and interviewed senior  DNDO officials responsible for managing the ASP program, and we  observed testing conducted at the DOE’s Nevada Test Site and Pacific  Northwest National Laboratory. We also interviewed DOE, national  laboratory, National Institute of Standards and Technology, and DHS  officials. We conducted this work, including our work to date on ASP  testing, in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
subsections: section_title: DNDO’s Program to Deploy Radiation Detection Portal Monitors at U.S. Ports of Entry Is Likely to Cost About $3 Billion
paragraphs: As we discuss in our report, our independent cost estimate suggested the  total cost of DNDO’s program to equip U.S. ports of entry with radiation  detection equipment will likely be about $3.1 billion, but could range  between $2.6 billion and $3.8 billion. We based our estimate on the  anticipated costs of DNDO implementing its 2006 project execution plan,  the most recent official documentation of the radiation portal monitor  project’s objectives, scope, schedule, costs, and funding requirements.  According to this plan, DNDO plans to buy and deploy several types of  ASPs, including those designed to screen rail cars, airport cargo, and  seaport cargo, as well as mobile ASPs that provide greater flexibility in  screening commerce. The plan also provides for the deployment of several  types of PVTs. Clearly, the numbers and types of portal monitors deployed  will significantly affect the total cost of the radiation portal monitor  project. In all, the project execution plan calls for the purchase of 2,754  portal monitors, with a total of 2,582 scheduled for deployment— approximately 1,034 ASPs and 1,548 PVTs—and 172 held in excess at the  project’s completion.
In developing our estimate, we categorized radiation portal monitor  project costs according to program phases—design and development,  procurement, deployment, maintenance, and operational sustainment. We  did not incorporate operational costs, in particular the cost of CBP  officers operating the radiation detection equipment. DNDO and CBP  believe that deploying ASPs will reduce the use of CBP staff resources for  radiation detection tasks, but currently there are no usable estimates of  how CBP’s staffing would change with the deployment of ASPs. Our  analysis included a period of 11 years: actual life-cycle expenses from  fiscal year 2007, and estimated life cycle costs from fiscal year 2008  through 2017.
DNDO’s estimate of $2.1 billion to deploy radiation detection equipment  (submitted as part of its budget request to OMB in March 2008) is  unreliable because it omits major project costs and relies on flawed  methodology. As a result, DNDO’s cost estimates and budget requests for  the radiation portal monitor project are too low, which could lead to  significant cost overruns later in the project. DNDO’s estimate contains  the following major deficiencies:  It does not appear to include the costs of all variations of ASPs contained  in DNDO’s project execution plan. DNDO’s current baseline considers only  the standard ASP cargo portal and ignores the costs of other types of  ASPs, such as those designed to screen rail cars, airport and seaport cargo,  and mobile detectors. According to senior DNDO officials, the current  approved project execution plan no longer reflects the agency’s  procurement and deployment plans for ASPs. In acknowledging that the  project execution plan should be revised and updated, these officials told  us that DNDO currently plans to field only the standard cargo ASP portal,  mainly at high-volume ports of entry. (We used the 2006 project execution  plan because it is the agency’s only official plan for ASP deployment.)
DNDO’s estimate considers only 8 years rather than 10 years, the  operational life expected by the manufacturer of sodium iodide crystals, a  key ASP component. DHS cost guidance maintains that a project’s life  cycle can be estimated over that period of time during which equipment  will remain available before it is exhausted, that is, decayed or  deteriorated. DNDO officials agreed that a 10-year life cycle cost estimate  would have been more appropriate and said that they would have used a  10-year estimate had they not been constrained by OMB budget  submission software, which limits the number of years of costs that can be  included.
DNDO’s cost estimate does not include all of the elements of the ASPs’ life  cycle, such as maintenance or operational sustainment. These costs are  approximately $999.2 million and $364.9 million, respectively, under our  cost estimate.
DNDO did not have detailed documentation of the costs to support its  estimates. According to OMB, DHS, and GAO guidelines, such  documentation is necessary to establish the basis of the estimates and to  provide assurances that the estimates are credible.
Assuming DNDO attempts to implement its authorized 2006 project  execution plan rather than its scaled-back plan, our report estimated a  $753 million budget shortfall for the radiation portal monitor project for  fiscal years 2008 through 2012 (the years included in both our analysis and  the DNDO analysis). Additionally, we estimated that DNDO will require  another $833 million from fiscal year 2013 through fiscal year 2017 to  complete the entire life cycle of the program.
According to DNDO officials, our cost estimate will result in estimates  higher than the program’s current true cost because it is based on the  outdated project execution plan. However, in the absence of more recent  documentation, we believe our independent cost estimate must be based  on the agency’s most recent approved plan (the 2006 project execution  plan). Furthermore, several official DNDO documents specify that multiple  versions of ASPs will be deployed. For example, the agency’s February  2006 expenditure plan submitted to the Congress foresees “several  variants” of ASP systems being deployed, with standard cargo, rail, and  automobile versions mentioned specifically. And DNDO’s submissions to  OMB for budget years 2008 and 2009 describe a program that includes land  crossings, seaports, rail lines, airports, and other ports of entry. Finally,  agency officials acknowledged the program requirements that would have  been fulfilled by the discontinued ASPs remain valid, including screening  rail cars, airport cargo, and cargo at seaport terminals, but the agency has  no current plans for how such screening will be accomplished. These  officials told us the technology to accomplish these requirements likely  will not be ASP monitors and could be a totally new technology. We  believe a comprehensive estimate of the cost to provide radiation  detection equipment for U.S. ports of entry should account for meeting  these objectives, even if DNDO decides that ASP technology is not suited  to them.
DNDO agreed in June 2008 to update its project execution plan so that we  could better estimate the costs of the agency’s current plans. DNDO also  agreed to provide an updated estimate of the standard portal’s life cycle  costs and to meet with us to reconcile our two estimates. However, DNDO  did not provide a revised project execution plan or cost estimate and  instead provided in July 2008 a one-page spreadsheet of summary  information outlining its plans to buy and deploy portal monitors for the 7- year period 2008 through 2014. DNDO’s summary data indicate that during  this time period the agency plans to deploy 717 ASPs and 1,005 PVTs. The  summary data do not provide the breadth and depth of information needed  to generate detailed and fully documented cost estimates. Furthermore,  according to subsequent discussions with a senior DNDO official, if  ongoing tests indicate the ASPs’ performance warrants it, the agency may  speed its deployment of ASPs over the next few months. In our view, the  frequent changes in deployment plans and the lack of available cost  documentation raise concerns about the overall management of the  radiation portal monitor project and whether it is guided by a sound and  stable strategy. Nonetheless, we used the agency’s summary data to  perform a more limited cost estimate for only the standard cargo portal.  We found that from 2008 to 2017 the total program cost for buying and  deploying standard cargo portals would likely be about $2 billion, but  could range from about $1.7 billion to $2.3 billion.
Our report recommended that the Secretary of Homeland Security direct  the Director of DNDO to (1) work with the Commissioner of CBP to  update the project execution plan to guide the entire radiation detection  program at U.S. ports of entry, (2) revise DNDO’s estimate of the  program’s cost and ensure that the estimate considers all the costs  associated with its project execution plan, and (3) communicate this  revised estimate to the Congress so that it is fully apprised of the  program’s scope and funding requirements. DHS agreed with our  recommendations in its comments responding to our draft report.
section_title: Preliminary Observations of Ongoing Testing of ASPs
paragraphs: We are currently reviewing the ongoing 2008 ASP testing program and  process leading to a decision by the Secretary of Homeland Security on  certification of ASPs, and we plan to issue a final report in November 2008,  prior to the Secretary’s decision on certification. Our work to date shows  that DNDO has made progress in addressing a number of problems we  identified in previous rounds of ASP testing. A particular area of  improvement is in the performance testing at the Nevada Test Site, where  DNDO conducts trials of the ability of ASPs to detect and identify  radioactive materials, particularly those that could be used in a nuclear  weapon. In 2007, we reported that DNDO had allowed ASP system  contractors to adjust their systems after preliminary runs using the same  radiological materials that would be used in the formal tests, potentially  biasing the test results. In contrast, in our more recent work, we found  that the plan for the 2008 round of performance testing stipulated that  there would be no system contractor involvement in test execution, and  no ASP system contractors were present at the test location on the day we  observed performance testing. Furthermore, DNDO officials told us, and  we observed, that they did not conduct preliminary runs with threat  objects that were used in the formal tests. A further area of improvement  in the performance testing concerns the use of handheld RIIDs.  Specifically, DNDO used actual CBP officers who adhered to nearly all of  the standard operating procedures in testing the RIIDs. This improvement  addresses our concern from last year that DNDO did not objectively test  the performance of the RIIDs because it did not use a critical CBP  standard operating procedure that is fundamental to the equipment’s  performance in the field.
Despite these improvements, we have identified potential areas of concern  based on our ongoing review. Our concerns fall into three areas:    DHS criteria for “significant increase in operational effectiveness.”
DNDO, CBP and the DHS management directorate agreed on six criteria  for the degree of improvement ASPs must provide over current generation  equipment—four criteria for primary screening and two for secondary  screening. These criteria appear to set a low bar for improvement. Most  notably, the criteria for primary screening require ASPs to perform at least  as well as current generation equipment when special nuclear material or  medical or industrial isotopes are present in cargo, but they do not specify  an actual improvement. Similarly, one of the criteria for secondary  screening requires ASPs to reduce the average time to release  conveyances but does not indicate a specific level of reduction. As such,  the criteria leave open the possibility of a Secretarial decision in favor of  certification even if ASPs do not provide a significantly higher probability  of detection compared with current generation PVTs when deployed for  primary screening and only a small reduction in the time required for  secondary screening. We recently requested additional information from  DNDO about the rationale behind these criteria, particularly in light of  seemingly stricter criteria found in other documents such as the ASP  performance specification, which requires an 80 percent probability of  detection, at a 95 percent confidence, of government-designated  representative threat objects. We are awaiting DNDO’s response.
The extent to which the 2008 testing program provides a sound basis for  determining ASPs’ operational effectiveness. While the performance  testing at Nevada Test Site could provide a useful comparison of the  performance of the ASPs and current generation equipment for both  primary and secondary screening given the improvements described  above, DNDO does not plan to complete “injection studies” that could  provide additional data on ASP performance prior to ASP certification.  (Injection studies are a type of computer simulation for testing the  response of the ASP threat identification algorithms to data on threat  objects combined with stream-of-commerce data from a port of entry.)  Rather, DNDO officials told us that testing using actual threat objects is  sufficient for certification. While we recognize the limitations of injection  studies such as the inability to exactly model threat objects hidden in  commerce, reliance on performance testing but not injection studies for  the Secretary’s decision on certification will result in less data with which  to evaluate the capabilities of ASPs, particularly for primary screening.  According to officials from Pacific Northwest National Laboratory, which  has a lead role in injection studies, computer simulations allow for an  increased statistical confidence in the measured probability of detection  compared with testing at Nevada Test Site. The ASP test and evaluation  master plan similarly emphasizes the importance of injection studies for a  complete test of radiation detection system performance, and a February  2008 DNDO presentation to us includes among its fiscal year 2008  objectives the use of preliminary results from injection studies to evaluate  performance of ASPs for primary screening. However, injection studies  have taken longer than projected due to a number of issues such as  deficiencies in the stream-of-commerce data.
The schedule leading up to a decision on ASP certification by the  Secretary of Homeland Security. The test schedule leading up to  certification is highly compressed and is running at least 8 weeks behind  the schedule DNDO provided to us in May 2008, leaving limited time for  analysis and review of test results. Key phases of the test schedule include  system qualification, followed by testing at Pacific Northwest National  Laboratory to demonstrate that ASPs are ready to be integrated into the  interdiction systems at U.S. ports of entry and concurrent testing at  Nevada Test Site to compare the ASP performance in detecting and  identifying threats with that of current generation equipment. According to  DNDO, the systems qualification and performance tests have been  completed, but only after being delayed due to problems with system  qualification. The final two phases, field validation at four ports of entry  operated by CBP and operational testing at one port of entry, have not yet  started. Field validation was to have started yesterday—September 24— but as with integration and performance testing this phase has also been  delayed, and CBP has not yet indicated when it will begin. Even before this  latest delay, the time between the scheduled end of operational testing in  early November and a Secretarial decision on certification on November  28 allowed less than a month for review and analysis of results.  Furthermore, DHS officials have indicated that the Secretary may make a  decision on certification on the basis of “quick look” (preliminary) rather  than final reports if the quick look reports are favorable to ASPs. The  limited time between completion of testing and a decision on certification,  combined with the potential reliance on quick look reports, increases the  risk of a decision in favor of certification being called into question later  by a more thorough analysis and review of results.
Assuming that DHS addresses these concerns—for example, by clarifying  the criteria for significant increase in operational effectiveness,  performing the injection studies, and delaying a Secretarial decision on  certification—the 2008 round of testing could provide an objective basis  for comparing ASPs with current generation equipment. However,  consistent with our March 2006 recommendation that DHS analyze the  benefits and costs of deploying ASPs to determine whether any additional  detection capability provided by ASPs is worth the cost, we would still  question the replacement of current generation equipment with ASPs,  particularly considering the gaps identified by DNDO in the global nuclear  detection architecture—essentially, an integrated system of radiation  detection equipment and interdiction activities to combat nuclear  smuggling in foreign countries, at the U.S. border, and inside the United  States. As we testified before this committee earlier this year, such gaps  include land border crossings into the United States between formal  points of entry, small maritime vessels, and international general aviation.  An updated cost-benefit analysis, based on the latest information on ASP  costs and test performance, could help policymakers address any trade- offs in addressing these gaps versus replacing current generation radiation  detection equipment already deployed at ports of entry. The current  generation equipment has known limitations, particularly with regard to  the limited ability to detect certain nuclear materials and the inability to  distinguish between types of radiological material. However, the existing  equipment provides at least some detection capability and may act as a  deterrent, and there is no evidence we are aware of that it is impeding the  flow of commerce.
Mr. Chairman, this concludes my prepared statement. I would be pleased  to respond to any questions that you or other members of the Committee  may have.
section_title: GAO Contact and Staff Acknowledgments
paragraphs: For further information on this testimony, please contact Gene Aloise at  (202) 512-3841 or by e-mail at aloisee@gao.gov. Individuals making key  contributions to this testimony include Joseph Cook, Jennifer Echard,  Brian Octeau, Alison O’Neill, Tim Persons (GAO’s Chief Scientist), Karen  Richey, Benjamin Shouse, Kevin Tarmann, Eugene Wisnoski, and Ned  Woodward.
This is a work of the U.S. government and is not subject to copyright protection in the  United States. The published product may be reproduced and distributed in its entirety  without further permission from GAO. However, because this work may contain  copyrighted images or other material, permission from the copyright holder may be  necessary if you wish to reproduce this material separately.