Combating Nuclear Smuggling: Inadequate Communication and Oversight Hampered DHS Efforts to Develop an Advanced Radiography System to Detect Nuclear Materials
Released: 2010-09-15T00:00:00

Letter

DNDO Planned for the Acquisition and Deployment of CAARS without Fully Understanding that It Could Not Feasibly Operate in a U.S. Port Environment

From the start of the CAARS program in 2005 until the course correction in December 2007, DNDO planned the acquisition and deployment of CAARS machines without understanding that they would not fit within existing primary inspection lanes at CBP ports of entry. This occurred because during the first year or more of the program DNDO and CBP had few discussions about operating requirements for primary inspection lanes at ports of entry. In addition, the CAARS program was among numerous acquisition programs about which we previously reported that appropriate DHS oversight was lacking. Furthermore, the development of the CAARS algorithms—a key part of the machine needed to identify shielded nuclear materials automatically—did not mature at a rapid enough pace to warrant acquisition and deployment. Moreover, the description of the progress of the CAARS program used to support funding requests in DNDO’s budget justifications for fiscal years 2009 through 2011 was misleading because it did not reflect the actual status of the program.

Inadequate Communication

From the inception of the CAARS program until the decision in December 2007 to cancel acquisition of the program, DNDO and CBP had few, if any, in-depth discussions about CBP’s requirements to be able to use radiography in primary inspection lanes. According to DNDO officials, they requested information from CBP on its user requirements for the CAARS system, but CBP was slow to respond to these requests. DNDO continued with its plans to develop CAARS machines because, according to DNDO officials, at the time it was thought that a solution was urgently needed to be able to detect shielded nuclear materials in primary inspection lanes. In discussing this with senior CBP officials, they said that DNDO officials did not attempt to meet with them during the beginning of the CAARS program. When CBP and DNDO officials met, shortly before the course correction, CBP officials said they made it clear to DNDO that they did not want the CAARS machines because they would not fit in primary inspections lanes and would slow down the flow of commerce through these lanes and cause significant delays.

In our view, had CBP and DNDO officials met early in the development of the program to discuss CBP’s needs and operational requirements, as stated in DHS’s acquisition policy at the time, it is unlikely that DNDO would have found reason to move forward with its plan to develop and acquire CAARS technology. Nonetheless, in September 2006, DNDO awarded contracts to three CAARS vendors. In December 2007, DNDO decided to cancel the acquisition of CAARS and limit any further work to a research and development effort. In recent joint discussions with CBP and DNDO officials, they acknowledged that communication between the two agencies could have been improved during the early part of the CAARS program. They said they communicate much more routinely now and that, in their view, it would be unlikely that the communication problems associated with the CAARS program would reoccur.

Inadequate Oversight

DNDO did not follow DHS acquisition protocols for the CAARS program. Specifically, in 2008, we reported that CAARS was among numerous major DHS acquisition programs that did not have a mission needs statement—a required DHS acquisition document that formally acknowledges that the need for an acquisition is justified and supported. DHS policy also called for programmatic reviews at key decision points and required certain analytical documents. However, CAARS did not undergo annual department level reviews as called for nor did DNDO program officials obtain or prepare basic analytical documents. For example, one of these documents, a concept of operations (CONOPS), was intended to demonstrate how CBP would use CAARS machines in primary inspection areas at the ports. However, as a result of inadequate communication and collaboration between CBP and DNDO discussed earlier, no CONOPS was developed during the early phase of the CAARS program. Ultimately, according to DNDO officials, once DNDO made the decision to cancel the acquisition portion of CAARS in December 2007, a CONOPS was no longer required.

Immature Technology

According to DNDO officials, at the time of the inception of the CAARS program, there was a widespread view within DNDO that something had to be done to provide CBP with the capability to detect highly shielded nuclear material in primary inspection lanes. DNDO officials acknowledged that the agency decided to move forward with the CAARS program despite the fact that automatic detection, a key feature of CAARS, depended on the rapid development of algorithms that were technologically immature. The algorithms are critical because they provide the capability for CAARS to automatically detect highly shielded nuclear material in primary inspection areas without the need for extensive operator review and interpretation of an image—two factors that could adversely affect CBP’s ability to avoid delays to the flow of commerce along with its overall effectiveness in detecting highly shielded nuclear material. Although algorithms supporting the CAARS technology were technologically immature, DNDO created an aggressive production and deployment schedule that was to begin in August 2008, the end of DNDO’s planned 2-year development period for the CAARS program. At the time it decided on this production milestone, DNDO officials said it was likely that the algorithms would be developed in time to meet the start of planned production. However, the technology did not develop as expected and contributed to DNDO’s decision to cancel the acquisition phase of CAARS.

Basis of CAARS Funding Requests

For fiscal year 2009 through fiscal year 2011, DHS justified annual budget requests to Congress by citing significant plans and accomplishments of the CAARS program, including that CAARS technology development and deployment was feasible, even though DNDO had made the decision in December 2007 to cancel the acquisition of CAARS. For example, in its fiscal year 2009 budget justification, DHS stated that a preliminary DNDO/CBP CAARS production and deployment program had been successfully developed and that CAARS machines would be developed that would detect both contraband and shielded nuclear material with little or no impact on CBP operations. The fiscal years 2010 and 2011 DHS budget justifications both cited that an ongoing testing campaign would lead to a cost benefit analysis, followed by rapid development of a prototype that would lead to a pilot deployment at a CBP point of entry. Furthermore, the fiscal year 2010 budget justification stated that while the CAARS technology was less mature than originally estimated, successful development was still feasible. However, DHS’s description and assessment of the CAARS program in its budget justification did not reflect the actual progress of the program. Specifically, DNDO officials told us that when they made their course correction and cancelled the acquisition part of the program in 2007, they also decided not to conduct a cost benefit analysis because such analyses are generally needed to justify going forward with acquisitions. In addition, DNDO completed CAARS testing in March 2010; however, as of today, the final test results for two of the three CAARS machines are not yet available. Currently, no CAARS machines have been deployed. CAARS machines from various vendors have either been disassembled or sit idle without being tested in a port environment, and CBP is considering whether to allow DNDO to collect operational data in a port environment. During recent discussions with DNDO officials, they agreed that the language in the budget justifications lacked clarity and stated that they are not planning to complete a cost benefit analysis since such analyses are generally associated with acquisition programs.

CAARS Offers Lessons Learned Regarding the Importance of Developing Requirements, Coordinating with Users, and Managing Acquisitions

Based on our review of the CAARS program and our reports on DNDO efforts to develop an advanced RPM called the advanced spectroscopic portal (ASP), we have identified lessons learned for DHS to consider in its continuing efforts to develop the next generation of radiography imaging technology.

Enhance Interagency Collaboration and Coordination

Despite the importance of coordinating crosscutting program efforts, we have reported that weak coordination of those efforts has been a long- standing problem in the federal government and has proven to be difficult to resolve. We have also reported that agencies can enhance and sustain their collaborative efforts. One way we reported that agencies can enhance coordination is to agree on roles and responsibilities and establish mutually reinforcing or joint strategies. As discussed, DNDO did not coordinate and collaborate with CBP early in the development of the CAARS program to identify CBP’s needs and requirements. According to DHS budget documents, in fiscal year 2011, the responsibility for research and development of advanced radiography will shift from DNDO to S&T. Leading up to this transition, there is confusion related to roles and responsibilities among DNDO, S&T, and CBP. For example, DNDO officials said they have requested permission from CBP to collect operational data in a port environment on an enhanced radiography machine. However, CBP officials stated that they had already purchased, operationally tested, and deployed 11 of these machines in secondary inspection areas. We recently discussed this issue at a joint meeting with DNDO and CBP officials. CBP and DNDO officials agreed that there was confusion over this issue, and both agencies agreed with the need to collect operational data on this enhanced radiography machine, and CBP has begun making arrangements to do so.

Also, S&T officials said that they are about to contract out for radiography imaging technology for CBP that will improve imaging capabilities. DNDO officials told us that S&T’s efforts will include development of radiography capabilities to detect shielded nuclear material, while S&T officials told us that this is not an area of their focus. As DHS transitions its research and development of radiography, DHS officials said that a draft memorandum of agreement intended to clarify roles and responsibilities for cooperation and coordination among DNDO, CBP, and S&T has not been finalized. Completing the memorandum of agreement to clarify roles and responsibilities before proceeding with the research, development, and deployment of radiography technology could give DHS reasonable assurance that problems resulting from a lack of clearly defined roles and responsibilities in the CAARS program do not recur. In discussions with senior officials from DHS, DNDO, CBP and S&T, they all agreed with the need for the memorandum and said that they intend to work toward finalizing the draft memorandum of agreement.

Engage in a Robust Oversight Review Process

DNDO officials said that they were aware of the DHS draft management directive in 2006 that was intended to guide management and oversight of acquisition programs like CAARS but did not follow it. DHS policy officials acknowledged that at the time CAARS was in its early stages, DHS was continuing the process of organizing and unifying its many disparate components and there was not strong oversight over its major programs, including CAARS. Policy officials told us the oversight review process is more robust today. However, we reported in June 2010 that DHS acquisitions need further improvement and sustained management attention. For example, while DHS’ current management directive includes more detailed guidance than the previous 2006 management directive for programs to use in preparing key documentation to support component and departmental decision making, it is not applied consistently and most major programs have not been reviewed.

Separate Research and Development from Acquisition Functions

DNDO was simultaneously engaged in a research and development phase while planning for an acquisition phase of the CAARS program. In this regard, we have previously reported that separating technology development from product development and acquisition is a best practice that can help reduce costs and deliver a product on time and within budget because separation of the technology development phase from production in particular helps to ensure that (1) a sound business case is made for the product, (2) product design is stable, and (3) production processes are mature and the design is reliable.

Determine the Technology Readiness Levels Before Acquisition

At the time that the CAARS program was in its early stages, DHS and DNDO did not have clearly defined ways to define and communicate the maturity of technology leading to acquisition. We have previously reported on the need for a disciplined and knowledge-based approach of assessing technology maturity, such as using technology readiness levels. In that report, we recommended that technologies need to reach a high readiness level before an agency should make a commitment to production. DNDO officials acknowledged that CAARS algorithm’s readiness level was not high enough to warrant entering into the acquisition phase.

Rigorously Test Devices Using Actual Agency Operational Tactics

As we testified in June 2009 on DNDO’s testing of ASPs, a primary lesson to be learned regarding testing is that the push to replace existing equipment with the new portal monitors led to an ASP testing program that lacked the necessary rigor. We reported that testing programs designed to validate a product’s performance against increasing standards for different stages in product development are a best practice for acquisition strategies for new technologies and if properly implemented, would provide rigor to DHS’s testing of other advanced technologies.

GAO Contacts and Staff Acknowledgements

For further information about this statement, please contact Gene Aloise at (202) 512-3841 or aloisee@gao.gov; or Stephen L. Caldwell at 202-512- 9610 or caldwells@gao.gov. Dr. Timothy Persons (Chief Scientist), Ned Woodward (Assistant Director), Mike Harmond, Jonathan Kucskar, Linda Miller, Ron Salo, Kiki Theodoropoulos, and Franklyn Yao also made key contributions to this testimony.

Appendix I: Related GAO Products

Maritime Security: DHS Progress and Challenges in Key Areas of Port Security. GAO-10-940T. Washington, D.C.: July 21, 2010.

Combating Nuclear Smuggling: DHS Has Made Some Progress but Not Yet Completed a Strategic Plan for Its Global Nuclear Detection Efforts or Closed Identified Gaps, GAO-10-883T, Washington D.C.: June 30, 2010.

Supply Chain Security: Feasibility and Cost-Benefit Analysis Would Assist DHS and Congress in Assessing and Implementing the Requirement to Scan 100 Percent of U.S.-Bound Containers. GAO-10-12. Washington, D.C.: Oct. 30, 2009.

Combating Nuclear Smuggling: Lessons Learned from DHS Testing of Advanced Radiation Detection Portal Monitors, GAO-09-804T, Washington, D.C.: June 25, 2009.

Combating Nuclear Smuggling: DHS Improved Testing of Advanced Radiation Detection Portal Monitors, but Preliminary Results Show Limits of the New Technology. GAO-09-655. Washington D.C.: May 21, 2009.

Nuclear Detection: Domestic Nuclear Detection Office Should Improve Planning to Better Address Gaps and Vulnerabilities. GAO-09-257. Washington, D.C.: Jan. 29, 2009, Combating Nuclear Smuggling: DHS’s Program to Procure and Deploy Advanced Radiation Detection Portal Monitors is Likely to Exceed the Department’s Previous Cost Estimates. GAO-08-1108R. Washington, D.C.: Sept. 22, 2008.

Supply Chain Security: CBP Works with International Entities to Promote Global Customs Security Standards and Initiatives, but Challenges Remain. GAO-08-538. Washington, D.C.: Aug. 15, 2008 Maritime Security: National Strategy and Supporting Plans Were Generally Well-Developed and Are Being Implemented. GAO-08-672. Washington, D.C.: June 20, 2008.

Supply Chain Security: Challenges to Scanning 100 Percent of U.S.- Bound Cargo Containers. GAO-08-533T. Washington, D.C.: June 12, 2008.

Supply Chain Security: U.S. Customs and Border Protection Has Enhanced Its Partnership with Import Trade Sectors, but Challenges Remain in Verifying Security Practices. GAO-08-240. Washington, D.C.: Apr. 25, 2008.

Supply Chain Security: Examination of High-Risk Cargo at Foreign Seaports Have Increased, but Improved Data Collection and Performance Measures Are Needed. GAO-08-187. Washington, D.C.: Jan. 25, 2008.

Department of Homeland Security: Billions Invested in Major Programs Lack Appropriate Oversight. GAO-09-29. Washington, D.C.: Nov. 18, 2008).

Maritime Security: The SAFE Port Act: Status and Implementation One Year Later. GAO-08-126T. Washington, D.C.: Oct. 30, 2007.

Combating Nuclear Smuggling: Additional Actions Needed to Ensure Adequate Testing of Next Generation Radiation Detection Equipment. GAO-07-1247T. Washington, D.C.: Sept. 18, 2007.

Department of Homeland Security: Progress Report on Implementation of Mission and Management Functions. GAO-07-454. Washington, D.C.: Aug. 17, 2007.

International Trade: Persistent Weaknesses in the In-Bond Cargo System Impede Customs and Border Protection’s Ability to Address Revenue, Trade, and Security Concerns. GAO-07-561. Washington, D.C.: Apr. 17, 2007.

Combating Nuclear Smuggling: DHS’s Decision to Procure and Deploy the Next Generation of Radiation Detection Equipment is Not Supported by Its Cost-Benefit Analysis. GAO-07-581T. Washington, D.C.: Mar. 14, 2007.

Combating Nuclear Smuggling: DNDO Has Not Yet Collected Most of the National Laboratories’ Test Results on Radiation Portal Monitors in Support of DNDO’s Testing and Development Program. GAO-07-347R. Washington, D.C.: Mar. 9, 2007.

Combating Nuclear Smuggling: DHS’s Cost-Benefit Analysis to Support the Purchase of New Radiation Detection Portal Monitors was Not Based on Available Performance Data and Did not Fully Evaluate All the Monitors’ Costs and Benefits. GAO-07-133R. Washington, D.C.: Oct. 17, 2006.

Cargo Container Inspections: Preliminary Observations on the Status of Efforts to Improve the Automated Targeting System. GAO-06-591T. Washington, D.C.: Mar. 30, 2006.

Combating Nuclear Smuggling: Challenges Facing U.S. Efforts to Deploy Radiation Detection Equipment in Other Countries and in the United States. GAO-06-558T. Washington, D.C.: Mar. 28, 2006.

Combating Nuclear Smuggling: DHS Has Made Progress Deploying Radiation Detection Equipment at U.S. Ports-of-Entry, but Concerns Remain. GAO-06-389. Washington, D.C.: Mar. 22, 2006.

This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.