Combating Nuclear Smuggling: Lessons Learned from Cancelled Radiation Portal Monitor Program Could Help Future Acquisitions
Published: 2013-05-13T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Preventing terrorists from smuggling radiological or nuclear material into the United States to carry out an attack is a national priority. DHS's DNDO develops and deploys radiation detection equipment to assist other federal agencies, such as CBP, in intercepting illicit radiological or nuclear materials that could be used to make a radiological dispersive device (dirty bomb) or a crude nuclear bomb. CBP uses RPMs at nearly all land border crossings and seaports to detect radiation in trucks and cargo. DHS recently cancelled acquisition of ASPs, which was originally envisioned as costing from $2 billion to $3 billion.
GAO was asked to provide updated information on the ASP program. This report examines, among other things, (1) the results of ASP testing conducted in 2009 and 2010 that led to DHS's decision to cancel the ASP program and (2) the benefits of lessons learned reviews and how DHS captures any lessons learned when programs are cancelled. GAO reviewed testing and acquisition documents and interviewed key agency officials, as well as seven experts the National Academies identified for their knowledge of leading practices in large-scale engineering and acquisition programs.
section_title: What GAO Found
paragraphs: The advanced spectroscopic portal monitor (ASP)--a next-generation radiation portal monitor (RPM) for screening trucks and cargo containers--did not pass field validation tests conducted in 2009 and 2010. The Department of Homeland Security's (DHS) Domestic Nuclear Detection Office (DNDO) intended to replace many currently deployed RPMs and handheld radiation detectors used by U.S. Customs and Border Protection (CBP) with ASPs. However, in the tests, ASP did not meet key requirements to detect radiation and identify its source. For example, ASP triggered too many false alarms from benign, naturally occurring radioactive material in common items such as kitty litter and granite, and it sometimes would not turn on or continue operating long enough to complete a day of testing. In addition, GAO's review identified analytical weaknesses related to the testing and program cancellation, including inconsistencies in DNDO's analysis of the settings used for testing the ASP. The final field validation test was conducted in November 2010, and the Secretary of Homeland Security notified Congress of her decision to cancel the program in October 2011.
Conducting lessons learned reviews when programs are cancelled benefits organizations by identifying things that worked well and did not work well in order to improve future acquisitions programs, according to experts GAO consulted. However, DHS does not have processes in place to ensure such reviews are conducted or that the results are disseminated. Experts identified by the National Academies told GAO that lessons learned reviews help identify reasons why programs were cancelled. The experts also said lessons learned reviews should be required and conducted promptly, and the results should be disseminated. At the direction of DHS management, DNDO reviewed the ASP program and submitted and disseminated a lessons learned report in November 2012. This report cited 32 lessons learned including having program officials work closely with end users to ensure equipment meets operational requirements. DHS guidance calls for lessons learned reviews immediately after programs are cancelled and states that the lessons learned are to be shared throughout the department, but this guidance is not a requirement. Before DHS's directive, there was confusion about whether a lessons learned review was needed for the ASP program, and DNDO officials did not intend to conduct such a review. Moreover, DHS officials were unable to provide examples of previous lessons learned reports from other cancelled programs. DHS officials also said they have no process for disseminating such reports but are planning one.
section_title: What GAO Recommends
paragraphs: DHS should require lessons learned reviews and develop processes to ensure such reviews are done in a timely manner and the results disseminated throughout the department. DHS agreed with all of GAO's recommendations and has planned and taken some actions to address them.

section_title: Letter
paragraphs: Preventing terrorists from smuggling radiological or nuclear material into  the United States to carry out an attack is a national priority. To  implement federal policy to protect the nation against radiological or  nuclear attacks, among other purposes, the Department of Homeland  Security’s (DHS) Domestic Nuclear Detection Office (DNDO) was  established in 2005. DNDO is tasked with coordinating development of a  global nuclear detection architecture and is responsible for implementing  the domestic portion of the architecture. It also develops, acquires, and  deploys radiation detection equipment to support the efforts of other  federal agencies. For example, DHS’s U. S. Customs and Border  Protection (CBP) uses radiation detection equipment at U.S. ports of  entry to screen cargo containers and trucks for illicit radiological  materials, including material that could be used in radiological dispersive  devices (dirty bombs) and special nuclear material that could be used to  make a nuclear weapon.
Potential pathways for illicit radiological materials to enter the United  States include being smuggled in trucks at border crossings or in cargo  on ships. According to DHS officials, nearly all trucks at land border  crossings and containerized cargo at seaports are screened for  radiological materials. To screen cargo at these locations, CBP uses  radiation portal monitors (RPM)—large stationary radiation detectors  through which trucks and cargo containers pass. (See fig. 1.)
According to DNDO the current generation of RPMs has limitations in that  it can detect but cannot identify sources of radiation. Because of this  inability, the RPMs’ radiation alarms can be triggered by benign, naturally  occurring radioactive material present in common items such as kitty litter  and granite. Cargo that has triggered an alarm during the initial  (“primary”) screening is sent for additional inspection (i.e., “secondary”  screening), first by another RPM to confirm the alarm and then by a CBP  officer using a handheld radiation detector—about the size of a shoe  box—that can identify the source of the radiation (see fig. 2).
One way to reduce the rate of alarms that are triggered by benign  materials—and thereby reduce the number of unnecessary secondary  screenings—would be to use next-generation RPMs that can both detect  and identify radiation sources. In 2005, DNDO began working with CBP  on a program to develop and test a type of next-generation RPM called  the advanced spectroscopic portal monitor (ASP), which was designed to  both detect radiation and identify the source as benign, suspect, or a  threat. The initial concept of the program was to develop, procure, and  deploy enough ASPs to replace many of CBP’s currently deployed RPMs  and handheld detectors at a cost of $2 billion to $3 billion, according to  DNDO.
Throughout the ASP program, DNDO and CBP faced many challenges,  on which we have reported and testified to Congress several times. Our  last in-depth review of the ASP program was in May 2009, when we  reported that the ASP had mixed performance during testing to determine  whether it could successfully be used for primary and secondary  screening. Challenges in the ASP program were also identified by the  National Research Council of the National Academies. In 2009, the  National Research Council recommended in an interim report that DNDO  use a better approach to testing, evaluation, cost-benefit assessment,  and deployment of ASPs. In 2010, the National Research Council found  in its final report that, among other things, DNDO’s 2008 testing had  shortcomings that impaired the ability to draw conclusions about ASP’s  likely performance, and DNDO’s draft cost-benefit analysis needed  substantial improvement to support decision making.
In July 2011, after several years of testing, the DNDO Director testified  that the Secretary of Homeland Security had directed DNDO and CBP to  “end the ASP program as originally conceived” and to instead use  existing ASPs to help CBP officers gain operational familiarity with ASP  and to gather data to support a future acquisition program. In October  2011, the Secretary of Homeland Security wrote to key congressional  committees informing them of her decision to cancel the ASP program.  In July 2012, the Under Secretary for Management stated that, in  accordance with the Secretary’s letter, the program was considered  cancelled. Once cancelled, programs such as ASP are subject to  provisions of DHS’s acquisition guidance regarding review of cancelled  programs and dissemination of the resulting lessons learned.
In this context, you asked us to provide updated information on the ASP  program. Our objectives were to determine: (1) the results of ASP testing  conducted in 2009 and 2010 that led to DHS’s decision to cancel the ASP  program; (2) the benefits of lessons learned reviews and how DHS  captures any lessons learned when programs are cancelled; and (3) what  additional testing, if any, of next-generation RPMs, including ASP, DHS is  conducting or planning.
To determine the results of the ASP testing conducted in 2009 and 2010  that led to the decision to cancel the ASP program, we reviewed ASP  testing documents and briefing materials from DNDO, CBP, the  Department of Energy’s (DOE) Pacific Northwest National Laboratory,  and Johns Hopkins University’s Applied Physics Laboratory and  interviewed officials from DNDO and CBP. To determine the benefits of  lessons learned reviews and how DHS captures any lessons learned  when programs are cancelled, we analyzed acquisition policy documents  from DHS and interviewed officials from DNDO, DHS’s Office of Program  Accountability and Risk Management (PARM), and DHS’s Enterprise  Business Management Office. In addition, we interviewed seven experts  identified by the National Academies to highlight leading practices for  reviewing cancelled acquisition programs; these experts were identified  on the basis on their knowledge of leading practices in large-scale  engineering and acquisition programs. (See app. I for a list of the experts  we interviewed.) To determine what additional testing of next-generation  RPMs, including ASP, DHS is conducting or planning, we reviewed  documents from DHS, DNDO, the National Institute of Standards and  Technology, the American National Standards Institute, and the  International Electrotechnical Commission, and we witnessed some of the  testing. We also interviewed officials from DNDO and DOE’s Oak Ridge  National Laboratory.
We conducted this performance audit from October 2011 to May 2013 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: DHS began testing next-generation RPMs with a preliminary small-scale  study performed in July 2004 by DOE’s Pacific Northwest National  Laboratory (PNNL). (See fig. 3.) Specifically, PNNL performed small- scale, side-by-side comparisons of next-generation RPMs and the RPMs  used by CBP. This study found that in some situations, the performance  of the next-generation RPMs was better than that of CBP’s RPMs, and  that in other situations, the performance of the two was equal. In a follow- up, small-scale study conducted for DHS in July 2005, PNNL found that  the next-generation RPMs performed no better than CBP’s currently  deployed RPMs but did produce fewer alarms for benign materials.
In October 2005, DNDO conducted a number of tests on next-generation  RPMs that DHS had commissioned from different manufacturers and  used the test results to help evaluate proposals for the ASP program. In  July 2006, DNDO awarded contracts for development and production of  ASPs. Among other things, according to DNDO officials, the design  specifications for ASP stated that it should operate on trucks traveling at 5  miles per hour during primary screening and 2 miles per hour during  secondary screening. DNDO began testing these ASPs at DOE’s Nevada  National Security Site in February 2007. As we have previously  reported, we identified a number of weaknesses in the methods used for  these tests that impaired the significance and validity of the test results.
Concerned about the performance and cost of ASP, Congress required  the Secretary of Homeland Security to certify that ASP would provide a  “significant increase in operational effectiveness” before DNDO obligated  funds for full-scale procurement of ASPs. In response, DNDO, CBP, and  DHS jointly issued criteria in July 2008 for determining whether ASP  provided a significant increase in operational effectiveness compared with  existing equipment. The criteria generally compare ASP with the RPMs  used by CBP under CBP’s standard operating procedure. Specifically, as  shown in table 1, there were four criteria for primary screening and two  criteria for secondary screening. In addition to these overarching criteria,  each phase of testing also had criteria that needed to be met to complete  that phase.
The final ASP testing campaign began in July 2008 and was concluded in  November 2010. DNDO started the campaign with performance testing at  DOE’s Nevada National Security Site. The performance tests determined  the probability of detection and identification of radiation sources,  including special nuclear material that could be used to make a nuclear  weapon. CBP then began field validation testing in 2009. Field validation  tests are the first step in determining the suitability of fully operational  equipment in the actual environment where the equipment is intended to  be used. The ASP field validation tests were performed at land border  crossings in Detroit, Michigan, and Laredo, Texas, and at seaports in  Long Beach, California, and in New York City, New York. CBP made  three attempts to complete this testing—in January and February 2009,  July and August 2009, and October and November 2010.
This testing, which we reviewed for this report, evaluated the compatibility  of ASP with CBP’s interdiction system on the basis of 10 additional  criteria CBP had specified for completion of this phase of testing.  Among other things, these criteria included whether, under normal field  operating conditions, ASP improved detection to the required degree  compared with the currently deployed CBP equipment. To pass the field  validation testing phase, ASP had to meet all 10 criteria. In addition,  DNDO’s ASP program manager specified that ASP could not have any  major unresolved issues that might render it ineffective or unsuitable for  use by CBP.
section_title: Test Results Show That ASP Did Not Meet Criteria to Pass Field Validation Testing, Leading DHS to Cancel the Program
paragraphs: Because of unsatisfactory test results, ASP did not pass field validation  testing, which led DHS to cancel the program. Specifically, in CBP’s first  and second attempts to pass field validation testing in 2009, ASP did not  meet 1 of the 4 criteria for demonstrating a significant increase in  operational effectiveness during primary screening–that is, it did not refer  at least 80 percent fewer trucks for secondary screening than CBP’s  currently deployed RPMs. When CBP began the first field validation test  in late January 2009, CBP found ASP produced a higher number of  referrals to secondary screening than CBP’s currently deployed RPMs.  CBP suspended the test about 2 weeks later in mid-February; CBP  officials reported that the problem with referrals required significant  corrective actions before testing could resume. DNDO, working with  Johns Hopkins University’s Applied Physics Laboratory, determined that  the problem could be addressed by revising ASP’s software to raise the  thresholds for triggering alarms.
CBP conducted a second field validation test of ASP in July and August  2009 using the revised ASP software. CBP and DNDO documents  indicate that, with the revised software, ASP was able to reduce referrals  to secondary screening by about 70 percent but not by the 80 percent  required to demonstrate a significant increase in operational  effectiveness. Significantly, according to these documents, the majority of  the ASP referrals to secondary screening were for alarms that falsely  indicated the presence of special nuclear material that could be used to  make a nuclear weapon. CBP officials told us these types of false alarms  were very disruptive in a port environment because they caused CBP  officers to take enhanced security precautions—specifically, CBP officers  had to conduct a thorough inspection to ensure no special nuclear  material was present before permitting the cargo to enter the country.  Moreover, repeated false alarms for such nuclear materials could cause  CBP officers to be skeptical about future alarms.
In the course of the second field validation test, ASP also experienced a  “critical failure,” which caused it to shut down. Importantly, during this  critical failure, ASP did not emit a signal to alert the CBP officer that it was  no longer screening cargo. Had this failure not occurred in a controlled  testing environment, in which CBP was using its RPMs in tandem with  ASP, the CBP officer would have permitted cargo to enter the country  unscreened for radiological material. Testing is performed so that issues  like this can be discovered and fixed prior to full-scale production and  deployment.
Following the second field validation test, DNDO’s Acting Director stated  that the ASP program would be scaled back to secondary screening  only. According to the Acting Director, DNDO’s decision was based on  ASP’s performance and cost. Specifically, the Acting Director stated that  DHS decided to not pursue ASP for primary screening because ASP did  not meet DHS’s criteria for a significant increase in operational  effectiveness. However, the Acting Director stated that results to date had  shown that ASP met secondary screening criteria by wide margins. The  Acting Director also stated that the cost to replace current RPMs with  ASP for secondary screening was relatively low—approximately $350  million—compared with the original total program cost of $2 billion to $3  billion to use ASP for primary and secondary screening. This is because  the original program called for many more ASPs to be deployed in CBP  primary screening than secondary screening.
In November 2010, ASP could not pass the third and final field validation  test for use in secondary screening because ASP did not meet 6 of the 10  criteria needed to complete the field validation testing phase, according to  our review of CBP test documents. According to these documents, 2 of  the 6 criteria that ASP did not meet involved issues that would  significantly affect operations unless they could be mitigated or resolved.  ASP did not meet these criteria in part because of its inability to turn on at  the beginning of each day and to continue operating long enough to  complete a full day of testing. In addition, when ASP was operating, it  produced an excessive number of alarms that incorrectly indicated  naturally occurring radioactive material in the cargo might be masking  special nuclear material. In some cases, testing of the ASP had to be  suspended because the high number of these alarms was backing up  traffic for secondary screening, delaying the flow of commerce. According  to the CBP documents, the remaining 4 of the 6 criteria that ASP did not  meet involved issues that could affect or delay port operations, although  not as significantly. For example, ASP did not meet one of these criteria  because of difficulty in integrating ASP into a CBP information network.
In addition to the 6 criteria that the ASP did not meet, DNDO test  documents identified several major unresolved issues that would have  needed to be resolved for successful completion of field validation testing.  In explaining these issues, DNDO and CBP officials told us that, for  example, at certain times of the day, direct sunlight impaired the ability of  the ASP to operate properly. See table 2 for a summary of the results for  the three field validation tests.
In addition to the issues identified during testing, our review identified  analytical weaknesses before and after the third field validation test. For  example, DNDO was inconsistent in its analysis of how to properly set up  the ASP for secondary screening. Specifically, ASPs have separate  settings (modes) for primary and secondary screening, and before the  third field validation test, DNDO performed an analysis that indicated ASP  would perform best during secondary screening if the machines were set  in primary screening mode. However, after the third field validation test,  DNDO officials analyzed the test results and concluded that ASP would  have performed better for secondary screening had the machines been  set in secondary screening mode. Furthermore, the analytical basis for a  key decision regarding the ASP program is unclear. Specifically, DNDO  could not provide us with any supporting analysis on a principal factor that  led to the decision to cancel the program. According to DNDO officials, for  ASP to work in secondary screening, truck speeds would need to average  2 miles per hour, as indicated in the design specifications. On the basis of  these specifications, the ASP Governance Board concluded in February  2011 that ASP was not operationally suitable for secondary screening  and should not be deployed because, according to CBP officials, it was  not possible to control truck speed to an average of 2 miles per hour.  However, when we asked DNDO officials for the data or analysis  supporting this conclusion, these officials could not provide such support.  Moreover, during congressional testimony in July 2012, the Acting  Director of DNDO said that it is possible to control truck speed to 2 miles  per hour at state-operated truck weigh stations. Accordingly, it is unclear  why such speeds cannot be achieved at CBP land border crossings and  seaports but can be achieved at truck weigh stations.
section_title: Lessons Learned Reviews Improve Future Acquisition Efforts, but DHS Does Not Have Processes Ensuring Such Reviews
paragraphs: Conducting lessons learned reviews when programs are cancelled  benefits organizations by identifying things that worked well and did not  work well in order to improve future acquisitions programs, according to  experts we consulted; however, DHS does not have processes in place to  ensure such reviews are conducted and reports documenting the results  of the review are disseminated. To determine the key benefits of  conducting lessons learned reviews, we consulted seven experts in large- scale engineering and acquisitions programs who were identified at our  request by the National Academies. (See app. I for a list of the experts we  interviewed.) When we asked these experts for their views on lessons  learned reviews and reports for cancelled acquisition programs, they  generally agreed on the following observations:
Lessons learned reviews help to determine the reasons why programs  were cancelled including problems with the technology, management,  or setting requirements. By identifying problematic practices and  causes for program failure, lessons learned reports allow an  organization to improve future efforts.
Lessons learned reports should be prepared promptly because  otherwise, knowledgeable personnel may not be available to  contribute to the reports, important details may not be recalled  accurately, and dissemination of lessons learned will be delayed.
Lessons learned reports should not be optional for an organization. A  requirement for lessons learned reports should be institutionalized— lessons learned reports should be required for programs administered  by the organization rather than being an ad hoc requirement.  However, an appropriate acquisitions executive should have the  authority to tailor or waive the lessons learned report based on factors  such as the size or cost of a program, its perceived importance and  effects, or the perceived causes for failure.
Lessons learned reports must be submitted to an appropriate  acquisitions executive to be useful. The acquisitions executive should  disseminate the lessons learned reports to other appropriate  acquisition officials. Lessons learned reports should be disseminated  widely; however, they should be edited to preserve the lessons  learned while minimizing potential damage to the reputations of  people and organizations that were involved with the program.
DHS interim acquisition guidance, issued in 2010, also recognizes the  benefits of timely lessons learned reviews for cancelled programs,  although as guidance it does not constitute a requirement. Similar to the  views of the experts identified by the National Academies, the DHS  acquisition guidance states that (1) the objective of the reviews is to share  lessons learned throughout the department to increase the probability of  success for future acquisition programs and (2) the reviews are to take  place immediately after programs are cancelled. In addition, an official  from the DHS’s Office of Program Accountability and Risk Management  (PARM)—DHS’s policy office for acquisitions—told us that it is important  to conduct these reviews immediately so that key people remain available  and do not forget the factors that contributed to programs being  cancelled. Under DHS acquisition guidance, upon completion of the  lessons learned review, a report documenting the lessons learned is to be  submitted to PARM.
DHS acquisition guidance on lessons learned provides general direction  on what is to happen when programs are cancelled, but it does not  specify processes to follow. PARM officials stated that there were no  documented processes in place for (1) conducting timely lessons learned  reviews, (2) preparing lessons learned reports, or (3) disseminating  lessons learned reports throughout the department. Under the federal  standards of internal control, agencies are to clearly document internal  controls (i.e., in management directives, administrative policies, or  operating manuals), and this documentation should be readily available  for examination. PARM officials told us they generally rely on DHS  component agencies, such as DNDO, to self-monitor many aspects of  their programs including conducting lessons learned reviews, preparing  lessons learned reports, and submitting lessons learned reports.
In the case of the ASP program, a lessons learned review was  conducted, and a lessons learned report was submitted to PARM and  disseminated within CBP and DNDO in November 2012 at the explicit  direction of DHS’s Under Secretary for Management. The lessons  learned report states that the program was cancelled by the Secretary in  October 2011. However, before the Under Secretary’s directive, there  was confusion about whether such a review should be conducted for  cancelled programs, as well as uncertainty about whether the program  had been cancelled. DNDO officials told us in March 2012 that, even  though they considered the program to have been cancelled by the  Secretary in October 2011, they did not intend to conduct a lessons  learned review because such reviews were not needed for cancelled  programs. Furthermore, there was uncertainty between DNDO and  PARM about whether and when the program had been cancelled. In this  regard, while DNDO relied on the Secretary’s October 2011 letter, PARM  officials told us that the program was not cancelled and would not be  cancelled until DNDO and CBP completed certain additional actions such  as gathering data to support a future acquisition program. These actions  were started but not completed, however, the uncertainty over the  program’s status was eventually resolved in July 2012 with a  memorandum from the Under Secretary for Management which stated  that, in accordance with the Secretary’s October 2011 letter, the program  was considered cancelled. In discussing this with DNDO, and DHS  officials, they acknowledged that the DHS acquisition guidance is  confusing on when programs are considered cancelled and when to begin  conducting a lessons learned review. DHS officials stated that they are in  the process of revising the guidance to provide greater clarity on these  issues.
To determine whether the confusion and uncertainty that affected the  ASP program were unique to that program, we asked PARM officials to  provide us with examples of previous lessons learned reports from any  cancelled programs across DHS, including several programs cancelled in  2011 that we had previously reported upon. The PARM officials told us  that they did not have any lessons learned reports for these programs or  any others in the department. Furthermore, there was confusion within  the department about the need to disseminate lessons learned reports.  Before the Under Secretary’s memorandum of July 2012 directing DNDO  and CBP to prepare and disseminate an ASP lessons learned report,  PARM officials had told us there was no requirement to disseminate  lessons learned outside individual component agencies. Subsequent to  the memorandum, PARM officials told us that they are developing plans  to disseminate lessons learned reports through department-wide Centers  of Excellence.
The lessons that can be learned from such reviews and reports can be  extensive and informative. In the case of the ASP program, DNDO and  CBP’s lessons learned report identified 32 lessons learned in the  following six categories: documentation of planning;  structuring and staffing of the program office; coordination between stakeholders and end users regarding technical  and operational requirements, as well as costs. (See app. II for a copy  of the lessons learned report.)
For example, concerning acquisition strategy, the lessons learned report  states the government should consider a “commercial first” approach— meaning consider whether commercially available equipment can meet  programmatic needs before developing new technologies. Such an  approach can reduce costs and risks to the program. Another lesson  learned from the ASP program is that the acquisition officials and the end  users must work closely together to ensure the needed capability meets  operational requirements. An additional lesson learned was to manage  the schedule of the acquisition program by events—not the calendar. In  other words, the program should not proceed to the next major phase  until all the criteria are met for completing the current phase. Furthermore,  the report stated that acquisition programs should be flexible enough to  respond to major issues that may be discovered after program initiation.  Specifically, management should take care not to believe in the program  so strongly that early signs of trouble are ignored, or cannot be detected  and acted upon in a timely manner.
At this point, however, PARM has no assurance that that lessons learned  reviews have been or will be conducted for other cancelled programs, as  called for in DHS acquisition guidance. Until DHS puts processes in place  to ensure component agencies conduct lessons learned reviews and  prepare lessons learned reports, and PARM implements its plans for  disseminating the reports, DHS risks limiting the probability of success for  the department’s future acquisition programs. Success in such acquisition  programs is important given the scale of DHS’s planned investments,  which DHS has estimated to be many billions of dollars during the next  few years.
section_title: DHS Is Part of International Tests of Next-Generation RPMs and Is Working with States to Gather ASP Data
paragraphs: DHS is participating in international tests of next-generation RPMs,  including commercially available RPMs that have characteristics similar to  ASP, and is working with states to gather data using ASPs at five truck  weigh stations. For the international tests, DNDO is working with its  European Union (EU) counterparts in the Illicit Trafficking Radiation  Assessment Program (ITRAP); this program is currently known as  ITRAP+10 because it is being revisited about 10 years after testing was  originally conducted. According to the test planning documents, the final  report is scheduled for issuance in August 2013. The documents further  indicate that DNDO has the following objectives for the ITRAP+10 tests:
Provide scientific and technical data on radiological and nuclear  detection equipment to policy makers.
Identify the best technologies based on results from repeatable  testing.
Promote harmonization of national and international standards and  guidelines.
Improve international exchange of information.
Provide manufacturers with feedback on how well their equipment  performed against standards.
Promote new research and development efforts.
The commercially available next-generation RPMs that DNDO and the  EU are testing come from a variety of vendors and are being tested along  with several other categories of radiation detection equipment. This  testing is being performed to national and international standards for  radiation detection set by the American National Standards Institute  (ANSI) and the International Electrotechnical Commission (IEC).  DNDO officials told us these tests are not part of any planned acquisition.  According to the test plan, Oak Ridge National Laboratory (ORNL) will  perform ITRAP+10 tests for DNDO on these next-generation RPMs. The  primary planning documents for the testing of the next-generation RPMs  were submitted by the National Institute of Standards and Technology  and were approved by DNDO and the European Commission’s Joint  Research Centre. These documents indicate that, in addition to testing  to the standards, ORNL will test to determine the outer limits of what the  next-generation commercial RPMs can detect.
In addition to participating in the ITRAP+10 tests, DNDO is working with  state agencies to gather data using five existing ASPs and installed at  state-operated truck weigh stations in five states. According to a DNDO  official, data collected will include information on the trucks and cargo  passing through the ASPs, as well as any radiation detected by ASP.  According to a DNDO official, data collected by the five ASPs will be sent  to DNDO’s Joint Analysis Center Collaborative Information System, which  informs DNDO about nuclear detection and coordinates responses to  nuclear detection alarms. This official told us the data will be analyzed to  determine the types of cargo transported and any trends that may be  apparent.
section_title: Conclusions
paragraphs: To its credit, DHS has cancelled the ASP program, and DNDO conducted  a lessons learned review about the cancelled program and prepared a  lessons learned report. However, DNDO prepared the lessons learned  report and disseminated it 2 years after the final ASP testing, and about a  year after the Secretary of Homeland Security notified Congress of her  decision to cancel the program. Moreover, the report was not prepared  until there was a specific directive from the DHS Under Secretary for  Management. For cancelled programs, DHS’s acquisition guidance calls  for a lessons learned review immediately after cancellation of a program,  preparation of a lessons learned report, and dissemination of such a  report. However, the acquisition guidance does not constitute an  institutional requirement, which the experts we consulted said it should  be. Moreover, DHS does not have documented processes for component  agencies to follow in (1) conducting timely lessons learned reviews, (2)  preparing lessons learned reports, or (3) disseminating lessons learned  reports throughout the department, instead relying on such agencies to  self-monitor. The amount of time between the final field validation testing  of the ASP, the program’s cancellation, and the conducting of a lessons  learned review and dissemination of a lessons learned report suggests  that timely identification of lessons learned was not a DHS priority. While  our review focused on the ASP program, the inability of PARM to identify  any lessons learned reports from cancelled DHS programs suggests that  the problem is broader than this one program. PARM is developing plans  for disseminating future lessons learned reports department-wide.  However, until DHS makes lessons learned reviews an institutional  requirement, puts processes in place to ensure component agencies  conduct such reviews and prepare lessons learned reports, and  implements plans for disseminating the reports, DHS may be missing  opportunities to improve its chances of success for billions of dollars in  future acquisitions.
section_title: Recommendations for Executive Action
paragraphs: To increase the probability of success for future acquisition programs, we  recommend that the Secretary of Homeland Security take the following  four actions for cancelled acquisition programs:  Make lessons learned reviews an institutional requirement, such as  through an agency directive or order or other appropriate means.
Put documented processes in place to ensure that component agencies  conduct timely lessons learned reviews, and prepare and submit lessons learned reports.
Complete and implement plans to disseminate lessons learned reports  throughout the department.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to DHS for review and comment. In its  written response, reproduced in appendix III, DHS agreed with all four of  our recommendations. DHS asked us to consider two of the  recommendations resolved and closed on the basis of existing guidance  and processes that were discussed in the body of this report. DHS  outlined steps that it intends to take to address the other two  recommendations. We are encouraged by some of the actions planned.  However, we do not consider two of the recommendations resolved and  closed because, as discussed below, we believe that DHS should take  steps to address these recommendations.
In DHS’s response to our recommendation to make lessons learned  reviews an institutional requirement, DHS commented that it has  institutionalized this requirement through existing departmental  acquisitions guidance that was discussed in the body of this report.  However, as discussed in this report and confirmed by PARM officials  during the course of our review, this guidance does not constitute a  requirement. DHS’s comments also refer to additional guidance called the  Department of Homeland Security (DHS) Capital Planning and  Investment Control (CPIC) Guide. However, officials from both PARM and  the DHS Office of the Chief Information Officer told us that CPIC applies  only to information technology programs, not to other programs such as  ASP. In our view, the existing guidance referenced in DHS’s comments  contributed to the confusion and uncertainty we discussed in the body of  this report, including confusion about whether lessons learned were  required for cancelled programs. Accordingly, we continue to believe that  DHS needs to take action to address the problem.
Similarly, in commenting on our recommendation to put processes in  place to ensure that component agencies conduct timely lessons learned  reviews, DHS stated that our recommendation is addressed by its existing  acquisition oversight process, as evidenced by the completion of a  lessons learned report for ASP. However, according to DNDO and CBP’s  lessons learned report, the Secretary of DHS cancelled the program in  October 2011, but the lessons learned report for this program was not  issued until more than a year later. Furthermore, as noted in this report,  the ASP lessons learned report is the only example PARM officials could  identify of such a report having been completed. In our view, relying on  this existing process seems unlikely to ensure that component agencies  conduct timely lessons learned reviews.
With regard to our final two recommendations—to put documented  processes in place to ensure that component agencies prepare and  submit lessons learned reports, and to complete and implement plans to  disseminate lessons learned reports throughout the department—DHS  stated that it is taking several actions. Specifically, DHS is revising and  clarifying its acquisitions guidance with respect to lessons learned; it  expects to complete these revisions by June 2014. In its comments, DHS  stated that it has developed an online forum and library for DHS  acquisition professionals that includes an area dedicated to lessons  learned and best practices. Furthermore, DHS has designated a lead  Center of Excellence and established a time frame of September 2013 for  collecting and posting information on lessons learned for cancelled  programs. In our view, as DHS completes more lessons learned reports,  an online forum and library could become a useful means of  disseminating these reports throughout the department.
DHS also provided technical comments that we incorporated in the report,  as appropriate.
As agreed with your offices, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date. At that time, we will send copies to the Secretary of  Homeland Security, the Director of DNDO, the Commissioner of CBP, the  Executive Director of PARM, the appropriate congressional committees,  and other interested parties. In addition, the report will be available at no  charge on the GAO website at http://www.gao.gov.
If you or your staff members have any questions about this report, please  contact David C. Trimble at (202) 512-3841 or trimbled@gao.gov or Dr.  Timothy M. Persons at (202) 512-6412 or personst@gao.gov. Contact  points for our Offices of Congressional Relations and Public Affairs may  be found on the last page of this report. GAO staff who made key  contributions to this report are listed in appendix IV.
section_title: Appendix I: Experts Identified by the National Academies Who Described Leading Practices for Reviewing Cancelled Acquisition Programs
paragraphs: At our request, the National Academies identified experts who could  highlight leading practices for reviewing cancelled acquisition programs.  These experts were identified on the basis on their knowledge of leading  practices in large-scale engineering and acquisition programs. The  National Academies provided us a list of 10 experts, 7 of whom were  available to participate in this effort. Six of the 7 experts who were  available were members of the National Academy of Engineering. We  conducted a group telephone conference with 6 of the experts. We  drafted conclusions from the conference and circulated them to all 7 of  the experts for their review and comment. With one exception, the experts  reached consensus on all of the observations. One expert felt that  lessons learned reports were so important that acquisitions executives  should have less authority to tailor or waive the requirements for such  reports.
section_title: Appendix II: DNDO-CBP ASP Lessons Learned Report
subsections: section_title: The following DNDO-CBP ASP lessons learned report was redacted on December 4, 2012, from the November 5, 2012, For Official Use Only version by DNDO.
section_title: Appendix IV: GAO Contacts and Staff Acknowledgments
subsections: section_title: GAO Contacts
section_title: Staff Acknowledgments
paragraphs: In addition to the individuals named above, Ned Woodward, Assistant  Director; Gene Aloise; Cheryl R. Arvidson; Antoinette C. Capaccio;  Frederick K. Childers; R. Scott Fletcher; Cindy K. Gilbert; David C.  Maurer; Cynthia Norris; Katrina E. Pekar-Carpenter; Kiki Theodoropoulos;  and Nathan A. Tranquilli made key contributions to this report.
section_title: Related GAO Products
paragraphs: Combating Nuclear Smuggling: DHS has Developed Plans for Its Global  Nuclear Detection Architecture, but Challenges Remain in Deploying  Equipment. GAO-12-941T. Washington, D.C.: July 26, 2012.
Combating Nuclear Smuggling: DHS Has Made Some Progress but Not  Yet Completed a Strategic Plan for Its Global Nuclear Detection Efforts or  Closed Identified Gaps. GAO-10-883T. Washington, D.C.: June 30, 2010.
Combating Nuclear Smuggling: Recent Testing Raises Issues About the  Potential Effectiveness of Advanced Radiation Detection Portal Monitors.  GAO-10-252T. Washington, D.C.: November 17, 2009.
Combating Nuclear Smuggling: Lessons Learned from DHS Testing of  Advanced Radiation Detection Portal Monitors. GAO-09-804T.  Washington, D.C.: June 25, 2009.
Combating Nuclear Smuggling: DHS Improved Testing of Advanced  Radiation Detection Portal Monitors, but Preliminary Results Show Limits  of the New Technology. GAO-09-655. Washington, D.C.: May 21, 2009.
Nuclear Detection: Domestic Nuclear Detection Office Should Improve  Planning to Better Address Gaps and Vulnerabilities. GAO-09-257.  Washington, D.C.: January 29, 2009.
Department of Homeland Security: Billions Invested in Major Programs  Lack Appropriate Oversight. GAO-09-29. Washington, D.C.: November  18, 2008.
Combating Nuclear Smuggling: DHS’s Phase 3 Test Report on Advanced  Portal Monitors Does Not Fully Disclose the Limitations of the Test  Results. GAO-08-979. Washington, D.C.: September 30, 2008.
Combating Nuclear Smuggling: DHS Needs to Consider the Full Costs  and Complete All Tests Prior to Making a Decision on Whether to  Purchase Advanced Portal Monitors. GAO-08-1178T. Washington, D.C.:  September 25, 2008.
Combating Nuclear Smuggling: DHS’s Program to Procure and Deploy  Advanced Radiation Detection Portal Monitors Is Likely to Exceed the  Department’s Previous Cost Estimates. GAO-08-1108R. Washington,  D.C.: September 22, 2008.
Combating Nuclear Smuggling: DHS’s Decision to Procure and Deploy  the Next Generation of Radiation Detection Equipment Is Not Supported  by Its Cost-Benefit Analysis. GAO-07-581T. Washington, D.C.: March 14,  2007.
Combating Nuclear Smuggling: DNDO Has Not Yet Collected Most of the  National Laboratories’ Test Results on Radiation Portal Monitors in  Support of DNDO’s Testing and Development Program. GAO-07-347R.  Washington, D. C.: March 9, 2007.
Combating Nuclear Smuggling: DHS’s Cost-Benefit Analysis to Support  the Purchase of New Radiation Detection Portal Monitors Was Not Based  on Available Performance Data and Did Not Fully Evaluate All the  Monitors’ Costs and Benefits. GAO-07-133R. Washington, D.C.: October  17, 2006.
Combating Nuclear Smuggling: Additional Actions Needed to Ensure  Adequate Testing of Next Generation Radiation Detection Equipment.  GAO-07-1247T. Washington, D.C.: September 18, 2007.
Combating Nuclear Smuggling: DHS Has Made Progress Deploying  Radiation Detection Equipment at U.S. Ports-of-Entry, but Concerns  Remain. GAO-06-389. Washington, D.C.: March 22, 2006.