COMBATING 

NUCLEAR 

SMUGGLING

Lessons Learned from 

Cancelled Radiation 

Portal Monitor 

Program Could Help 

Future Acquisitions

Report to Congressional Requesters

May 2013

GAO-13-256

United States Government Accountability Office

GAO
United States Government Accountability Office

Highlights of GAO-13-256, a report to 

congressional requesters

May 2013

COMBATING NUCLEAR SMUGGLING 

Lessons Learned from Cancelled Radiation Portal 

Monitor Program Could Help Future Acquisitions

Why GAO Did This Study

Preventing terrorists from smuggling 

radiological or nuclear material into the 

United States to carry out an attack is 

a national priority. DHS’s DNDO 

develops and deploys radiation 

detection equipment to assist other 

federal agencies, such as CBP, in 

intercepting illicit radiological or nuclear 

materials that could be used to make a 

radiological dispersive device (dirty 

bomb) or a crude nuclear bomb. CBP 

uses RPMs at nearly all land border 

crossings and seaports to detect 

radiation in trucks and cargo. DHS

recently cancelled acquisition of ASPs, 

which was originally envisioned as 

costing from $2 billion to $3 billion. 

GAO was asked to provide updated 

information on the ASP program. This 

report examines, among other things, 

(1) the results of ASP testing 

conducted in 2009 and 2010 that led to 

DHS’s decision to cancel the ASP 

program and (2) the benefits of lessons 

learned reviews and how DHS 

captures any lessons learned when 

programs are cancelled. GAO 

reviewed testing and acquisition 

documents and interviewed key 

agency officials, as well as seven 

experts the National Academies 

identified for their knowledge of leading 

practices in large-scale engineering 

and acquisition programs.

What GAO Recommends

DHS should require lessons learned 

reviews and develop processes to 

ensure such reviews are done in a 

timely manner and the results 

disseminated throughout the 

department. DHS agreed with all of

GAO’s recommendations and has 

planned and taken some actions to 

address them.

What GAO Found

The advanced spectroscopic portal monitor (ASP)—a next-generation radiation 

portal monitor (RPM) for screening trucks and cargo containers—did not pass 

field validation tests conducted in 2009 and 2010. The Department of Homeland 

Security’s (DHS) Domestic Nuclear Detection Office (DNDO) intended to replace 

many currently deployed RPMs and handheld radiation detectors used by U.S. 

Customs and Border Protection (CBP) with ASPs. However, in the tests, ASP did 

not meet key requirements to detect radiation and identify its source. For 

example, ASP triggered too many false alarms from benign, naturally occurring 

radioactive material in common items such as kitty litter and granite, and it 

sometimes would not turn on or continue operating long enough to complete a 

day of testing. In addition, GAO’s review identified analytical weaknesses related 

to the testing and program cancellation, including inconsistencies in DNDO’s 

analysis of the settings used for testing the ASP. The final field validation test 

was conducted in November 2010, and the Secretary of Homeland Security 

notified Congress of her decision to cancel the program in October 2011.

Conducting lessons learned reviews when programs are cancelled benefits 

organizations by identifying things that worked well and did not work well in order 

to improve future acquisitions programs, according to experts GAO consulted. 

However, DHS does not have processes in place to ensure such reviews are 

conducted or that the results are disseminated. Experts identified by the National 

Academies told GAO that lessons learned reviews help identify reasons why 

programs were cancelled. The experts also said lessons learned reviews should 

be required and conducted promptly, and the results should be disseminated. At 

the direction of DHS management, DNDO reviewed the ASP program and 

submitted and disseminated a lessons learned report in November 2012. (See 

timeline below.) This report cited 32 lessons learned including having program 

officials work closely with end users to ensure equipment meets operational 

requirements. DHS guidance calls for lessons learned reviews immediately after 

programs are cancelled and states that the lessons learned are to be shared 

throughout the department, but this guidance is not a requirement. Before DHS’s 

directive, there was confusion about whether a lessons learned review was 

needed for the ASP program, and DNDO officials did not intend to conduct such 

a review. Moreover, DHS officials were unable to provide examples of previous 

lessons learned reports from other cancelled programs. DHS officials also said 

they have no process for disseminating such reports but are planning one.

Events from Final Test of ASP to Lessons Learned Report

View GAO-13-256. For more information, 

contact David C. Trimble at (202) 512-3841 or 

trimbled@gao.gov or Dr. Timothy M. Persons 

at (202) 512-6412 or personst@gao.gov. 
Page i GAO-13-256 Combating Nuclear Smuggling

Letter 1 

Background 6

Test Results Show That ASP Did Not Meet Criteria to Pass Field 

Validation Testing, Leading DHS to Cancel the Program 10

Lessons Learned Reviews Improve Future Acquisition Efforts, but 

DHS Does Not Have Processes Ensuring Such Reviews 13

DHS Is Part of International Tests of Next-Generation RPMs and Is 

Working with States to Gather ASP Data 18

Conclusions 20

Recommendations for Executive Action 20

Agency Comments and Our Evaluation 21

Appendix I Experts Identified by the National Academies Who 

Described Leading Practices for Reviewing Cancelled 

Acquisition Programs 24

Appendix II DNDO-CBP ASP Lessons Learned Report 25

Appendix III Comments from the Department of Homeland Security 39

Appendix IV GAO Contacts and Staff Acknowledgments 42

Related GAO Products 43

Tables

Table 1: Criteria for Demonstrating a Significant Increase in 

Operational Effectiveness 9

Table 2: Summary of Results for Field Validation Testing 12

Contents 
Page ii GAO-13-256 Combating Nuclear Smuggling

Figures

Figure 1: Trucks Passing though Radiation Portal Monitors 2

Figure 2: CBP Officer Screening a Truck with a Handheld Detector 

That Can Identify Sources of Radiation 3

Figure 3: Timeline of DHS Studies and Tests of Next-Generation 

RPMs, Including ASP 7

Abbreviations

ANSI American National Standards Institute

ASP advanced spectroscopic portal monitor

CBP Customs and Border Protection

DHS Department of Homeland Security

DNDO Domestic Nuclear Detection Office

DOE Department of Energy

EU European Union

IEC International Electrotechnical Commission

ITRAP Illicit Trafficking Radiation Assessment Program

ITRAP+10 ITRAP revisited after about 10 years

ORNL Oak Ridge National Laboratory

PARM Office of Program Accountability and Risk Management

PNNL Pacific Northwest National Laboratory

This is a work of the U.S. government and is not subject to copyright protection in the 

United States. The published product may be reproduced and distributed in its entirety 

without further permission from GAO. However, because this work may contain 

copyrighted images or other material, permission from the copyright holder may be 

necessary if you wish to reproduce this material separately.
Page 1 GAO-13-256 Combating Nuclear Smuggling

United States Government Accountability Office

Washington, DC 20548

May 13, 2013

The Honorable Dan Maffei

Ranking Member

Subcommittee on Oversight

Committee on Science, Space, and Technology

House of Representatives

The Honorable Donna F. Edwards

Ranking Member

Subcommittee on Space

Committee on Science, Space, and Technology

House of Representatives

Preventing terrorists from smuggling radiological or nuclear material into 

the United States to carry out an attack is a national priority. To 

implement federal policy to protect the nation against radiological or 

nuclear attacks, among other purposes, the Department of Homeland 

Security’s (DHS) Domestic Nuclear Detection Office (DNDO) was 

established in 2005.1 DNDO is tasked with coordinating development of a 

global nuclear detection architecture and is responsible for implementing 

the domestic portion of the architecture.2 It also develops, acquires, and 

deploys radiation detection equipment to support the efforts of other 

federal agencies. For example, DHS’s U. S. Customs and Border 

Protection (CBP) uses radiation detection equipment at U.S. ports of 

entry to screen cargo containers and trucks for illicit radiological 

materials, including material that could be used in radiological dispersive 

devices (dirty bombs) and special nuclear material that could be used to 

make a nuclear weapon.3

 

1

National Security Presidential Directive 43 / Homeland Security Presidential Directive 14, 

Domestic Nuclear Detection, April 15, 2005. DNDO was established in statute by the 

Security and Accountability for Every Port Act of 2006 (SAFE Port) Act, Pub. L. No. 109-

347, § 501, 120 Stat. 1884, 1932 (codified as amended at 6 U.S.C. § 591).

2

The global nuclear detection architecture is an integrated system of radiation detection

equipment and interdiction activities to combat nuclear smuggling in foreign countries, at 

the U.S. border, and inside the United States.

3

Special nuclear material includes plutonium, uranium enriched in the isotope 233 or in 

the isotope 235; and any material artificially enriched by any of the foregoing. 42 U.S.C. § 

2014 (2006).

 
 

Page 2 GAO-13-256 Combating Nuclear Smuggling

Potential pathways for illicit radiological materials to enter the United 

States include being smuggled in trucks at border crossings or in cargo 

on ships. According to DHS officials, nearly all trucks at land border 

crossings and containerized cargo at seaports are screened for 

radiological materials. To screen cargo at these locations, CBP uses 

radiation portal monitors (RPM)—large stationary radiation detectors 

through which trucks and cargo containers pass. (See fig. 1.)

Figure 1: Trucks Passing though Radiation Portal Monitors

Note: Dashed frame contains one radiation portal monitor.

According to DNDO the current generation of RPMs has limitations in that 

it can detect but cannot identify sources of radiation. Because of this 

inability, the RPMs’ radiation alarms can be triggered by benign, naturally 
 

Page 3 GAO-13-256 Combating Nuclear Smuggling

occurring radioactive material present in common items such as kitty litter 

and granite. Cargo that has triggered an alarm during the initial 

(“primary”) screening is sent for additional inspection (i.e., “secondary” 

screening), first by another RPM to confirm the alarm and then by a CBP 

officer using a handheld radiation detector—about the size of a shoe 

box—that can identify the source of the radiation (see fig. 2). 

Figure 2: CBP Officer Screening a Truck with a Handheld Detector That Can Identify 

Sources of Radiation
 

Page 4 GAO-13-256 Combating Nuclear Smuggling

One way to reduce the rate of alarms that are triggered by benign 

materials—and thereby reduce the number of unnecessary secondary 

screenings—would be to use next-generation RPMs that can both detect 

and identify radiation sources. In 2005, DNDO began working with CBP 

on a program to develop and test a type of next-generation RPM called 

the advanced spectroscopic portal monitor (ASP), which was designed to 

both detect radiation and identify the source as benign, suspect, or a 

threat. The initial concept of the program was to develop, procure, and 

deploy enough ASPs to replace many of CBP’s currently deployed RPMs 

and handheld detectors at a cost of $2 billion to $3 billion, according to 

DNDO.4

Throughout the ASP program, DNDO and CBP faced many challenges, 

on which we have reported and testified to Congress several times.5 Our 

last in-depth review of the ASP program was in May 2009, when we 

reported that the ASP had mixed performance during testing to determine 

whether it could successfully be used for primary and secondary 

screening.6 Challenges in the ASP program were also identified by the 

National Research Council of the National Academies.7 In 2009, the 

National Research Council recommended in an interim report that DNDO 

use a better approach to testing, evaluation, cost-benefit assessment, 

and deployment of ASPs.8 In 2010, the National Research Council found 

in its final report that, among other things, DNDO’s 2008 testing had 

shortcomings that impaired the ability to draw conclusions about ASP’s 

 

4

DNDO, letter from the Acting Director of DNDO to the Chairman of the Senate 

Committee on Homeland Security and Governmental Affairs, February 24, 2010.

5

For a listing of past reports and testimonies, see Related Products at the end of this 

report.

6

GAO, Combating Nuclear Smuggling: DHS Improved Testing of Advanced Radiation 

Detection Portal Monitors, but Preliminary Results Show Limits of the New Technology, 

GAO-09-655 (Washington, D.C.: May 21, 2009).

7

The National Academies comprise four organizations: the National Academy of 

Engineering, National Academy of Sciences, Institute of Medicine, and National Research 

Council.

8

National Research Council, Evaluating Testing, Costs, and Benefits of Advanced 

Spectroscopic Portals for Screening Cargo at Ports of Entry – Interim Report (Washington, 

D.C.: 2009).
 

Page 5 GAO-13-256 Combating Nuclear Smuggling

likely performance, and DNDO’s draft cost-benefit analysis needed 

substantial improvement to support decision making.9

In July 2011, after several years of testing, the DNDO Director testified 

that the Secretary of Homeland Security had directed DNDO and CBP to 

“end the ASP program as originally conceived” and to instead use 

existing ASPs to help CBP officers gain operational familiarity with ASP 

and to gather data to support a future acquisition program. In October 

2011, the Secretary of Homeland Security wrote to key congressional 

committees informing them of her decision to cancel the ASP program.10

In July 2012, the Under Secretary for Management stated that, in 

accordance with the Secretary’s letter, the program was considered 

cancelled. Once cancelled, programs such as ASP are subject to 

provisions of DHS’s acquisition guidance regarding review of cancelled 

programs and dissemination of the resulting lessons learned.

In this context, you asked us to provide updated information on the ASP 

program. Our objectives were to determine: (1) the results of ASP testing 

conducted in 2009 and 2010 that led to DHS’s decision to cancel the ASP 

program; (2) the benefits of lessons learned reviews and how DHS 

captures any lessons learned when programs are cancelled; and (3) what 

additional testing, if any, of next-generation RPMs, including ASP, DHS is 

conducting or planning.

To determine the results of the ASP testing conducted in 2009 and 2010 

that led to the decision to cancel the ASP program, we reviewed ASP 

testing documents and briefing materials from DNDO, CBP, the 

Department of Energy’s (DOE) Pacific Northwest National Laboratory, 

and Johns Hopkins University’s Applied Physics Laboratory and 

interviewed officials from DNDO and CBP. To determine the benefits of 

lessons learned reviews and how DHS captures any lessons learned 

when programs are cancelled, we analyzed acquisition policy documents 

from DHS and interviewed officials from DNDO, DHS’s Office of Program 

Accountability and Risk Management (PARM), and DHS’s Enterprise 

 

9

National Research Council, Evaluating Testing, Costs, and Benefits of Advanced 

Spectroscopic Portals – Final Report, (Washington, D.C.: 2010).

10DHS, letter from the Secretary of Homeland Security to the Chairman and Ranking 

Members of the House Committee on Homeland Security, the Senate Committee on 

Homeland Security and Governmental Affairs, and the House and Senate Appropriations 

Subcommittees on Homeland Security, October 3, 2011. 
 

Page 6 GAO-13-256 Combating Nuclear Smuggling

Business Management Office. In addition, we interviewed seven experts 

identified by the National Academies to highlight leading practices for 

reviewing cancelled acquisition programs; these experts were identified 

on the basis on their knowledge of leading practices in large-scale 

engineering and acquisition programs. (See app. I for a list of the experts 

we interviewed.) To determine what additional testing of next-generation 

RPMs, including ASP, DHS is conducting or planning, we reviewed 

documents from DHS, DNDO, the National Institute of Standards and 

Technology, the American National Standards Institute, and the 

International Electrotechnical Commission, and we witnessed some of the 

testing. We also interviewed officials from DNDO and DOE’s Oak Ridge 

National Laboratory.

We conducted this performance audit from October 2011 to May 2013 in 

accordance with generally accepted government auditing standards. 

Those standards require that we plan and perform the audit to obtain 

sufficient, appropriate evidence to provide a reasonable basis for our 

findings and conclusions based on our audit objectives. We believe that 

the evidence obtained provides a reasonable basis for our findings and 

conclusions based on our audit objectives.

DHS began testing next-generation RPMs with a preliminary small-scale 

study performed in July 2004 by DOE’s Pacific Northwest National 

Laboratory (PNNL). (See fig. 3.) Specifically, PNNL performed smallscale, side-by-side comparisons of next-generation RPMs and the RPMs 

used by CBP. This study found that in some situations, the performance 

of the next-generation RPMs was better than that of CBP’s RPMs, and 

that in other situations, the performance of the two was equal. In a followup, small-scale study conducted for DHS in July 2005, PNNL found that 

the next-generation RPMs performed no better than CBP’s currently 

deployed RPMs but did produce fewer alarms for benign materials.

Background 
 

Page 7 GAO-13-256 Combating Nuclear Smuggling

Figure 3: Timeline of DHS Studies and Tests of Next-Generation RPMs, Including ASP

In October 2005, DNDO conducted a number of tests on next-generation 

RPMs that DHS had commissioned from different manufacturers and 

used the test results to help evaluate proposals for the ASP program. In 

July 2006, DNDO awarded contracts for development and production of 

ASPs. Among other things, according to DNDO officials, the design 

specifications for ASP stated that it should operate on trucks traveling at 5 

miles per hour during primary screening and 2 miles per hour during 

secondary screening. DNDO began testing these ASPs at DOE’s Nevada 

National Security Site in February 2007.11 As we have previously 

 

11The Nevada National Security Site was formerly known as the Nevada Test Site.
 

Page 8 GAO-13-256 Combating Nuclear Smuggling

reported, we identified a number of weaknesses in the methods used for 

these tests that impaired the significance and validity of the test results.12

Concerned about the performance and cost of ASP, Congress required 

the Secretary of Homeland Security to certify that ASP would provide a 

“significant increase in operational effectiveness” before DNDO obligated 

funds for full-scale procurement of ASPs.13 In response, DNDO, CBP, and 

DHS jointly issued criteria in July 2008 for determining whether ASP 

provided a significant increase in operational effectiveness compared with 

existing equipment. The criteria generally compare ASP with the RPMs 

used by CBP under CBP’s standard operating procedure. Specifically, as 

shown in table 1, there were four criteria for primary screening and two 

criteria for secondary screening.14 In addition to these overarching criteria, 

each phase of testing also had criteria that needed to be met to complete 

that phase.

 

12GAO, Combating Nuclear Smuggling: Additional Actions Needed to Ensure Adequate 

Testing of Next Generation Radiation Detection Equipment, GAO-07-1247T (Washington, 

D. C.: Sept. 18, 2007); and Combating Nuclear Smuggling: DHS’s Phase 3 Test Report on 

Advanced Portal Monitors Does Not Fully Disclose the Limitations of the Test Results, 

GAO-08-979 (Washington, D.C.: Sept. 30, 2008).

13Consolidated Appropriations Act, 2008, Pub. L. No. 110-161, 121 Stat. 1844, 2069 

(2007); Consolidated Security, Disaster Assistance, and Continuing Appropriations Act, 

2009, Pub. L. No. 110-329, 121 Stat. 3574, 3679 (2008); Department of Homeland 

Security Appropriations Act, 2010, Pub. L. No. 111-83, 123 Stat. 2142, 2167 (2009). The 

requirement was extended by continuing resolutions in fiscal year 2011.

14We reviewed these criteria in 2009 and found that they would require a marginal 

improvement in the ability to detect threats and a large reduction in alarms for benign 

materials. See GAO-09-655 (Washington, D. C.: May 21, 2009).
 

Page 9 GAO-13-256 Combating Nuclear Smuggling

Table 1: Criteria for Demonstrating a Significant Increase in Operational Effectiveness 

Primary screening criteria Secondary screening criteria

1. When special nuclear material is present in cargo without 

naturally occurring radioactive material, ASP’s probability of 

detecting the material must be equal to or greater than that of 

the RPMs used by CBP.

2. When special nuclear material is present in cargo along with 

naturally occurring radioactive material, the ASP must 

increase the probability of detecting and identifying the 

material compared with CBP’s currently deployed RPMs and 

handheld detectors.

3. When medical or industrial radiological sources are present 

in cargo, ASP’s probability of detecting the sources must be 

equal to or greater than that of CBP’s currently deployed 

RPMs.

4. When the only radiological source present in the cargo is 

naturally occurring radioactive material, the ASP must refer 

at least 80 percent fewer trucks for secondary screening than 

CBP’s currently deployed RPMs.

1. When compared with CBP’s currently deployed handheld 

detectors, ASP must reduce, by at least a factor of 2, the 

probability that special nuclear material is misidentified as 

naturally occurring radioactive material, a medical or industrial 

radiological source, an unknown radiological source, or no 

radiological source at all.

2. When compared with CBP’s currently deployed handheld 

detectors, ASP must reduce the average time required to 

correctly release trucks from secondary screening.

Source: GAO summary of DHS documentation.

The final ASP testing campaign began in July 2008 and was concluded in 

November 2010. DNDO started the campaign with performance testing at 

DOE’s Nevada National Security Site. The performance tests determined 

the probability of detection and identification of radiation sources, 

including special nuclear material that could be used to make a nuclear 

weapon. CBP then began field validation testing in 2009. Field validation 

tests are the first step in determining the suitability of fully operational 

equipment in the actual environment where the equipment is intended to 

be used. The ASP field validation tests were performed at land border 

crossings in Detroit, Michigan, and Laredo, Texas, and at seaports in 

Long Beach, California, and in New York City, New York. CBP made 

three attempts to complete this testing—in January and February 2009, 

July and August 2009, and October and November 2010.

This testing, which we reviewed for this report, evaluated the compatibility 

of ASP with CBP’s interdiction system on the basis of 10 additional 

criteria CBP had specified for completion of this phase of testing.15

Among other things, these criteria included whether, under normal field 

operating conditions, ASP improved detection to the required degree 

 

15These criteria were contained in CBP’s field validation test plan.
 

Page 10 GAO-13-256 Combating Nuclear Smuggling

compared with the currently deployed CBP equipment. To pass the field 

validation testing phase, ASP had to meet all 10 criteria. In addition, 

DNDO’s ASP program manager specified that ASP could not have any 

major unresolved issues that might render it ineffective or unsuitable for 

use by CBP.16

Because of unsatisfactory test results, ASP did not pass field validation 

testing, which led DHS to cancel the program. Specifically, in CBP’s first 

and second attempts to pass field validation testing in 2009, ASP did not 

meet 1 of the 4 criteria for demonstrating a significant increase in 

operational effectiveness during primary screening–that is, it did not refer 

at least 80 percent fewer trucks for secondary screening than CBP’s 

currently deployed RPMs.17 When CBP began the first field validation test 

in late January 2009, CBP found ASP produced a higher number of 

referrals to secondary screening than CBP’s currently deployed RPMs. 

CBP suspended the test about 2 weeks later in mid-February; CBP 

officials reported that the problem with referrals required significant 

corrective actions before testing could resume. DNDO, working with 

Johns Hopkins University’s Applied Physics Laboratory, determined that 

the problem could be addressed by revising ASP’s software to raise the 

thresholds for triggering alarms.

CBP conducted a second field validation test of ASP in July and August 

2009 using the revised ASP software. CBP and DNDO documents 

indicate that, with the revised software, ASP was able to reduce referrals 

to secondary screening by about 70 percent but not by the 80 percent 

required to demonstrate a significant increase in operational 

effectiveness. Significantly, according to these documents, the majority of 

the ASP referrals to secondary screening were for alarms that falsely 

indicated the presence of special nuclear material that could be used to 

make a nuclear weapon. CBP officials told us these types of false alarms 

were very disruptive in a port environment because they caused CBP 

officers to take enhanced security precautions—specifically, CBP officers 

 

16Department of Homeland Security, DNDO, Issue Management and Adjudication 

Summary (ASP 2010 field validation exit briefing presented by Paul Burrowes, ASP 

Program Manager, at the meeting of the ASP Governance Board, Washington, D.C.: 

January 2011).

17This criterion applies when only naturally occurring radioactive material is present, which 

was the case for the first and second field validation tests. 

Test Results Show 

That ASP Did Not 

Meet Criteria to Pass 

Field Validation 

Testing, Leading DHS 

to Cancel the Program
 

Page 11 GAO-13-256 Combating Nuclear Smuggling

had to conduct a thorough inspection to ensure no special nuclear 

material was present before permitting the cargo to enter the country. 

Moreover, repeated false alarms for such nuclear materials could cause 

CBP officers to be skeptical about future alarms.

In the course of the second field validation test, ASP also experienced a 

“critical failure,” which caused it to shut down. Importantly, during this 

critical failure, ASP did not emit a signal to alert the CBP officer that it was 

no longer screening cargo. Had this failure not occurred in a controlled 

testing environment, in which CBP was using its RPMs in tandem with 

ASP, the CBP officer would have permitted cargo to enter the country 

unscreened for radiological material. Testing is performed so that issues 

like this can be discovered and fixed prior to full-scale production and 

deployment.

Following the second field validation test, DNDO’s Acting Director stated 

that the ASP program would be scaled back to secondary screening 

only.18 According to the Acting Director, DNDO’s decision was based on 

ASP’s performance and cost. Specifically, the Acting Director stated that 

DHS decided to not pursue ASP for primary screening because ASP did 

not meet DHS’s criteria for a significant increase in operational 

effectiveness. However, the Acting Director stated that results to date had 

shown that ASP met secondary screening criteria by wide margins. The 

Acting Director also stated that the cost to replace current RPMs with 

ASP for secondary screening was relatively low—approximately $350 

million—compared with the original total program cost of $2 billion to $3 

billion to use ASP for primary and secondary screening. This is because 

the original program called for many more ASPs to be deployed in CBP 

primary screening than secondary screening.

In November 2010, ASP could not pass the third and final field validation 

test for use in secondary screening because ASP did not meet 6 of the 10 

criteria needed to complete the field validation testing phase, according to 

our review of CBP test documents. According to these documents, 2 of 

the 6 criteria that ASP did not meet involved issues that would 

significantly affect operations unless they could be mitigated or resolved. 

ASP did not meet these criteria in part because of its inability to turn on at 

 

18DNDO, letter from the Acting Director of DNDO to the Chairman of the Senate 

Committee on Homeland Security and Governmental Affairs, February 24, 2010.
 

Page 12 GAO-13-256 Combating Nuclear Smuggling

the beginning of each day and to continue operating long enough to 

complete a full day of testing. In addition, when ASP was operating, it 

produced an excessive number of alarms that incorrectly indicated 

naturally occurring radioactive material in the cargo might be masking 

special nuclear material. In some cases, testing of the ASP had to be 

suspended because the high number of these alarms was backing up 

traffic for secondary screening, delaying the flow of commerce. According 

to the CBP documents, the remaining 4 of the 6 criteria that ASP did not 

meet involved issues that could affect or delay port operations, although 

not as significantly. For example, ASP did not meet one of these criteria 

because of difficulty in integrating ASP into a CBP information network.

In addition to the 6 criteria that the ASP did not meet, DNDO test 

documents identified several major unresolved issues that would have 

needed to be resolved for successful completion of field validation testing. 

In explaining these issues, DNDO and CBP officials told us that, for 

example, at certain times of the day, direct sunlight impaired the ability of 

the ASP to operate properly. See table 2 for a summary of the results for 

the three field validation tests.

Table 2: Summary of Results for Field Validation Testing

Type of test Date of testing Summary of results

First field validation test:

primary and secondary 

screening

January and February

2009

ASP had more referrals from primary screening to secondary screening 

than CBP’s currently deployed RPMs. The requirement for a significant 

increase in operation effectiveness was to have 80 percent fewer referrals 

to secondary screening.

Second field validation test: 

primary and secondary 

screening

July and August

2009

ASP had 70 percent fewer referrals to secondary screening than CBP’s 

currently deployed RPMs instead of the 80 percent fewer required for a 

significant increase in operation effectiveness. Also, the majority of 

referrals were for alarms falsely indicating the presence of special nuclear 

material, and such alarms were very disruptive in the port environment.

Third field validation test: 

secondary screening only

October and November 

2010

ASP did not meet 6 of 10 criteria required to pass the field validation 

testing phase. For example, ASP produced excessive alarms incorrectly 

indicating that naturally occurring radioactive material might be masking 

special nuclear material. At times, testing had to be suspended because 

these alarms were backing up traffic.

Source: GAO analysis of ASP program documents.

In addition to the issues identified during testing, our review identified 

analytical weaknesses before and after the third field validation test. For 

example, DNDO was inconsistent in its analysis of how to properly set up 

the ASP for secondary screening. Specifically, ASPs have separate 

settings (modes) for primary and secondary screening, and before the 

third field validation test, DNDO performed an analysis that indicated ASP 
 

Page 13 GAO-13-256 Combating Nuclear Smuggling

would perform best during secondary screening if the machines were set 

in primary screening mode. However, after the third field validation test, 

DNDO officials analyzed the test results and concluded that ASP would 

have performed better for secondary screening had the machines been 

set in secondary screening mode. Furthermore, the analytical basis for a 

key decision regarding the ASP program is unclear. Specifically, DNDO 

could not provide us with any supporting analysis on a principal factor that 

led to the decision to cancel the program. According to DNDO officials, for 

ASP to work in secondary screening, truck speeds would need to average 

2 miles per hour, as indicated in the design specifications. On the basis of 

these specifications, the ASP Governance Board concluded in February 

2011 that ASP was not operationally suitable for secondary screening 

and should not be deployed because, according to CBP officials, it was 

not possible to control truck speed to an average of 2 miles per hour.19

However, when we asked DNDO officials for the data or analysis 

supporting this conclusion, these officials could not provide such support. 

Moreover, during congressional testimony in July 2012, the Acting 

Director of DNDO said that it is possible to control truck speed to 2 miles 

per hour at state-operated truck weigh stations. Accordingly, it is unclear 

why such speeds cannot be achieved at CBP land border crossings and 

seaports but can be achieved at truck weigh stations.

Conducting lessons learned reviews when programs are cancelled 

benefits organizations by identifying things that worked well and did not 

work well in order to improve future acquisitions programs, according to 

experts we consulted; however, DHS does not have processes in place to 

ensure such reviews are conducted and reports documenting the results 

of the review are disseminated. To determine the key benefits of 

conducting lessons learned reviews, we consulted seven experts in largescale engineering and acquisitions programs who were identified at our 

request by the National Academies. (See app. I for a list of the experts we 

interviewed.) When we asked these experts for their views on lessons 

 

19The ASP Governance Board consisted of officials from DHS Office of Policy, DHS 

Science and Technology Directorate, the Office of the Deputy Secretary, DHS Office of 

Program Accountability and Risk Management (PARM), CBP, and DNDO. It was 

constituted in November 2009 to examine the ASP program. 

Lessons Learned 

Reviews Improve 

Future Acquisition 

Efforts, but DHS Does 

Not Have Processes 

Ensuring Such 

Reviews 
 

Page 14 GAO-13-256 Combating Nuclear Smuggling

learned reviews and reports for cancelled acquisition programs, they 

generally agreed on the following observations:20

• Lessons learned reviews help to determine the reasons why programs 

were cancelled including problems with the technology, management, 

or setting requirements. By identifying problematic practices and 

causes for program failure, lessons learned reports allow an 

organization to improve future efforts.

• Lessons learned reports should be prepared promptly because 

otherwise, knowledgeable personnel may not be available to 

contribute to the reports, important details may not be recalled 

accurately, and dissemination of lessons learned will be delayed.

• Lessons learned reports should not be optional for an organization. A 

requirement for lessons learned reports should be institutionalized—

lessons learned reports should be required for programs administered 

by the organization rather than being an ad hoc requirement. 

However, an appropriate acquisitions executive should have the 

authority to tailor or waive the lessons learned report based on factors 

such as the size or cost of a program, its perceived importance and 

effects, or the perceived causes for failure.

• Lessons learned reports must be submitted to an appropriate 

acquisitions executive to be useful. The acquisitions executive should 

disseminate the lessons learned reports to other appropriate 

acquisition officials. Lessons learned reports should be disseminated 

widely; however, they should be edited to preserve the lessons 

learned while minimizing potential damage to the reputations of 

people and organizations that were involved with the program.

DHS interim acquisition guidance, issued in 2010, also recognizes the 

benefits of timely lessons learned reviews for cancelled programs, 

although as guidance it does not constitute a requirement.21 Similar to the 

 

20With one exception, the experts reached consensus on all of the observations. One 

expert felt that lessons learned reports were so important that acquisitions executives 

should have less authority to tailor or waive the requirements for such reports.

21DHS Systems Engineering Life Cycle Version 2.0 (INTERIM), September 2010, 

Appendix B of DHS Instruction Manual 102-01-001 Acquisition Management 

Instruction/Guidebook, October 2011. 
 

Page 15 GAO-13-256 Combating Nuclear Smuggling

views of the experts identified by the National Academies, the DHS 

acquisition guidance states that (1) the objective of the reviews is to share 

lessons learned throughout the department to increase the probability of 

success for future acquisition programs and (2) the reviews are to take 

place immediately after programs are cancelled. In addition, an official 

from the DHS’s Office of Program Accountability and Risk Management 

(PARM)—DHS’s policy office for acquisitions—told us that it is important 

to conduct these reviews immediately so that key people remain available 

and do not forget the factors that contributed to programs being 

cancelled. Under DHS acquisition guidance, upon completion of the 

lessons learned review, a report documenting the lessons learned is to be 

submitted to PARM.

DHS acquisition guidance on lessons learned provides general direction 

on what is to happen when programs are cancelled, but it does not 

specify processes to follow. PARM officials stated that there were no 

documented processes in place for (1) conducting timely lessons learned 

reviews, (2) preparing lessons learned reports, or (3) disseminating 

lessons learned reports throughout the department. Under the federal 

standards of internal control, agencies are to clearly document internal 

controls (i.e., in management directives, administrative policies, or 

operating manuals), and this documentation should be readily available 

for examination.22 PARM officials told us they generally rely on DHS 

component agencies, such as DNDO, to self-monitor many aspects of 

their programs including conducting lessons learned reviews, preparing 

lessons learned reports, and submitting lessons learned reports.

In the case of the ASP program, a lessons learned review was 

conducted, and a lessons learned report was submitted to PARM and 

disseminated within CBP and DNDO in November 2012 at the explicit 

direction of DHS’s Under Secretary for Management.23 The lessons 

learned report states that the program was cancelled by the Secretary in 

October 2011. However, before the Under Secretary’s directive, there 

was confusion about whether such a review should be conducted for 

 

22GAO, Standards for Internal Control in the Federal Government, GAO/AIMD-00-21.3.1

(Washington, D.C.: November 1999).

23DHS, Acquisition Decision Memorandum from the DHS Under Secretary for 

Management to the Acting Director of DNDO and the Acting Commissioner of CBP, July 

16, 2012.
 

Page 16 GAO-13-256 Combating Nuclear Smuggling

cancelled programs, as well as uncertainty about whether the program 

had been cancelled. DNDO officials told us in March 2012 that, even 

though they considered the program to have been cancelled by the 

Secretary in October 2011, they did not intend to conduct a lessons 

learned review because such reviews were not needed for cancelled 

programs. Furthermore, there was uncertainty between DNDO and 

PARM about whether and when the program had been cancelled. In this 

regard, while DNDO relied on the Secretary’s October 2011 letter, PARM 

officials told us that the program was not cancelled and would not be 

cancelled until DNDO and CBP completed certain additional actions such 

as gathering data to support a future acquisition program. These actions 

were started but not completed, however, the uncertainty over the 

program’s status was eventually resolved in July 2012 with a 

memorandum from the Under Secretary for Management which stated 

that, in accordance with the Secretary’s October 2011 letter, the program 

was considered cancelled.24 In discussing this with DNDO, and DHS 

officials, they acknowledged that the DHS acquisition guidance is 

confusing on when programs are considered cancelled and when to begin 

conducting a lessons learned review. DHS officials stated that they are in 

the process of revising the guidance to provide greater clarity on these 

issues. 

To determine whether the confusion and uncertainty that affected the 

ASP program were unique to that program, we asked PARM officials to 

provide us with examples of previous lessons learned reports from any 

cancelled programs across DHS, including several programs cancelled in

2011 that we had previously reported upon.25 The PARM officials told us 

that they did not have any lessons learned reports for these programs or 

any others in the department. Furthermore, there was confusion within 

the department about the need to disseminate lessons learned reports. 

Before the Under Secretary’s memorandum of July 2012 directing DNDO 

and CBP to prepare and disseminate an ASP lessons learned report, 

 

24This statement was contained in the same memorandum directing that a lessons 

learned report be prepared. 

25See GAO, Homeland Security: DHS Requires More Disciplined Investment 

Management to Help Meet Mission Needs, GAO-12-833 (Washington, D.C.: Sept. 18, 

2012). The cancelled programs that we identified were (1) the Secure Border Initiative 

Network, (2) the Electronic Records Management System, (3) the Transformation and 

Systems Consolidation, (4) the Grants Management Integrated Environment, and (5) the 

Online Tracking Information System.
 

Page 17 GAO-13-256 Combating Nuclear Smuggling

PARM officials had told us there was no requirement to disseminate 

lessons learned outside individual component agencies. Subsequent to 

the memorandum, PARM officials told us that they are developing plans 

to disseminate lessons learned reports through department-wide Centers 

of Excellence.26

The lessons that can be learned from such reviews and reports can be 

extensive and informative. In the case of the ASP program, DNDO and 

CBP’s lessons learned report identified 32 lessons learned in the 

following six categories:

• acquisition strategy;

• documentation of planning;

• structuring and staffing of the program office;

• scheduling;

• testing; and

• coordination between stakeholders and end users regarding technical 

and operational requirements, as well as costs. (See app. II for a copy 

of the lessons learned report.)

For example, concerning acquisition strategy, the lessons learned report 

states the government should consider a “commercial first” approach—

meaning consider whether commercially available equipment can meet 

programmatic needs before developing new technologies. Such an 

approach can reduce costs and risks to the program. Another lesson 

learned from the ASP program is that the acquisition officials and the end

users must work closely together to ensure the needed capability meets 

operational requirements. An additional lesson learned was to manage 

the schedule of the acquisition program by events—not the calendar. In 

other words, the program should not proceed to the next major phase 

until all the criteria are met for completing the current phase. Furthermore, 

the report stated that acquisition programs should be flexible enough to 

 

26DHS’s eight Centers of Excellence for Acquisition and Program Management work with 

acquisition managers, program managers, and subject matter experts across DHS to 

strengthen acquisition and program management.
 

Page 18 GAO-13-256 Combating Nuclear Smuggling

respond to major issues that may be discovered after program initiation. 

Specifically, management should take care not to believe in the program 

so strongly that early signs of trouble are ignored, or cannot be detected 

and acted upon in a timely manner.

At this point, however, PARM has no assurance that that lessons learned 

reviews have been or will be conducted for other cancelled programs, as 

called for in DHS acquisition guidance. Until DHS puts processes in place 

to ensure component agencies conduct lessons learned reviews and 

prepare lessons learned reports, and PARM implements its plans for 

disseminating the reports, DHS risks limiting the probability of success for 

the department’s future acquisition programs. Success in such acquisition 

programs is important given the scale of DHS’s planned investments, 

which DHS has estimated to be many billions of dollars during the next 

few years.27

DHS is participating in international tests of next-generation RPMs, 

including commercially available RPMs that have characteristics similar to 

ASP, and is working with states to gather data using ASPs at five truck 

weigh stations. For the international tests, DNDO is working with its 

European Union (EU) counterparts in the Illicit Trafficking Radiation 

Assessment Program (ITRAP); this program is currently known as 

ITRAP+10 because it is being revisited about 10 years after testing was 

originally conducted. According to the test planning documents, the final 

report is scheduled for issuance in August 2013. The documents further 

indicate that DNDO has the following objectives for the ITRAP+10 tests:

• Provide scientific and technical data on radiological and nuclear 

detection equipment to policy makers.

• Identify the best technologies based on results from repeatable 

testing.

• Promote harmonization of national and international standards and 

guidelines.

 

27Department of Homeland Security, Future Years Homeland Security Program (FYHSP), 

Fiscal Years 2012-2016, Vol. 2, “Program Resources, Milestones, Performance 

Measures, and Capital Investments” (Washington, D.C.: May 2011).

DHS Is Part of 

International Tests of 

Next-Generation 

RPMs and Is Working 

with States to Gather 

ASP Data
 

Page 19 GAO-13-256 Combating Nuclear Smuggling

• Improve international exchange of information.

• Provide manufacturers with feedback on how well their equipment 

performed against standards.

• Promote new research and development efforts.

The commercially available next-generation RPMs that DNDO and the 

EU are testing come from a variety of vendors and are being tested along 

with several other categories of radiation detection equipment. This 

testing is being performed to national and international standards for 

radiation detection set by the American National Standards Institute 

(ANSI)28 and the International Electrotechnical Commission (IEC).29

DNDO officials told us these tests are not part of any planned acquisition. 

According to the test plan, Oak Ridge National Laboratory (ORNL) will 

perform ITRAP+10 tests for DNDO on these next-generation RPMs. The 

primary planning documents for the testing of the next-generation RPMs 

were submitted by the National Institute of Standards and Technology 

and were approved by DNDO and the European Commission’s Joint 

Research Centre.30 These documents indicate that, in addition to testing 

to the standards, ORNL will test to determine the outer limits of what the 

next-generation commercial RPMs can detect.

In addition to participating in the ITRAP+10 tests, DNDO is working with 

state agencies to gather data using five existing ASPs and installed at 

state-operated truck weigh stations in five states. According to a DNDO 

official, data collected will include information on the trucks and cargo 

passing through the ASPs, as well as any radiation detected by ASP. 

According to a DNDO official, data collected by the five ASPs will be sent 

to DNDO’s Joint Analysis Center Collaborative Information System, which 

informs DNDO about nuclear detection and coordinates responses to 

nuclear detection alarms. This official told us the data will be analyzed to 

 

28ANSI is a not-for-profit organization that promotes and facilitates voluntary consensus 

standards that directly impact U. S. businesses in nearly every sector. 

29IEC is a not-for-profit organization for the preparation and publication of international 

standards for all electrical, electronic and related technologies, known collectively as 

“electrotechnology.” 

30The Joint Research Centre is the European Commission’s in-house science service 

center. 
 

Page 20 GAO-13-256 Combating Nuclear Smuggling

determine the types of cargo transported and any trends that may be 

apparent.

To its credit, DHS has cancelled the ASP program, and DNDO conducted 

a lessons learned review about the cancelled program and prepared a 

lessons learned report. However, DNDO prepared the lessons learned 

report and disseminated it 2 years after the final ASP testing, and about a 

year after the Secretary of Homeland Security notified Congress of her 

decision to cancel the program. Moreover, the report was not prepared 

until there was a specific directive from the DHS Under Secretary for 

Management. For cancelled programs, DHS’s acquisition guidance calls 

for a lessons learned review immediately after cancellation of a program, 

preparation of a lessons learned report, and dissemination of such a 

report. However, the acquisition guidance does not constitute an 

institutional requirement, which the experts we consulted said it should 

be. Moreover, DHS does not have documented processes for component 

agencies to follow in (1) conducting timely lessons learned reviews, (2) 

preparing lessons learned reports, or (3) disseminating lessons learned 

reports throughout the department, instead relying on such agencies to 

self-monitor. The amount of time between the final field validation testing 

of the ASP, the program’s cancellation, and the conducting of a lessons 

learned review and dissemination of a lessons learned report suggests 

that timely identification of lessons learned was not a DHS priority. While 

our review focused on the ASP program, the inability of PARM to identify 

any lessons learned reports from cancelled DHS programs suggests that 

the problem is broader than this one program. PARM is developing plans 

for disseminating future lessons learned reports department-wide. 

However, until DHS makes lessons learned reviews an institutional 

requirement, puts processes in place to ensure component agencies 

conduct such reviews and prepare lessons learned reports, and 

implements plans for disseminating the reports, DHS may be missing 

opportunities to improve its chances of success for billions of dollars in 

future acquisitions.

To increase the probability of success for future acquisition programs, we 

recommend that the Secretary of Homeland Security take the following 

four actions for cancelled acquisition programs:

Make lessons learned reviews an institutional requirement, such as 

through an agency directive or order or other appropriate means.

Put documented processes in place to ensure that component agencies

Conclusions 

Recommendations for 

Executive Action
 

Page 21 GAO-13-256 Combating Nuclear Smuggling

• conduct timely lessons learned reviews, and

• prepare and submit lessons learned reports.

Complete and implement plans to disseminate lessons learned reports 

throughout the department.

We provided a draft of this report to DHS for review and comment. In its 

written response, reproduced in appendix III, DHS agreed with all four of 

our recommendations. DHS asked us to consider two of the 

recommendations resolved and closed on the basis of existing guidance 

and processes that were discussed in the body of this report. DHS 

outlined steps that it intends to take to address the other two 

recommendations. We are encouraged by some of the actions planned. 

However, we do not consider two of the recommendations resolved and 

closed because, as discussed below, we believe that DHS should take 

steps to address these recommendations.

In DHS’s response to our recommendation to make lessons learned 

reviews an institutional requirement, DHS commented that it has

institutionalized this requirement through existing departmental 

acquisitions guidance that was discussed in the body of this report.31

However, as discussed in this report and confirmed by PARM officials 

during the course of our review, this guidance does not constitute a 

requirement. DHS’s comments also refer to additional guidance called the 

Department of Homeland Security (DHS) Capital Planning and 

Investment Control (CPIC) Guide. However, officials from both PARM and 

the DHS Office of the Chief Information Officer told us that CPIC applies 

only to information technology programs,32 not to other programs such as 

ASP. In our view, the existing guidance referenced in DHS’s comments 

contributed to the confusion and uncertainty we discussed in the body of 

this report, including confusion about whether lessons learned were 

 

31DHS Systems Engineering Life Cycle Version 2.0 (INTERIM), September 2010, 

Appendix B of DHS Instruction Manual 102-01-001 Acquisition Management 

Instruction/Guidebook, October 2011.

32Department of Homeland Security (DHS), Office of the Chief Information Officer, Capital 

Planning and Investment Control (CPIC) Guide, Version 7.1 (August 2010). 

https://learn.dau.mil/CourseWare/809481_1/resources/cpic_guide.pdf (accessed May 3, 

2013). 

Agency Comments 

and Our Evaluation
 

Page 22 GAO-13-256 Combating Nuclear Smuggling

required for cancelled programs. Accordingly, we continue to believe that 

DHS needs to take action to address the problem. 

Similarly, in commenting on our recommendation to put processes in 

place to ensure that component agencies conduct timely lessons learned 

reviews, DHS stated that our recommendation is addressed by its existing 

acquisition oversight process, as evidenced by the completion of a 

lessons learned report for ASP. However, according to DNDO and CBP’s 

lessons learned report, the Secretary of DHS cancelled the program in 

October 2011,33 but the lessons learned report for this program was not 

issued until more than a year later. Furthermore, as noted in this report, 

the ASP lessons learned report is the only example PARM officials could 

identify of such a report having been completed. In our view, relying on 

this existing process seems unlikely to ensure that component agencies 

conduct timely lessons learned reviews.

With regard to our final two recommendations—to put documented 

processes in place to ensure that component agencies prepare and 

submit lessons learned reports, and to complete and implement plans to 

disseminate lessons learned reports throughout the department—DHS

stated that it is taking several actions. Specifically, DHS is revising and 

clarifying its acquisitions guidance with respect to lessons learned; it 

expects to complete these revisions by June 2014. In its comments, DHS 

stated that it has developed an online forum and library for DHS 

acquisition professionals that includes an area dedicated to lessons 

learned and best practices. Furthermore, DHS has designated a lead 

Center of Excellence and established a time frame of September 2013 for 

collecting and posting information on lessons learned for cancelled 

programs. In our view, as DHS completes more lessons learned reports, 

an online forum and library could become a useful means of 

disseminating these reports throughout the department.

DHS also provided technical comments that we incorporated in the report, 

as appropriate.

 

33Department of Homeland Security, Domestic Nuclear Detection Office (DNDO), 

Advanced Spectroscopic Portal (ASP) Program Post Implementation/Lessons Learned 

(Redacted), Document number 600-ASP-119650v1.11 (Dec. 4, 2012). See the Executive 

Summary, which is included as appendix III of this report.
 

Page 23 GAO-13-256 Combating Nuclear Smuggling

As agreed with your offices, unless you publicly announce the contents of 

this report earlier, we plan no further distribution until 30 days from the 

report date. At that time, we will send copies to the Secretary of 

Homeland Security, the Director of DNDO, the Commissioner of CBP, the 

Executive Director of PARM, the appropriate congressional committees, 

and other interested parties. In addition, the report will be available at no 

charge on the GAO website at http://www.gao.gov. 

If you or your staff members have any questions about this report, please 

contact David C. Trimble at (202) 512-3841 or trimbled@gao.gov or Dr. 

Timothy M. Persons at (202) 512-6412 or personst@gao.gov. Contact 

points for our Offices of Congressional Relations and Public Affairs may 

be found on the last page of this report. GAO staff who made key 

contributions to this report are listed in appendix IV.

David C. Trimble

Director, Natural Resources and Environment

Timothy M. Persons, Ph.D., Chief Scientist

Director, Center for Science, Technology, and Engineering
Appendix I: Experts Identified by the National 

Academies Who Described Leading Practices 

for Reviewing Cancelled Acquisition Programs

Page 24 GAO-13-256 Combating Nuclear Smuggling

At our request, the National Academies identified experts who could 

highlight leading practices for reviewing cancelled acquisition programs. 

These experts were identified on the basis on their knowledge of leading 

practices in large-scale engineering and acquisition programs. The 

National Academies provided us a list of 10 experts, 7 of whom were 

available to participate in this effort. Six of the 7 experts who were 

available were members of the National Academy of Engineering. We 

conducted a group telephone conference with 6 of the experts. We 

drafted conclusions from the conference and circulated them to all 7 of 

the experts for their review and comment. With one exception, the experts 

reached consensus on all of the observations. One expert felt that 

lessons learned reports were so important that acquisitions executives 

should have less authority to tailor or waive the requirements for such 

reports.

Hon. Philip E. Coyle, consultant, former Assistant Secretary of Defense 

and Director of Operational Test and Evaluation

Dr. Joseph M. DeSimone; Chancellor’s Eminent Professor of Chemistry 

at the University of North Carolina; Member of the National Academy of 

Engineering; Member of the National Academy of Science

Dr. Donald Fraser; Draper Laboratory, retired; former Principal Under 

Secretary of Defense for Acquisition; Member of the National Academy of 

Engineering

Dr. Mary L. Good; Dean Emeritus and Special Assistant to the 

Chancellor, Center for Innovation and Commercialization at the University 

of Arkansas at Little Rock; former Under Secretary for Technology in the 

Department of Commerce; Member of the National Academy of 

Engineering

Don R. Kozlowski; Senior Vice President of the Boeing Company, retired; 

Member of the National Academy of Engineering

Dr. Lawrence Papay; Chief Executive Officer of PQR, LLC; former Sector 

Vice President of SAIC; Member of the National Academy of Engineering

A. Thomas Young; Executive Vice President of the Lockheed Martin 

Corporation, retired; former President and Chief Operating Officer of the 

Martin Marietta Corporation; Member of the National Academy of 

Engineering

Appendix I: Experts Identified by the 

National Academies Who Described Leading 

Practices for Reviewing Cancelled 

Acquisition Programs
Appendix II: DNDO-CBP ASP Lessons Learned 

Report

Page 25 GAO-13-256 Combating Nuclear Smuggling

The following DNDO-CBP ASP lessons learned report was redacted on December 4, 

2012, from the November 5, 2012, For Official Use Only version by DNDO.

Appendix II: DNDO-CBP ASP Lessons 

Learned Report
Appendix II: DNDO-CBP ASP Lessons Learned 

Report

Page 26 GAO-13-256 Combating Nuclear Smuggling
Appendix II: DNDO-CBP ASP Lessons Learned 

Report

Page 27 GAO-13-256 Combating Nuclear Smuggling
Appendix II: DNDO-CBP ASP Lessons Learned 

Report

Page 28 GAO-13-256 Combating Nuclear Smuggling
Appendix II: DNDO-CBP ASP Lessons Learned 

Report

Page 29 GAO-13-256 Combating Nuclear Smuggling
Appendix II: DNDO-CBP ASP Lessons Learned 

Report

Page 30 GAO-13-256 Combating Nuclear Smuggling
Appendix II: DNDO-CBP ASP Lessons Learned 

Report

Page 31 GAO-13-256 Combating Nuclear Smuggling
Appendix II: DNDO-CBP ASP Lessons Learned 

Report

Page 32 GAO-13-256 Combating Nuclear Smuggling
Appendix II: DNDO-CBP ASP Lessons Learned 

Report

Page 33 GAO-13-256 Combating Nuclear Smuggling
Appendix II: DNDO-CBP ASP Lessons Learned 

Report

Page 34 GAO-13-256 Combating Nuclear Smuggling
Appendix II: DNDO-CBP ASP Lessons Learned 

Report

Page 35 GAO-13-256 Combating Nuclear Smuggling
Appendix II: DNDO-CBP ASP Lessons Learned 

Report

Page 36 GAO-13-256 Combating Nuclear Smuggling
Appendix II: DNDO-CBP ASP Lessons Learned 

Report

Page 37 GAO-13-256 Combating Nuclear Smuggling
Appendix II: DNDO-CBP ASP Lessons Learned 

Report

Page 38 GAO-13-256 Combating Nuclear Smuggling
Appendix III: Comments from the Department 

of Homeland Security

Page 39 GAO-13-256 Combating Nuclear Smuggling

Appendix III: Comments from the 

Department of Homeland Security
Appendix III: Comments from the Department 

of Homeland Security

Page 40 GAO-13-256 Combating Nuclear Smuggling
Appendix III: Comments from the Department 

of Homeland Security

Page 41 GAO-13-256 Combating Nuclear Smuggling
Appendix IV: GAO Contacts and Staff 

Acknowledgments

Page 42 GAO-13-256 Combating Nuclear Smuggling

David C. Trimble, (202) 512-3841 or trimbled@gao.gov

Timothy M. Persons, Ph.D., (202) 512-6412 or personst@gao.gov

In addition to the individuals named above, Ned Woodward, Assistant 

Director; Gene Aloise; Cheryl R. Arvidson; Antoinette C. Capaccio; 

Frederick K. Childers; R. Scott Fletcher; Cindy K. Gilbert; David C. 

Maurer; Cynthia Norris; Katrina E. Pekar-Carpenter; Kiki Theodoropoulos; 

and Nathan A. Tranquilli made key contributions to this report.

Appendix IV: GAO Contacts and Staff 

Acknowledgments 

GAO Contacts

Staff 

Acknowledgments 
Related GAO Products

Page 43 GAO-13-256 Combating Nuclear Smuggling

Combating Nuclear Smuggling: DHS has Developed Plans for Its Global 

Nuclear Detection Architecture, but Challenges Remain in Deploying 

Equipment. GAO-12-941T. Washington, D.C.: July 26, 2012.

Combating Nuclear Smuggling: DHS Has Made Some Progress but Not 

Yet Completed a Strategic Plan for Its Global Nuclear Detection Efforts or 

Closed Identified Gaps. GAO-10-883T. Washington, D.C.: June 30, 2010.

Combating Nuclear Smuggling: Recent Testing Raises Issues About the 

Potential Effectiveness of Advanced Radiation Detection Portal Monitors.

GAO-10-252T. Washington, D.C.: November 17, 2009.

Combating Nuclear Smuggling: Lessons Learned from DHS Testing of 

Advanced Radiation Detection Portal Monitors. GAO-09-804T. 

Washington, D.C.: June 25, 2009.

Combating Nuclear Smuggling: DHS Improved Testing of Advanced 

Radiation Detection Portal Monitors, but Preliminary Results Show Limits 

of the New Technology. GAO-09-655. Washington, D.C.: May 21, 2009.

Nuclear Detection: Domestic Nuclear Detection Office Should Improve 

Planning to Better Address Gaps and Vulnerabilities. GAO-09-257. 

Washington, D.C.: January 29, 2009.

Department of Homeland Security: Billions Invested in Major Programs 

Lack Appropriate Oversight. GAO-09-29. Washington, D.C.: November 

18, 2008.

Combating Nuclear Smuggling: DHS’s Phase 3 Test Report on Advanced 

Portal Monitors Does Not Fully Disclose the Limitations of the Test 

Results. GAO-08-979. Washington, D.C.: September 30, 2008.

Combating Nuclear Smuggling: DHS Needs to Consider the Full Costs 

and Complete All Tests Prior to Making a Decision on Whether to 

Purchase Advanced Portal Monitors. GAO-08-1178T. Washington, D.C.: 

September 25, 2008.

Combating Nuclear Smuggling: DHS’s Program to Procure and Deploy 

Advanced Radiation Detection Portal Monitors Is Likely to Exceed the 

Department’s Previous Cost Estimates. GAO-08-1108R. Washington, 

D.C.: September 22, 2008.

Related GAO Products
Related GAO Products

Page 44 GAO-13-256 Combating Nuclear Smuggling

Combating Nuclear Smuggling: DHS’s Decision to Procure and Deploy 

the Next Generation of Radiation Detection Equipment Is Not Supported 

by Its Cost-Benefit Analysis. GAO-07-581T. Washington, D.C.: March 14, 

2007.

Combating Nuclear Smuggling: DNDO Has Not Yet Collected Most of the 

National Laboratories’ Test Results on Radiation Portal Monitors in 

Support of DNDO’s Testing and Development Program. GAO-07-347R. 

Washington, D. C.: March 9, 2007.

Combating Nuclear Smuggling: DHS’s Cost-Benefit Analysis to Support 

the Purchase of New Radiation Detection Portal Monitors Was Not Based 

on Available Performance Data and Did Not Fully Evaluate All the 

Monitors’ Costs and Benefits. GAO-07-133R. Washington, D.C.: October 

17, 2006.

Combating Nuclear Smuggling: Additional Actions Needed to Ensure 

Adequate Testing of Next Generation Radiation Detection Equipment.

GAO-07-1247T. Washington, D.C.: September 18, 2007.

Combating Nuclear Smuggling: DHS Has Made Progress Deploying 

Radiation Detection Equipment at U.S. Ports-of-Entry, but Concerns 

Remain. GAO-06-389. Washington, D.C.: March 22, 2006.

(361322)
The Government Accountability Office, the audit, evaluation, and 

investigative arm of Congress, exists to support Congress in meeting its 

constitutional responsibilities and to help improve the performance and 

accountability of the federal government for the American people. GAO 

examines the use of public funds; evaluates federal programs and 

policies; and provides analyses, recommendations, and other assistance 

to help Congress make informed oversight, policy, and funding decisions. 

GAO’s commitment to good government is reflected in its core values of 

accountability, integrity, and reliability.

The fastest and easiest way to obtain copies of GAO documents at no 

cost is through GAO’s website (http://www.gao.gov). Each weekday 

afternoon, GAO posts on its website newly released reports, testimony, 

and correspondence. To have GAO e-mail you a list of newly posted 

products, go to http://www.gao.gov and select “E-mail Updates.”

The price of each GAO publication reflects GAO’s actual cost of 

production and distribution and depends on the number of pages in the 

publication and whether the publication is printed in color or black and 

white. Pricing and ordering information is posted on GAO’s website, 

http://www.gao.gov/ordering.htm. 

Place orders by calling (202) 512-6000, toll free (866) 801-7077, or 

TDD (202) 512-2537. 

Orders may be paid for using American Express, Discover Card, 

MasterCard, Visa, check, or money order. Call for additional information.

Connect with GAO on Facebook, Flickr, Twitter, and YouTube. 

Subscribe to our RSS Feeds or E-mail Updates. Listen to our Podcasts. 

Visit GAO on the web at www.gao.gov. 

Contact:

Website: http://www.gao.gov/fraudnet/fraudnet.htm

E-mail: fraudnet@gao.gov

Automated answering system: (800) 424-5454 or (202) 512-7470

Katherine Siggerud, Managing Director, siggerudk@gao.gov, (202) 512-

4400, U.S. Government Accountability Office, 441 G Street NW, Room 

7125, Washington, DC 20548

Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800

U.S. Government Accountability Office, 441 G Street NW, Room 7149 

Washington, DC 20548

GAO’s Mission

Obtaining Copies of 

GAO Reports and 

Testimony 

Order by Phone

Connect with GAO

To Report Fraud, 

Waste, and Abuse in 

Federal Programs

Congressional 

Relations 

Public Affairs

Please Print on Recycled Paper.