SUPPLY CHAIN 

SECURITY

CBP Needs to 

Conduct Regular 

Assessments of Its 

Cargo Targeting 

System

Report to Congressional Requesters

October 2012

GAO-13-9

United States Government Accountability Office

GAO
United States Government Accountability Office

Highlights of GAO-13-9, a report to 

congressional requesters 

October 2012 

SUPPLY CHAIN SECURITY 

CBP Needs to Conduct Regular Assessments of Its 

Cargo Targeting System

Why GAO Did This Study

The U.S. economy is dependent on the 

expeditious flow of millions of tons of 

cargo each day. Cargo containers are 

an important instrument of global trade 

but also can present security concerns.

CBP is responsible for administering 

container security programs, and its

strategy for securing maritime cargo 

containers includes analyzing 

information to identify shipments that 

may contain terrorist weapons or other 

contraband. Because CBP has 

insufficient resources to examine every 

container, targeters use ATS to target

which container shipments should be 

examined. GAO was asked to assess 

CBP’s targeting efforts. This report 

addresses (1) how ATS supports 

CBP’s targeting of maritime cargo 

container shipments for national 

security purposes and (2) the extent to 

which CBP assesses the effectiveness 

of ATS’s national security targeting 

rules. GAO analyzed fiscal year 2011 

CBP data on shipments and containers 

arriving at U.S. ports and containers 

scanned at these ports. GAO also 

visited six CBP units selected on the 

basis of the percentage of maritime 

shipments that were scored as high 

risk or medium risk for national security 

purposes at these locations in fiscal 

year 2011, among other factors. GAO 

also analyzed documents, such as 

CBP’s ATS performance measures. 

What GAO Recommends

GAO recommends that CBP (1) ensure 

that future updates to the weight set 

are based on assessments of its 

performance and (2) establish targets 

for performance measures and use 

those measures to regularly assess

effectiveness of the weight set. DHS 

concurred with these 

recommendations.

What GAO Found

U.S. Customs and Border Protection (CBP), within the Department of Homeland 

Security (DHS), employs a risk-based approach that uses the Automated 

Targeting System (ATS) and other tools to identify (target) maritime cargo 

shipments for further examination. ATS is a web-based enforcement and 

decision support system that includes a set of rules to assess the risk level for 

each arriving cargo shipment. This set of rules is referred to as the maritime 

national security weight set (weight set) because each rule in the set has a 

specific weighted value assigned to it. CBP classifies the risk scores from the 

weight set as low, medium, or high risk. CBP policy states that a shipment’s risk 

score is to determine, in part, actions taken by CBP officers (targeters) at the 

ports. Specifically, targeters are generally required to review shipment data for all 

medium-risk and high-risk shipments and hold high-risk shipments for 

examination. The risk score, however, is not the sole factor that determines 

whether a targeter reviews the data for a shipment or whether CBP examines a 

shipment. In particular, targeters at each of the six ports GAO visited explained 

that they use the ATS risk score as a starting point for the targeting process but 

that their decisions regarding which shipments to examine are ultimately based 

on additional research. Targeters at the six ports GAO visited said they also use 

tools outside of ATS, such as web searches, to research shipments.

CBP efforts to assess the weight set’s effectiveness in identifying the risk of 

shipments have been limited. CBP has performance measures—represented by 

the percentage of shipments targeted as high risk that contain a threat and the 

percentage of shipments targeted as high risk that do not contain a threat—that 

enable CBP to determine the accuracy of the weight set, given a particular 

workload or examination rate. However, CBP did not assess the weight set to 

verify its effectiveness when implementing an updated version in early 2011. 

Prior to implementing the updated version of the weight set, CBP assessed the

potential impact of the update on CBP’s workload but did not conduct an 

assessment to determine whether the updated version of the weight set would be 

more effective in identifying high-risk shipments than the previous version or 

other alternatives. Assessing the potential effectiveness of alternative versions of 

the weight set prior to selecting one for implementation could help CBP make 

more informed decisions about future updates. Doing so could also provide CBP 

reasonable assurance that the version it selects is the most effective of the 

alternatives and is more effective than the previous version it replaces. 

Furthermore, since implementing the updated version of the weight set in early

2011, CBP has not regularly assessed the weight set to monitor its performance 

and to help determine when changes are needed. For example, CBP conducted 

the first assessment of the current version of the weight set, using the 

performance measures, in the summer of 2012—18 months after the weight set’s 

implementation in early 2011. Regular assessments of the weight set’s 

effectiveness could help CBP determine when updates are needed in a timelier 

manner and ensure that targeters have the best information available to make 

targeting decisions. Moreover, CBP has not established targets for the 

performance measures so that it is not clear whether a particular change in the 

weight set’s performance is significant enough to suggest that changes are 

needed to improve the effectiveness of the weight set. 

View GAO-13-9. For more information, contact 

Stephen L. Caldwell at (202) 512-9610 or 

caldwells@gao.gov. 
Page i GAO-13-9 Supply Chain Security

Letter 1 

Background 8

CBP Uses ATS and Other Tools to Target Maritime Cargo 

Shipments for National Security Purposes 15

CBP’s Efforts to Assess the Weight Set Have Been Limited 19

Conclusions 25

Recommendations for Executive Action 26

Agency Comments 26

Appendix I Federal Strategy for Ensuring the Security of Maritime Cargo 

Container Shipments 28

Appendix II Past Audit Findings and Recommendations to Improve the Targeting 

Process 30

Appendix III Comments from the Department of Homeland Security 38

Appendix IV GAO Contact and Staff Acknowledgments 40

Related GAO Products 41

Tables

Table 1: Description of CBP’s Core Cargo Security Programs 28

Table 2: Status of GAO Recommendations from February 2004 31

Figure

Figure 1: Key Steps for Targeting High-Risk Shipments throughout 

the Maritime Supply Chain Process 10

Contents
Page ii GAO-13-9 Supply Chain Security

Abbreviations

A&T Analysis and Targeting Division

ATS Automated Targeting System

ATU Advance Targeting Unit

CBP U.S. Customs and Border Protection

CERTS Cargo Enforcement Reporting and Tracking System

DHS Department of Homeland Security

FMFIA Federal Managers' Financial Integrity Act

FPR false positive rate

NII nonintrusive inspection

NTC-C National Targeting Center–Cargo

OIG Office of Inspector General

OMB Office of Management and Budget

TPR true positive rate

WMD weapons of mass destruction

This is a work of the U.S. government and is not subject to copyright protection in the 

United States. The published product may be reproduced and distributed in its entirety 

without further permission from GAO. However, because this work may contain 

copyrighted images or other material, permission from the copyright holder may be 

necessary if you wish to reproduce this material separately.
Page 1 GAO-13-9 Supply Chain Security

United States Government Accountability Office 

Washington, DC 20548

October 25, 2012

The Honorable Susan M. Collins

Ranking Member

Committee on Homeland Security and Governmental Affairs

United States Senate

The Honorable Cliff Stearns

Chairman

The Honorable Diana DeGette

Ranking Member

Subcommittee on Oversight and Investigations

Committee on Energy and Commerce

House of Representatives

The economic well-being of the United States is dependent on the 

expeditious flow of millions of tons of cargo each day. According to the 

U.S. Department of Transportation, the majority of U.S. imports arrive by 

ocean vessel, and much of that is transported in cargo containers.1 In 

fiscal year 2011, for example, about 13.4 million cargo containers arrived 

at U.S. seaports. Cargo containers are an important segment of the 

global supply chain—the flow of goods from manufacturers to retailers—

and can present significant security concerns. Within the federal 

government, U.S. Customs and Border Protection (CBP), part of the 

Department of Homeland Security (DHS), is responsible for administering 

container security and reducing the vulnerabilities associated with the 

supply chain. Balancing security concerns with the need to facilitate the 

free flow of commerce, part of CBP’s mission, remains an ongoing 

challenge for the public and private sectors alike.2

 

1

U.S. Department of Transportation, Research and Innovative Technology Administration, 

Bureau of Transportation Statistics, America’s Container Ports: Linking Markets at Home 

and Abroad (Washington, D.C.: January 2011).

CBP officials believe 

that the likelihood of terrorists smuggling weapons of mass destruction 

(WMD) into the United States in cargo containers is relatively low; 

however, the consequences of such an event could be catastrophic. 

2

In addition to its priority mission of keeping terrorists and their weapons out of the United 

States, CBP is also responsible for securing the border, facilitating international trade and 

travel, collecting duties, and enforcing numerous U.S. laws and regulations pertaining to 

immigration and illicit drugs, among other things.

 
 

Page 2 GAO-13-9 Supply Chain Security

Although there have been no known incidents of cargo containers being 

used to transport WMD, ensuring the security of cargo containers remains 

an important role for CBP given that criminals have exploited containers 

for other illegal purposes, such as smuggling weapons, people, and illicit 

substances.

Since September 11, 2001, Congress has passed various laws to 

address concerns about the security of cargo containers in the global 

supply chain. The enactment of the Maritime Transportation Security Act 

of 20023 called for the establishment of a program to evaluate and certify 

secure systems of international intermodal transportation, including 

standards and procedures for screening and evaluating cargo containers 

prior to loading onto vessels and for securing and monitoring cargo while 

in transit.4 In 2006, the Security and Accountability For Every Port Act 

was enacted,5 which required, among other things, that pilot projects be 

established at three ports to test the feasibility of scanning 100 percent of 

U.S.-bound cargo containers at foreign ports.6 Subsequently, the 

Implementing Recommendations of the 9/11 Commission Act of 2007 

(9/11 Act)7 required, among other things, that by July 2012, 100 percent 

of U.S.-bound cargo containers be scanned at foreign ports with both 

radiation detection and nonintrusive inspection (NII) equipment before 

being placed on U.S.-bound vessels.8

 

3

Pub. L. No. 107-295, 116 Stat. 2064.

In May 2012, the Secretary of 

Homeland Security authorized a 2-year extension—until July 2014—of

the deadline for implementing the requirement that containers not enter 

4

See 46 U.S.C. § 70116.

5

Pub. L. No. 109-347, 120 Stat. 1884.

6

6 U.S.C. § 981. A similar requirement was enacted that same year by the Department of 

Homeland Security Appropriations Act, 2007 (Pub. L. No. 109-295, 120 Stat. 1355 (2006)) 

and is codified at 6 U.S.C. § 981a. Both statutes specify scanning as examination with 

both radiation detection equipment and nonintrusive imaging equipment. 6 U.S.C. 

§§ 981(a), 981a(a)(1).

7

Pub. L. No. 110-53, § 1701(a), 121 Stat. 266, 489-90 (amending 6 U.S.C. § 982(b)).

8

Radiation detection equipment detects radiation being emitted from a container, and 

through an NII scan, CBP can identify anomalies in a container’s image that could, among 

other things, indicate the presence of shielding material.
 

Page 3 GAO-13-9 Supply Chain Security

the United States unless they were scanned at foreign ports prior to being 

loaded on vessels.

9

We reported in October 2009 that CBP faced numerous challenges in 

implementing 100 percent scanning at pilot ports, and on the basis of 

work we have completed since then we know that CBP has not yet 

achieved 100 percent scanning.10 In October 2009, we recommended, 

among other things, that CBP conduct a feasibility analysis of 

implementing the 100 percent scanning requirement and provide the 

results to Congress along with any suggestions of cost-effective 

alternatives to implementing the 100 percent scanning requirement, as 

appropriate. DHS stated that CBP concurred with these 

recommendations, but CBP has not yet taken action to address them. In 

its report to Congress in May 2012 on the planned deadline extension, 

DHS stated that it recognizes the need to proceed with container security 

programs in a manner that maximizes the security of maritime cargo and 

facilitates its movement and reported that it plans to continue working with 

other federal agencies and international partners to develop technology 

and enhance risk management processes, in addition to continuing 

existing programs that enhance cargo security.11

 

9

The 9/11 Act scanning provision includes possible extensions for containers loaded at a 

port or ports for which DHS certifies that at least two out of a list of specific conditions 

exist. Among others, these conditions include the following: (1) adequate scanning 

equipment is not available or cannot be integrated with existing systems, (2) a port does 

not have the physical characteristics to install the equipment, or (3) use of the equipment 

will significantly affect trade capacity and the flow of cargo. See 6 U.S.C. § 982(b)(4).

However, given that the 

feasibility of 100 percent scanning remains in doubt and DHS and CBP 

have not identified alternatives that could achieve the same goals as 100 

percent scanning, uncertainty persists regarding the scope of DHS’s and 

CBP’s existing container security programs and how these programs will 

collectively affect the movement of goods between trading partners.

10GAO, Supply Chain Security: Container Security Programs Have Matured, but 

Uncertainty Persists over the Future of 100 Percent Scanning, GAO-12-422T 

(Washington, D.C.: Feb. 7, 2012), and Supply Chain Security: Feasibility and Cost-Benefit 

Analysis Would Assist DHS and Congress in Assessing and Implementing the 

Requirement to Scan 100 Percent of U.S.-Bound Containers, GAO-10-12 (Washington, 

D.C.: Oct. 30, 2009).

11DHS, Scanning of Maritime Cargo Containers: Fiscal Year 2012 Report to Congress

(Washington, D.C.: May 3, 2012). 
 

Page 4 GAO-13-9 Supply Chain Security

CBP’s strategy for securing the maritime supply chain consists of 

programs that intersect with key points in the supply chain. These 

programs include, among other things, analyzing information to identify 

shipments that may be at high risk of transporting WMD or other 

contraband; working with foreign governments to examine U.S.-bound, 

high-risk shipments at foreign ports; and examining U.S.-bound, high-risk 

shipments that were not examined overseas upon their arrival at a 

domestic port.12 To aid in this process, CBP uses the Automated 

Targeting System (ATS), which is an intranet-based enforcement and 

decision support system that compares traveler, cargo, and conveyance 

information against intelligence and other enforcement data. Among other 

things, ATS uses a set of rules that assess different factors in data 

provided by supply chain parties, such as importers, to determine the risk 

level for a shipment. CBP officers (targeters) use information in ATS to 

identify (target) which shipments should be held for an examination, 

which may include an NII scan or a physical inspection.13 Because CBP 

does not scan 100 percent of U.S.-bound cargo containers, the 

effectiveness of CBP’s security strategy depends on CBP’s ability to use 

ATS, among other tools, to effectively target shipments in the supply 

chain that pose the greatest security risks.

 

12CBP refers to the automated process of analyzing data and classifying shipments by risk 

level as screening, and according to CBP, it screens (but does not scan) all U.S.-bound 

cargo shipments before they are loaded onto vessels at foreign ports. In this report, we 

discuss the screening process in terms of assessing the risk of a shipment. An 

examination refers to either (1) the scanning of a container or other cargo conveyance 

using large-scale NII technology, which may use X-rays or gamma rays to create an 

image of the contents of the container or other conveyance, or (2) a physical inspection of 

a container or other cargo conveyance. If the results of an NII scan indicate that a threat 

may be present, CBP may choose to conduct a physical inspection. In addition to an NII 

exam, scanning can also refer to the use of radiation detection equipment, such as 

radiation portal monitors. According to CBP, 99 percent of containers are scanned through 

radiation portal monitors prior to leaving a domestic port.

13In this report, we use the term “targeting” to refer to the synthesis and use of information 

from a variety of sources, including the results of screening, to identify shipments that may 

be a potential security risk.
 

Page 5 GAO-13-9 Supply Chain Security

In response to your request, we analyzed certain aspects of CBP’s 

maritime national security targeting efforts. Specifically, this report 

addresses the following objectives:

• How does ATS support CBP’s process for targeting maritime cargo 

container shipments for national security purposes?

• To what extent does CBP assess the effectiveness of the national 

security targeting rules in ATS?

To address the first objective, we obtained data from CBP for each of the 

115 U.S. seaports for fiscal year 2011 on the number of (1) shipments it 

placed in each of three categories—high risk, medium risk, and low risk—

arriving at each of the ports;14 (2) container arrivals at these ports; and (3) 

containers scanned at these ports using NII equipment. We selected and 

visited six CBP units responsible for targeting at domestic ports. 

Specifically, these ports were selected from among the largest ports in 

the United States using the following criteria:15 (1) the percentage of 

maritime shipments that were scored as high risk or medium risk for 

national security purposes, (2) the percentage of cargo containers that 

were examined using NII equipment, and (3) whether a CBP official from 

the port participated in the most recent CBP conference to discuss 

changes to ATS cargo targeting rules.16

 

14CBP collects data on the number of shipments as well as the number of containers 

arriving in the United States. A shipment is the tender of one lot of cargo at one time from 

one shipper to one recipient. In some cases, a shipment will refer to all the contents in a 

single container. In other cases, a shipment may refer to the cargo in multiple containers. 

Additionally, a single container could hold multiple shipments from different supply chain 

parties.

The six targeting units we visited 

15We limited our selection to the 20 ports with the largest volume of arriving shipments in 

fiscal year 2011 because the consideration of our other criteria—specifically the 

percentage of high-risk or medium-risk shipments and the percentage of containers 

scanned using NII equipment—could be disproportionately influenced by ports with 

smaller volumes. 

16According to CBP officials, prior to making an update to the weight set, CBP hosts a 

rules conference to discuss potential updates to the rules in the weight set. In February 

2009, CBP hosted a conference (the Importer Security Filing rules conference) in which 

CBP officials explained policies regarding the receipt of new data from importers and 

vessel carriers and how the data were to be used in targeting. Targeters from various 

ports attended this conference as representatives of their ports and subject matter 

experts. According to CBP officials, this conference was different from rules conferences 

in the past. However, CBP could not provide attendee lists for the rules conference that 

preceded the Importer Security Filing conference, and therefore we used attendance at 

the Importer Security Filing conference as part of our criteria.
 

Page 6 GAO-13-9 Supply Chain Security

were responsible for targeting efforts at 15 ports that collectively received 

about 60 percent of the maritime shipments that arrived in the United 

States in fiscal year 2011. As part of these site visits, we observed port 

operations, including the scanning of containers. At each location, we 

also interviewed CBP targeters, including those who participated in the 

most recent CBP conference to discuss changes to cargo targeting rules, 

and observed their use of ATS and other tools to conduct cargo targeting 

activities. The results from our visits to these six targeting units cannot be 

generalized to ports nationwide; however, visits to these locations allowed 

us to directly observe the targeting process and provided insights into 

how ATS assigns risk scores to maritime shipments and how CBP 

integrates the scores into its targeting process. We synthesized the 

information from these site visits to describe the targeting process and 

also analyzed CBP policies and guidance, such as CBP’s National 

Maritime Targeting Policy and course materials from CBP’s Sea Cargo 

Targeting Training. We also visited CBP’s National Targeting Center–

Cargo (NTC-C) to interview targeters responsible for conducting nationallevel targeting and to observe their targeting activities.17

To address the second objective, we analyzed CBP’s performance 

measures related to CBP’s national security targeting rules in ATS that 

are used to assess potential risks in maritime cargo container shipments. 

These performance measures include the true positive rate and false 

positive rate, as discussed later in this report. In particular, we reviewed a 

consulting firm’s evaluation of ATS from 2006 and CBP’s project plan for 

implementing the most recent update to the targeting rules, including 

CBP’s plans for monitoring ATS through the use of performance 

measures and an established methodology. We also reviewed our past 

work on ATS and its effectiveness, as well as an audit report from the 

DHS Office of Inspector General.18

 

17NTC-C analyzes advance cargo tactical and strategic information using ATS before 

shipments reach the United States. NTC-C also promotes information sharing with other 

federal agencies and foreign governments to detect and seize threats at U.S. and foreign 

ports.

The Office of Inspector General’s 

assessment of CBP’s modifications to the ATS national security targeting 

rules was based on a series of interviews with agency officials and a 

18DHS Office of Inspector General, Cargo Targeting and Examinations, OIG-10-34 

(Washington, D.C.: Jan. 6, 2010). Because the reports we issued in February 2004 and 

August 2006 regarding CBP’s targeting practices contain sensitive information, they are 

not publicly available.
 

Page 7 GAO-13-9 Supply Chain Security

review of relevant documentation and was sufficient to address the issue 

of documentation of changes to the ATS rules that we present in this 

report. We analyzed documentation of CBP’s most recent update to the 

national security targeting rules, which CBP implemented in early 2011,

and the extent to which this documentation addressed effectiveness. 

Specifically, we evaluated CBP’s impact assessments, which provide 

information on the number of shipments the updated rules would assess 

as high risk, thereby affecting CBP’s examination workload. In addition, 

we evaluated the extent to which CBP used its methodology to assess 

the current national security targeting rules and analyzed the results of 

CBP’s assessments. To assess the reliability of the results of CBP’s 

assessments, we reviewed documentation on the methodology created 

for CBP by a consulting firm in 2006. We interviewed knowledgeable CBP 

officials about any adjustments to this methodology since the contract 

expired and CBP analysts began conducting the performance 

assessments. On the basis of this information we determined that the 

results of the assessments are sufficiently reliable for the purposes of our 

report. To determine the extent to which CBP conducts such 

assessments on a regular basis, we analyzed documentation of recent 

assessments of the national security targeting rules conducted in spring 

2011 and summer 2012. We compared this information with key elements 

for a risk management approach and Standards for Internal Control in the 

Federal Government.

19

DHS deemed some of the information in a draft version of this report as 

sensitive, and therefore, this report omits sensitive details regarding 

specific information available in ATS, examples of how targeters may use 

We also reviewed our prior work on risk 

management practices and compared our analysis of CBP’s actions and 

assessments with those practices. Finally, we interviewed officials at CBP 

headquarters who are responsible for maintaining and updating ATS to 

obtain information on past efforts to assess and update the national 

security targeting rules.

 

19GAO, Standards for Internal Control in the Federal Government, GAO/AIMD-00-21.3.1

(Washington, D.C.: November 1999). These standards, issued pursuant to the 

requirements of the Federal Managers’ Financial Integrity Act of 1982 (FMFIA), provide 

the overall framework for establishing and maintaining internal control in the federal 

government. Also pursuant to FMFIA, the Office of Management and Budget (OMB) 

issued Circular A-123, revised December 21, 2004, to provide the specific requirements 

for assessing the reporting on internal controls. Internal control standards, and the 

definition of internal control in OMB Circular A-123, are based on GAO’s Standards for 

Internal Control in the Federal Government. 
 

Page 8 GAO-13-9 Supply Chain Security

that information, and specific dates associated with changes to CBP’s 

targeting criteria. These omissions did not affect the presentation of the 

key information and findings that support our conclusions and 

recommendations.

We conducted our work from October 2011 through October 2012 in 

accordance with generally accepted government auditing standards. 

Those standards require that we plan and perform the audit to obtain 

sufficient, appropriate evidence to provide a reasonable basis for our 

findings and conclusions based on our audit objectives. We believe that 

the evidence obtained provides a reasonable basis for our findings and 

conclusions based on our audit objectives.

CBP’s security strategy for maritime cargo uses a risk-based approach to 

focus limited resources on targeting and examining cargo shipments that 

pose a potential risk while allowing other cargo shipments to proceed 

without unduly disrupting commerce into or out of the United States. The 

strategy is based, in part, on obtaining advance cargo information. In 

particular, through what is referred to as the 24-hour rule, CBP generally 

requires vessel carriers to electronically transmit cargo manifests to CBP 

24 hours before cargo is loaded onto U.S.-bound vessels at foreign 

ports.20 Through the Importer Security Filing and Additional Carrier 

Requirements (known as the 10+2 rule), CBP requires importers and 

vessel carriers to provide data elements for improved identification of 

containerized cargo shipments that may pose a risk for terrorism.21

 

2019 C.F.R. § 4.7(b). Cargo manifests are prepared by the ocean carrier and are 

composed of bills of lading for each shipment loaded onto a vessel to describe the 

contents of the shipments. Bills of lading are documents issued by a carrier describing the 

goods, the details of the intended voyage, and the conditions of transportation.

Importers are responsible for supplying CBP with 10 shipping data 

elements—such as country of origin—24 hours prior to loading, while 

vessel carriers are required to provide 2 data elements—container status 

21Importer Security Filing and Additional Carrier Requirements, 73 Fed. Reg. 71,730 (Nov. 

25, 2008) (codified at 19 C.F.R. pts. 4, 12, 18, 101, 103, 113, 122, 123, 141, 143, 149, 

178, & 192).

Background

CBP’s Maritime Cargo 

Container Security 

Strategy
 

Page 9 GAO-13-9 Supply Chain Security

messages and stow plans22

 

22Container status messages report terminal container movements, such as loading and 

discharging the vessel, and report the change in the status of containers, such as if they 

are empty or full. Container status messages also report conveyance movements, such as 

vessel arrivals and departures. A vessel stow plan includes information such as the vessel 

operator, voyage number, the stow position of each container, hazardous material code (if 

applicable), and the port of discharge. 

—that are not required by the 24-hour rule. 

Other aspects of CBP’s maritime cargo container security strategy include 

using technology, such as radiation detection equipment, to detect 

potential threats and partnering with foreign governments and the trade 

industry to examine containers prior to their arrival in the United States 

and implement security measures throughout the supply chain process, 

respectively. A brief description of the core programs that constitute 

CBP’s security strategy for cargo containers is provided in appendix I. 

Throughout the supply chain process, and underlying many of the 

programs within CBP’s security strategy for cargo containers, CBP 

assesses the national security risks posed by shipments throughout the 

supply chain process, as shown in figure 1.
 

Page 10 GAO-13-9 Supply Chain Security

Figure 1: Key Steps for Targeting High-Risk Shipments throughout the Maritime Supply Chain Process

a

The Container Security Initiative places CBP staff at participating foreign ports to work with host 

country customs officials to target and examine high-risk cargo containers for WMD before they are 

shipped to the United States. CBP officials identify the containers that may pose a risk for terrorism 

and request that their foreign counterparts examine the contents of the containers. 

b

Host government officials at foreign ports that do not participate in the Container Security Initiative 

may also scan containers using radiation portal monitors as part of their operations. 
 

Page 11 GAO-13-9 Supply Chain Security

According to CBP, ATS is the cornerstone of CBP’s targeting efforts that

underlie the other programs that constitute CBP’s security strategy for 

maritime cargo containers. CBP targeters review shipment records in 

ATS prior to the cargo being loaded onto U.S.-bound vessels, during 

shipment transit, and upon arrival at domestic ports to identify potential 

threats and determine if additional action, such as an examination, is 

required. When shipment data are updated with additional or amended 

information, CBP targeters using ATS might identify new risks or mitigate 

previously identified risks. These targeting efforts, and their reliance on 

ATS, are key to the operations of CBP’s other security programs. For 

example, through ATS, CBP determines which shipments may be 

scanned overseas as part of the Container Security Initiative in an effort 

to prevent potentially dangerous cargo from being loaded onto U.S.-

bound vessels. Additionally, through ATS, CBP is able to reduce the 

likelihood of examinations for members of the Customs-Trade Partnership 

Against Terrorism. The Customs-Trade Partnership Against Terrorism is 

a program through which CBP provides facilitated processing, such as 

reduced likelihood of security-based examinations, for members that 

implement the program’s minimum security criteria or guidelines and best 

practices. 

The overall effectiveness of CBP’s strategy for securing maritime cargo 

containers depends largely on the effectiveness of ATS. We have 

previously reported on CBP’s efforts to assess the effectiveness of ATS 

and have made recommendations to help CBP achieve the objectives of 

its overall maritime cargo targeting strategy. In response to these 

recommendations, CBP has made improvements to ATS and its targeting 

process, but in some cases CBP’s implementation efforts have been 

slow, leaving CBP without the benefits of these improvements for several 

years. For more information about our prior work and recommendations 

regarding ATS, see appendix II.

CBP targeters use ATS in the targeting process to help them determine 

whether to take further security actions, such as holding the shipment for 

examination, for cargo shipments they are reviewing. ATS consolidates 

data from various sources to create a single, comprehensive record for 

each U.S.-bound shipment. For example, carriers and importers provide 

data in compliance with the 24-hour rule and the 10+2 rule through other 

CBP systems, and these systems automatically feed the data into ATS. 

Role of ATS in CBP’s 

Maritime Cargo Container 

Security Strategy

ATS and National Security 

Targeting Rules
 

Page 12 GAO-13-9 Supply Chain Security

ATS assesses and presents these data for every cargo shipment, 

including containerized shipments and bulk shipments.23 For 

containerized shipments, a single shipment may consist of one or more 

containers, or multiple shipments may be consolidated into a single 

container for transport. Because ATS collects and presents data on 

shipments, CBP targets shipments—rather than individual containers—for 

examination. For a shipment that includes multiple containers, CBP may 

select some containers to examine or may examine all the containers in 

the shipment. If a targeted shipment is packed into a container with other 

shipments, CBP may examine the entire container.24

ATS performs a risk-based assessment of cargo shipments that CBP 

uses to focus its resources for conducting examinations and enhance its 

ability to identify potential violations of U.S. law, possible terrorist threats, 

and other threats to border security. To do so, ATS incorporates two 

types of targeting rules—strategic and tactical—to identify risk factors in 

shipment data.

• Strategic rules: Rules that identify general intelligence or threats or 

that identify relationships among different data elements within a 

single record or across multiple records. The process to update 

strategic rules involves iterations of testing to ensure that rules have 

their intended effect.

• Tactical rules: Rules that identify risks posed by specific intelligence 

or threats and are typically based on specific entries for one or more 

shipment data elements. Tactical rules can generally be updated in 

time to react to specific intelligence.

ATS has many rules, and one set of rules within ATS is programmed to 

check for information or patterns that could be indicative of suspicious or 

terrorist activity. This set of rules is referred to collectively as the maritime 

 

23Bulk cargo is shipped loose in the hold of a ship, not in packages or containers. For 

example, grain, coal, oil, and chemicals are usually bulk cargo. Manifest data for bulk 

cargo must be submitted electronically to CBP 24 hours prior to arrival in the United 

States, rather than 24 hours prior to cargo being loaded onto U.S.-bound vessels at 

foreign ports, as is required for containerized cargo. See 19 C.F.R. § 4.7(b)(4).

24Examinations using NII equipment are conducted on containers (i.e., the resulting image 

from the exam will depict all contents of the container). If the examination includes a 

physical inspection, CBP may focus its efforts on the portion of the container that holds 

the targeted shipment.
 

Page 13 GAO-13-9 Supply Chain Security

national security weight set (the weight set) because each rule in the set 

has a specific weight value assigned to it, and for each risk factor that 

rules identify, the weight values are added together to calculate an overall 

risk score for the shipment. For example, some rules in the weight set 

determine if any of the supply chain parties have possible matches to 

known terrorists or previous violations of U.S. law, and other rules in the 

weight set evaluate the completeness of the data, for example, whether 

addresses are provided as required for supply chain parties listed in the 

data, such as the importer. The weight set includes both strategic and 

tactical rules.

CBP classifies the risk scores from the weight set as low, medium, or high 

risk. Shipments with connections to known or suspected terrorists as well 

as those that include invalid information are more likely to be classified as 

high risk, and shipments from trusted shippers who participate in CBP’s 

Customs-Trade Partnership Against Terrorism are more likely to be 

classified as low risk.

CBP can make updates to the weight set when the need arises. 

According to CBP, some updates, such as responding to specific 

intelligence information, can be implemented in a short amount of time. 

Substantial updates to the weight set, such as integrating the 10+2 rule 

data, can take longer to develop and implement. As part of this process, 

CBP may add new rules, change existing rules, or adjust the weights 

assigned to rules. To make such substantial updates to the weight set, 

CBP uses a multistep process that includes (1) identifying new 

requirements, (2) designing alternative versions of the weight set that 

may address the new requirements in different ways, (3) testing and 

evaluating those alternative versions, (4) selecting and implementing the 

new version to replace the existing version of the weight set, and (5) 

monitoring the performance of the new version of the weight set after 

implementation. According to CBP’s Rules Development Concept of 

Operations, which describes the processes for updating the weight set, a 

substantial update provides the opportunity for a full assessment and 

evaluation of alternatives before CBP implements a new version of the 
 

Page 14 GAO-13-9 Supply Chain Security

weight set. CBP completed the most recent substantial update to the 

weight set in early 2011.25

Within CBP, targeters are stationed at different locations with varying 

responsibilities and focuses depending on their location, as described 

below.

• Container Security Initiative ports: CBP places targeters at 

designated foreign ports to work with their foreign counterparts to 

identify shipments that may pose a high risk for containing WMD or 

other terrorist contraband before the shipments are loaded onto U.S.-

bound vessels.26

• Advance Targeting Units (ATU): ATUs are located at select 

domestic ports, and their targeting efforts are focused on shipments 

destined for ports within their respective regions. An ATU may be 

responsible for targeting shipments arriving at multiple ports in a 

region. For example, targeters at the Port of Houston are also 

responsible for targeting shipments that are bound for Freeport, 

Texas. CBP targeters at ATUs can review data as soon as carriers 

and importers submit the required data in accordance with the 24-

hour rule and the 10+2 rule, and the data are available in ATS. After 

When CBP targeters at a Container Security 

Initiative port determine through advance information that a U.S.-

bound cargo shipment poses a high risk, CBP typically requests that 

the host government scan the container(s) with radiation detection 

and NII equipment. If these scans indicate the potential presence of 

WMD or other contraband, CBP is to request that the host 

government conduct a physical inspection of the shipment. If the host 

government declines, CBP can either issue a “do not load” order to 

prevent the shipment from being loaded onto a U.S.-bound vessel or 

flag the shipment for further examination upon arrival at the domestic 

port.

 

25In February 2009, CBP convened a conference of subject matter experts, including 

targeters at domestic ports, to discuss integrating the 10+2 rule data into the weight set. 

Following the conference, CBP designed and evaluated five alternative versions of the 

weight set. In early 2011, CBP selected one of these alternative versions and 

implemented it, making it the current version of the weight set that CBP targeters use in 

their targeting efforts.

26The Container Security Initiative has been operational at 58 foreign ports since fiscal 

year 2007.

National Security 

Targeting Responsibilities 

within CBP
 

Page 15 GAO-13-9 Supply Chain Security

reviewing the data, if the shipment data indicate a substantial risk that 

cannot be mitigated through an overseas examination at a Container 

Security Initiative port or by obtaining additional information about the 

shipment through additional research, the targeters may seek 

approval for a “do not load” order from the NTC-C Director before a 

shipment is loaded onto a U.S.-bound vessel. Once a shipment is 

loaded onto a vessel, targeters continue to review shipment data in 

ATS and use other sources, such as public records, to assess 

whether the shipment could pose a risk, in which case a targeter may 

target the shipment for examination upon arrival.

• NTC-C: In contrast to targeters at Container Security Initiative ports 

and ATUs, who focus their reviews on those shipments that are 

transiting from or to their respective ports, targeters at NTC-C review 

shipments for security risks from a national perspective. For example, 

if there is specific intelligence regarding an attempt to smuggle a 

terrorist weapon in a container, NTC-C targeters can use ATS to

identify whether any shipments destined for the United States match 

the intelligence information, regardless of the port of arrival. NTC-C 

targeters also serve as a resource for other CBP targeters stationed 

at foreign and domestic ports because targeters at NTC-C may have 

access to research tools, such as classified databases, that may not 

be available to CBP targeters at these other locations.

ATS is the primary system that CBP targeters use to review maritime 

cargo shipments for national security purposes, and targeters we spoke 

with were generally satisfied with how ATS and its weight set of national 

security rules have assisted in their targeting efforts. For example, 

targeters at one ATU we visited said that because ATS filters information 

and presents key information to the targeters, the targeters are able to 

better focus their targeting efforts than before they had ATS.27

 

27CBP introduced ATS in 1999.

Those 

targeters as well as the ATU supervisor at another ATU noted that, in 

particular, the risk scores that the current version of the weight set 

produces are helpful in balancing their targeting workload. The risk score, 

however, is not the sole factor that determines whether a targeter reviews 

the data for a shipment or whether the shipment is selected for a security 

examination. In particular, targeters at each of the six ATUs we visited 

explained that they use the ATS risk score as a starting point for the 

targeting process, but that their decisions are ultimately based on 

CBP Uses ATS and 

Other Tools to Target 

Maritime Cargo 

Shipments for 

National Security 

Purposes
 

Page 16 GAO-13-9 Supply Chain Security

additional research. To conduct this research, they may use information 

within ATS or other tools and information outside of ATS. On the basis of 

the ATS risk score and the research conducted, targeters make a 

qualitative assessment of the risk and determine whether to hold a 

shipment for examination. Targeters at one of the ATUs we visited 

emphasized the important role that targeters’ expertise and experience 

play in the risk assessment process, stating that although they believe 

ATS’s capabilities are helpful, they believe that there could be negative 

effects from further automating the targeting process, such as decreased 

use of targeters’ expertise regarding the different types of shipments that 

arrive at their respective ports.28

According to CBP policy, the risk scores assigned by the weight set in 

ATS determine, in part, what actions CBP officers at the ports are to take 

to address potential threats. Targeters at ATUs are required to review 

data in ATS for all medium-risk and high-risk shipments that arrive at their 

respective ports. For example, a targeter may review individual data 

elements, such as the name of the importer or other supply chain parties. 

A targeter may also review the rules that detected potential threats and, 

therefore, contributed to the calculation of the risk score.29 ATU targeters 

are also required to hold high-risk shipments for examination unless they 

can mitigate the risk through additional research or analysis of available 

information.30

 

28We have previously reported similar findings related to CBP’s targeting for intellectual 

property violations. In particular, we reported in April 2007 that CBP officials at several 

ports we visited expressed the view that there is no substitute for the skills and experience 

of a well-trained CBP officer; however, other officials noted that automated systems can 

assist with targeting because they can better handle data for vast volumes of shipments. 

For more information, see GAO, Intellectual Property: Better Data Analysis and Integration 

Could Help U.S. Customs and Border Protection Improve Border Enforcement Efforts, 

CBP targeters at each of the six ATUs we visited 

demonstrated how they implement this policy at their respective ports. In 

addition to actions targeters take in accordance with CBP policy, targeters 

have discretion over which low-risk and medium-risk shipments to select 

GAO-07-735 (Washington, D.C.: Apr. 26, 2007).

29If a rule detects a potential threat based on the data for a shipment, the weighted value 

for that rule contributes to the calculation for the shipment’s risk score. If a rule does not 

detect a potential threat in the data, the rule does not contribute any points to the risk 

score. 

30Although it is possible to get a waiver to exempt a high-risk shipment from examination 

based on information collected, CBP policy states that such waivers should be used 

judiciously and only when based upon articulable reasons.
 

Page 17 GAO-13-9 Supply Chain Security

for security examinations, and CBP expects targeters to select shipments 

based on discretionary factors.31

Targeters use various features within ATS to assist them in their research 

into shipments of interest. Officials at each of the six ATUs we visited 

discussed or demonstrated the following features and how targeters use 

these features when targeting:

• Queries: Through ATS’s querying capabilities, targeters are able to 

search for shipments that meet specific criteria, such as shipments 

from a particular country. For example, targeters at each ATU we 

visited use queries to identify shipments for review, although the set of 

queries used varied at each ATU. Five of the six ATUs we visited use 

queries to ensure that all shipments, regardless of risk score, are 

reviewed prior to arrival. For example, targeters at one ATU run a 

query for each arriving vessel to ensure that all shipments on the 

vessel have been reviewed. Targeters at another ATU have a query 

for each risk level (high, medium, or low) and targeters reviewing the 

results of each query are to ensure that all shipments of a particular 

risk level have been reviewed. In addition, targeters at five of the six 

ATUs we visited said that they run additional queries of interest for 

discretionary targeting after completing their assigned duties. Such 

discretionary targeting could be for national security purposes or for 

other efforts, such as counternarcotics. For example, targeters at one 

ATU may independently create queries to identify items of interest, 

such as all shipments of a particular commodity or those coming from 

 

31In DHS’s Annual Performance Report for fiscal years 2008 through 2010, DHS reported 

that CBP did not meet its target for the percentage of maritime cargo containers scanned 

for contraband in fiscal year 2008. DHS reported that one of the reasons for not meeting 

the target was that updates to ATS targeting rules resulted in fewer mandatory 

examinations based on ATS’s assessment of a shipment as high risk, and that CBP did

not compensate for this decrease by increasing the number of discretionary (CBP officer 

targeted) exams. DHS reported that CBP planned to increase the number of discretionary 

exams. According to a CBP official responsible for tracking this information, CBP 

exceeded its target in fiscal years 2009 through 2011. For example, in fiscal year 2011, 

CBP scanned 4.13 percent of maritime cargo containers, which exceeded CBP’s goal of 

3.80 percent. The containers scanned included those that were targeted based on their 

risk score or other risk factors identified by targeters, as well as for other reasons. For 

example, CBP officials at all six ATUs we visited said that their ports examine some 

containers regardless of the risk assessed by ATS or a targeter (e.g., scan every seventh 

container being unloaded from a vessel). CBP also has a Compliance Measurement 

Program that supplements ATS by randomly selecting shipments to be inspected to 

determine whether the shipments comply with supply chain security and trade laws. 
 

Page 18 GAO-13-9 Supply Chain Security

a particular country of origin. In addition to queries that targeters at 

the ATUs run, NTC-C targeters run nationwide queries daily to identify 

shipments with the potential for containing chemical, biological, 

radiological, nuclear, or conventional weapons, among other things.

• Targeters’ notes: ATS has a feature that allows a targeter to 

annotate a shipment with the targeter’s conclusion based on research 

regarding whether the shipment is considered a potential threat. The 

notes feature within ATS facilitates the sharing of research findings 

with CBP targeters at other locations.

• Targeters’ reviews: ATS also indicates whether the shipment data 

have been reviewed by a targeter at the targeter’s own location or at 

another CBP targeting location, such as a Container Security Initiative 

port or NTC-C.

CBP targeters also use tools outside of ATS to conduct research. During 

our interviews at the six domestic port ATUs we visited, targeters 

explained that they use web-based and other research tools to aid in their 

assessments of shipments. Such tools include web searches, which 

targeters use to find general information on a company or address; a 

third-party database of public and proprietary records, which targeters 

use to research business names and associated information such as a 

business’s locations, officers, and assets (e.g., registered vehicles); and 

the State Department’s Consular Consolidated Database, which targeters 

may use to obtain visa and passport information for foreign individuals 

involved in a shipment. Targeters review and analyze all of the 

information collected to make a decision as to whether a shipment should 

be examined. On the basis of such research and analysis, a targeter 

could select a low-risk shipment for examination. A targeter could also 

determine that an examination is not necessary for a medium-risk 

shipment—for example, the weight set may assign a medium-risk score 

to a shipment based on the data available, but the targeter could 

determine through research that the score is based on a clerical error in 

the data provided.

Targeters’ experience may also inform targeting decisions. For example, 

targeters at ATUs may have information about recent seizures and can 

look for recurring patterns to identify future shipments that may be part of 

a trend of illegal shipments. Targeters may also share such information 

with other targeting units to help inform targeting decisions. Also, 

targeters at all six of the ATUs we visited said they communicate regularly 

with targeters at NTC-C regarding shipments of interest.
 

Page 19 GAO-13-9 Supply Chain Security

We have previously reported that ensuring controls to assess ATS’s 

effectiveness in identifying high-risk shipments was important for 

providing CBP with the best information to inform its targeting efforts.32 In 

2005, in response to our work and an external peer review33 of ATS 

conducted in 2005, CBP contracted with a consulting firm to develop 

performance measures and a methodology to determine the effectiveness 

of the weight set in identifying high-risk shipments. The resulting 

performance measures and methodology, which the consulting firm 

provided to CBP in April 2006, balanced targeting accuracy with 

examination workload and enabled CBP to compare the weight set’s 

performance with the effectiveness of examinations conducted through a 

random selection program. We then reported in August 2006 that the 

performance measures and methodology developed by the consulting 

firm were sufficient to assess the performance of the weight set and 

provide a baseline against which future assessments may be 

conducted;34 however, we also reported that data limitations and 

uncertainties existed, and we noted that CBP must interpret the 

evaluations cautiously.35

 

32Because the report we issued in August 2006 regarding CBP’s targeting practices 

contains sensitive information, it is not publicly available.

We also reported in August 2006 that CBP 

33An external peer review is a process that includes an assessment of the model by 

independent and qualified external experts. 

34In 2006, CBP referred to its two performance measures as the estimated accuracy rate 

and the estimated inspection rate, respectively. In more recent assessments, it refers to 

these measures as the true positive rate and the false positive rate, respectively. In 2006, 

CBP also calculated the relative performance factor, which provides an assessment of 

how much better or worse ATS performs at targeting oceangoing cargo shipped in 

containers than a random selection method.

35To determine which shipments to use for the calculations, the consulting firm developed 

a proxy positive definition of shipments that contain indications of materials or behaviors 

that are believed to be similar in physical attributes, magnitude, and intent to the materials 

and behavior of shipments associated with terrorist threats. The firm had to develop this 

definition because CBP inspections of cargo shipped in containers at that time had not 

resulted in the identification of any direct terrorism threats. Because CBP had not 

identified any terrorist-related materials in shipments at the time this report was written, 

there was no way to validate the proxy positive shipments.

CBP’s Efforts to 

Assess the Weight Set 

Have Been Limited

CBP Developed 

Performance Measures to 

Assess the Weight Set but 

Is Continuing to Update 

the Methodology
 

Page 20 GAO-13-9 Supply Chain Security

planned to continue using the consulting firm’s methodology in making 

future adjustments to the weight set. For more information about GAO’s 

past audit findings and recommendations to improve the targeting 

process, see appendix II. 

Currently, CBP assesses the performance of the weight set using the 

following performance measures:

• True positive rate (TPR) which reflects the percentage of maritime 

shipments that ATS assessed as high risk within the population of 

shipments in which CBP identified a threat during an examination.

• False positive rate (FPR) which reflects the percentage of maritime 

shipments that ATS assessed as high risk within the population of 

shipments in which CBP did not identify a threat during an 

examination.

Taken in combination, the TPR and FPR measures enable CBP to 

determine the effectiveness of the weight set by providing information 

about the accuracy of the weight set and its impact on examination 

workload. The TPR enables CBP to determine the accuracy of the weight 

set in identifying high-risk shipments. The TPR and FPR measures also 

enable CBP to determine the workload or examination rate for ports 

based on the results of the weight set scores. For example, a high FPR 

would unnecessarily increase the workload or number of examinations at 

ports because officials would be required to examine a higher number of 

shipments that do not contain an actual threat. Using data from the 

version of the weight set CBP was using in 2005, the consulting firm used 

its methodology to conduct a performance assessment, which involved 

calculating the TPR and FPR to ultimately indicate the effectiveness of 

the weight set at that time. Since the contract with the firm ended in 2011, 

CBP has taken on the role of assessing the effectiveness of the weight 

set. In its project plan for the most recent update to the weight set 

implemented in early 2011, as it had previously done, CBP planned to 

assess the performance using these measures to compare the targeting 

effectiveness of the weight set with other measurements.

CBP officials stated that they face ongoing challenges with the 

performance measures and the methodology by which they are 

calculated. In particular, the FPR and TPR may not accurately reflect the 

weight set’s performance in identifying national security threats because 

they rely on indirect measures given that no true security threat has been 

found in a cargo container. We reported in 2006 that CBP planned to take 

steps to improve the process for assessing ATS performance. Since the 
 

Page 21 GAO-13-9 Supply Chain Security

most recent update to the weight set in early 2011, CBP formed working 

groups and has begun taking steps to ensure that the methodology it 

uses to approximate threats accurately reflects what CBP considers to be 

a national security threat (to the extent possible). For example, these 

working groups plan to (1) create a new definition of “national security” for 

the purposes of clarifying what the weight set should target and (2) revisit 

the current definitions of threats in containers to provide consistency with 

the new definition of national security. CBP officials stated that they 

expect the working groups’ activities to be completed by April 2013. 

Prior to implementing the current version of the weight set in early 2011, 

CBP did not conduct an assessment to determine whether the updated 

version of the weight set would be more effective than the previous 

version of the weight set or other alternatives that were considered during

the update process. For the 2011 update, CBP developed and evaluated 

five alternative versions of the weight set. CBP’s consideration for which 

alternative to select focused on two of the five versions because, 

according to CBP officials, these two versions incorporated the newly 

required 10+2 data and reflected current threat information about 

countries of interest. CBP ultimately implemented one of these two 

versions of the weight set, but CBP could not provide any documentation 

to demonstrate that the version selected was more effective than either 

the other alternative or the version it was to replace.

CBP’s process for updating the weight set involves assessing the impact 

of alternative versions of the weight set. For example, for the most recent 

update to the weight set, CBP’s impact assessment provides information 

on how many shipments would be assessed as high risk under each 

alternative version of the weight set and would, therefore, affect CBP 

targeters’ workload at ports of arrival because such high-risk shipments, 

under CBP policy, are to be held for examination, for example, through 

the use of NII equipment. CBP officials stated that they believed the 

impact assessment that CBP conducted during the update process 

indicates the reasons for selecting the chosen version of the weight set.

While, according to CBP officials, the impact assessment provides CBP’s 

reason for replacing the prior version of the weight set, we found that the 

impact assessment primarily evaluates how the chosen alternative 

version of the weight set could affect targeter workload and does not 

address measures of accuracy in identifying high-risk shipments. 

Therefore, the impact assessment does not fully account for the 

effectiveness of each alternative of the weight set. Although managing 

CBP Does Not Have 

Reasonable Assurance 

That the Updated Weight 

Set Is More Effective than 

Alternative Versions or the 

Version It Replaced
 

Page 22 GAO-13-9 Supply Chain Security

resources is an element of risk management, effectiveness in reducing 

risks is also an important consideration when evaluating alternatives to 

manage risk. CBP’s impact assessment does not address the balance 

between targeting accuracy and workload.

Assessing the potential effectiveness of alternative versions of the weight 

set prior to selecting one for implementation would provide CBP with 

more information to make an informed decision. In January 2010, the 

DHS Office of Inspector General recommended that CBP enhance its 

documentation efforts to ensure that each stage of the process for 

analyzing and developing ATS rules is documented, and CBP concurred 

with this recommendation.36 As part of this recommendation, the DHS 

Office of Inspector General recommended documenting the rationale for 

making changes to ATS rules but did not specify what types of analyses 

could demonstrate or support the rationale for making changes to the 

rules. On the basis of our analysis, the rationale for updates to the weight 

set could be further strengthened through assessments of effectiveness 

beyond workload. For example, determining the expected TPR and FPR 

for an alternative version of the weight set and comparing these 

measures against the TPR and FPR for the existing version of the weight 

set could enable CBP to determine if the alternative version of the weight 

set could be expected to result in improved effectiveness, based on these 

performance measures. This would enable CBP to quantitatively compare 

the effectiveness of the alternative versions of the weight set being 

considered prior to selecting one for implementation. Doing so, in addition 

to the impact assessment, would provide CBP with reasonable assurance 

that the version of the weight set it selects for implementation is the most 

effective of the alternatives considered after taking into account any 

resource constraints. Furthermore, assessing the alternative versions of 

the weight set in the future would provide CBP with better assurance that 

the version it selects for implementation is more effective than the 

previous version of the weight set. CBP officials stated that they plan to 

calculate and document measures of effectiveness during the planned 

update to the weight set that will begin in the fall of 2012.

 

36DHS Office of Inspector General, Cargo Targeting and Examinations, OIG-10-34 

(Washington, D.C.: Jan. 6, 2010).
 

Page 23 GAO-13-9 Supply Chain Security

Since implementing the current version of the weight set in early 2011, 

CBP has not regularly assessed the weight set against established 

performance targets to monitor its performance and obtain information to 

determine when updates to the weight set are necessary. We reported in 

August 2006 that CBP intended to establish targets for the performance 

measures to assess future performance of ATS, but CBP did not 

establish such targets for those measures.

Targets could help CBP determine when updates are needed to improve 

targeting effectiveness. For example, according to CBP’s analysis, the 

TPR for summer 2011 through spring 2012 shows that, among shipments 

CBP found to contain a potential threat during an examination, the weight 

set accurately identified 6.3 percent as high risk, meaning that the weight 

set classified 93.7 percent of shipments that carried a potential threat as 

either medium risk or low risk. Furthermore, the FPR for that time period 

shows that, among all the arriving shipments that CBP examined during 

that time that did not pose a threat, the weight set identified 3.6 percent 

as high risk. However, because CBP did not establish targets for either 

TPR or FPR, it is not clear whether 6.3 percent for the TPR is sufficiently 

low or 3.6 percent for the FPR is sufficiently high to suggest that changes 

are needed to improve the performance of the weight set.

CBP’s project plan calls for conducting periodic performance 

assessments by determining recurring measures of TPR and FPR. 

Furthermore, according to CBP officials, the performance assessments 

are to be conducted as part of quarterly reporting responsibilities. 

However, CBP did not calculate these measures at the end of each 

quarter, but instead calculated them as part of a single assessment in the 

summer of 2012 and divided the results into quarters.37

 

37We compared the results of assessments for the previous version of the weight set with 

the results of assessments for the current version of the weight set. For the previous 

version of the weight set, the TPR was about 20 percent from the end of fiscal year 2009 

through fiscal year 2010, whereas the TPR for the current weight set has generally been 

below 10 percent from the third quarter of fiscal year 2011 through the third quarter of 

fiscal year 2012. According to CBP officials, the measures for each quarter are calculated 

using data from the prior 1-year period. However, according to CBP officials, the 

methodologies used to assess each weight set were different, and because it is not clear 

to what extent this change in methodology accounts for any changes in the performance 

measures, it is not possible to make a meaningful comparison between the results.

Accordingly, CBP 

was not aware of the ongoing performance of the weight set from its 

implementation in early 2011 through spring 2012, and CBP was 

CBP Has Not Regularly 

Assessed the Weight Set 

against Performance 

Targets to Determine when 

Updates Are Needed
 

Page 24 GAO-13-9 Supply Chain Security

therefore unable to determine for 18 months whether the weight set was 

performing at a level that could require changes or updates to improve its 

effectiveness. According to CBP officials, the summer 2012 assessment 

was conducted at that time in preparation for a conference to discuss 

updates to the weight set planned for the fall of 2012. CBP had decided to 

hold this conference before CBP conducted the assessment, meaning its 

decision for when to update the weight set was not based on information 

about the weight set’s effectiveness from ongoing monitoring of CBP’s 

performance measures for the weight set.

Ongoing monitoring is a key element of a risk management approach, 

and CBP’s project plan calls for such periodic performance monitoring to 

determine targeting effectiveness. In addition, standard practices for 

internal control indicate that (1) ongoing monitoring should occur in the 

course of normal operations and can be accomplished by periodic review 

of performance measures and (2) in the process of ongoing performance 

monitoring, actions should include continuous comparison of performance 

data against planned targets and analysis of any differences to take 

corrective actions as necessary.

CBP officials stated that personnel have not been consistently available 

to conduct performance assessments since the initial contract with the 

consulting firm ended in July 2011 and that resource concerns, such as 

funding, the availability of subject matter experts, and the availability of 

programmers may affect the timing of weight set updates. Nevertheless, 

given the importance of the weight set to CBP’s process for targeting 

cargo containers, regular performance assessments of the weight set that 

include evaluating results against established performance targets could 

help CBP determine when updates are needed in a timelier manner and 
 

Page 25 GAO-13-9 Supply Chain Security

help it better prioritize the resources it needs to complete the updates.38

Furthermore, CBP officials stated that they intend to continue adjusting 

the methodology for calculating the performance measures to mitigate 

data limitations and more accurately reflect the performance of the weight 

set. Such steps could help CBP ensure that its targeters have the best 

information available regarding the risk of maritime cargo container 

shipments arriving in the United States.

CBP recognizes the importance of and challenges to ensuring the 

security of the global supply chain while facilitating the flow of legitimate 

commerce. Although no events have occurred to date, terror-related 

attacks on the supply chain could have devastating effects on the nation’s 

security and economic well-being, and it is imperative that CBP use the 

best information and tools available to continually mitigate potential 

threats and address vulnerabilities. DHS and CBP face difficulties in 

achieving 100 percent scanning of cargo containers prior to loading at 

foreign ports and have, instead, advocated a risk-based approach to 

target and scan those cargo containers that pose the highest risk. Given 

the critical role that ATS plays as part of this risk-based approach, it is 

important to ensure that ATS is performing effectively.

CBP plans to continue enhancing risk management processes, including 

the use of ATS and its associated targeting rules. CBP’s determination of 

which containerized shipments to review or to hold for examination is 

based, in part, on the risk score. Thus, updating the weight set in ATS 

that calculates this risk score is important for ensuring that targeters are 

 

38In the past, CBP’s decision to update the performance of the weight set was dependent 

upon feedback from targeters or external reasons—such as the receipt of new data 

because of the 10+2 rule. However, feedback from targeters is not systematic and may 

not provide a reliable assessment of the weight set’s ability to identify high-risk containers. 

CBP utilizes ATU targeters at rules conferences to assist in rule and weight set updates 

because they use ATS for daily operations. CBP receives weight set input from ATU 

targeters from a variety of ports nationally. CBP headquarters officials consider all input 

and determine the best way to use the information to indicate risk. In some cases, 

changes are made to rules that will affect targeting nationwide, whereas in other cases, 

rule changes may not be implemented because they might have a negative effect across 

all ports. Further, in some cases, rules that affect only a single port or region may be 

added to a weight set. According to CBP officials, they do not rely on a particular amount 

of input to initiate the process of updating the weight set; rather, they subjectively 

determine the point at which the feedback is substantial enough to warrant an evaluation 

of the weight set.

Conclusions
 

Page 26 GAO-13-9 Supply Chain Security

using the most effective tools in making targeting decisions. CBP has 

assessed workload impacts when making updates to the weight set, but it 

did not fully assess the weight set’s effectiveness as part of the most 

recent update. As a result, CBP does not have reasonable assurance that 

the implemented version is the most effective. Further, CBP did not 

conduct periodic assessments as part of ongoing monitoring efforts. 

Specifically, CBP did not conduct an assessment of the weight set until 

18 months after CBP implemented the new weight set. We believe it is 

important that CBP more regularly assess the performance of the weight 

set in ATS that produces the risk scores and compare the results of this 

assessment against established performance targets. Such steps could 

help CBP determine when changes may be needed and ensure that its 

targeters have the best information available regarding the risk of 

maritime cargo container shipments arriving in the United States.

To enhance its targeting of maritime cargo containers and better position 

CBP to provide reasonable assurance of the effectiveness of ATS, we 

recommend that the Commissioner of CBP take the following two actions:

• ensure that future updates to the weight set are based on results of 

assessments that demonstrate that the chosen version of the weight 

set is more effective than other alternatives, including the existing 

version, and

• establish targets for CBP’s performance measures and use those 

measures to assess the effectiveness of the weight set on a regular 

basis to better determine when updates to the weight set are needed.

On October 17, 2012, DHS provided written comments on a draft of this 

report, which are reprinted in appendix III. DHS concurred with the two 

recommendations. Specifically, DHS concurred with the recommendation 

to ensure that future updates to the weight set are based on the results of 

assessments and stated that CBP plans to conduct analyses to ensure 

that future versions of the weight set result in increased effectiveness. 

DHS also noted that CBP is to conduct these analyses during the 

development and deployment of future versions of the weight set. 

According to DHS, these analyses would include performance measures, 

subject matter expert input, current threat information, and other 

intelligence. DHS stated that it expects these actions to be completed by 

April 2013. Such actions should address the intent of the 

recommendation to ensure improvements in the effectiveness of future 

versions of the weight set. DHS also concurred with the recommendation 

Recommendations for 

Executive Action

Agency Comments
 

Page 27 GAO-13-9 Supply Chain Security

to establish targets for CBP’s performance measures and stated that CBP 

is working to improve the current performance measures methodology. 

DHS stated that, following approval of this methodology, CBP plans to 

conduct quarterly reviews of the weight set to inform decision making. 

DHS stated that it expects these actions to be completed by September 

2013. If CBP takes these steps as planned and includes targets for any 

performance measures that are part of the updated methodology, this 

should address the intent of our recommendation. DHS also provided 

technical comments, which we incorporated as appropriate.

As agreed with your office, unless you publicly announce its contents 

earlier, we plan no further distribution of this report until 30 days after its 

issue date. At that time, we will send copies of this report to the Secretary 

of Homeland Security, appropriate congressional committees, and other 

interested parties. In addition, the report will be available at no charge on 

the GAO Web site at http://www.gao.gov.

If you or your staffs have any questions on this report, please contact me 

at (202) 512-9610 or caldwells@gao.gov. Contact points for our Offices of 

Congressional Relations and Public Affairs may be found on the last page 

of this report. Staff acknowledgments are provided in appendix IV.

Stephen L. Caldwell

Director

Homeland Security and Justice
Appendix I: Federal Strategy for Ensuring the 

Security of Maritime Cargo Container 

Shipments

Page 28 GAO-13-9 Supply Chain Security

This appendix describes the core programs related to U.S. Customs and 

Border Protection’s (CBP) strategy for ensuring the security of maritime 

cargo container shipments. CBP has developed this strategy to mitigate 

the risk of weapons of mass destruction, terrorist-related material, or 

other contraband being smuggled into the United States in cargo 

containers. CBP’s strategy is based on related programs that attempt to 

focus resources on potentially risky cargo shipped in containers while 

allowing other cargo containers to proceed without unduly disrupting 

commerce into the United States. The strategy includes obtaining 

advanced cargo information to identify high-risk containers, using 

technology to inspect cargo containers, and partnering with foreign 

governments and the trade industry. Table 1 provides a brief description 

of the core programs that compose this security strategy.

Table 1: Description of CBP’s Core Cargo Security Programs

Program and year introduced Description 

Obtaining advanced information to identify high-risk containers 

Automated Targeting System (ATS), 1999 ATS is an intranet-based enforcement and decision support system that compares 

traveler, cargo, and conveyance information against intelligence and other 

enforcement data by incorporating risk-based targeting scenarios and 

assessments. ATS assigns a risk score to arriving cargo shipments based on 

shipping information to help CBP identify and prevent potential terrorists and 

terrorist weapons from entering the United States.

24-hour rule, 2002 CBP generally requires vessel carriers to electronically transmit cargo manifests to 

CBP’s Automated Manifest System 24 hours before U.S.-bound cargo is loaded 

onto a vessel at a foreign port. The information is used by ATS in its calculation of 

risk scores. The cargo manifest information is submitted by vessel carriers for all 

arriving cargo shipments. 

Importer Security Filing and Additional Carrier 

Requirements (also known as 10+2), 2009 

CBP requires importers and vessel carriers to provide data elements for improved 

identification of containerized shipments that may pose a risk for terrorism. The 

importer is responsible for supplying CBP with 10 shipping data elements, such as 

country of origin, 24 hours prior to loading, while the vessel carrier is required to 

provide two data elements, container status messages and stow plans, not 

required by the 24-hour rule.a

Domestic scanning technology deployments 

 

Nonintrusive inspection (NII) equipment, 2001 CBP uses NII equipment to actively scan both randomly selected containers and 

those identified by ATS as high risk. NII uses X-rays or gamma rays to scan a 

container and create images of the container’s contents without opening it. 

According to CBP, as of August 2012, it had deployed 93 NII systems to U.S. ports 

to scan containers. In fiscal year 2011, CBP scanned 4.13 percent of containers 

arriving at U.S. ports. 

Radiation portal monitors, 2002 CBP’s program to scan 100 percent of containers arriving in the United States with 

radiation detection equipment prior to leaving a domestic port. As of July 2012, 

DHS had deployed 447 radiation portal monitors at U.S. seaports, through which 

approximately 99 percent of all containers arriving by sea pass. 

Appendix I: Federal Strategy for Ensuring the 

Security of Maritime Cargo Container 

Shipments 
Appendix I: Federal Strategy for Ensuring the 

Security of Maritime Cargo Container 

Shipments

Page 29 GAO-13-9 Supply Chain Security

Program and year introduced Description 

Partnerships with foreign governments and the trade industry

Container Security Initiative, 2002 CBP places staff at participating foreign ports to work with host country customs 

officials to target and examine high-risk container cargo for weapons of mass 

destruction before they are shipped to the United States. CBP officials identify the 

containers that may pose a risk for terrorism and request that their foreign 

counterparts examine the contents of the containers. 

Secure Freight Initiative, 2006 CBP and the Department of Energy initiated this program at selected ports to scan 

100 percent of U.S.-bound container cargo for nuclear and radiological materials 

overseas using integrated examination systems that couple NII and radiation 

detection equipment. Since its inception, all but one of the selected ports has 

reverted to Container Security Initiative operations.

Customs-Trade Partnership Against Terrorism, 

2001 

CBP develops voluntary partnerships with members of the international trade 

community composed of importers; manufacturers; customs brokers; forwarders; air, 

sea, and land carriers; and contract logistics providers. Private companies that 

implement specific security measures and best practices receive facilitated 

processing, such as a reduced likelihood of security-based examinations of their 

cargo. 

Mutual recognition arrangements, various 

years

Through mutual recognition arrangements with other countries, the security-related 

practices and programs taken by the customs administration of one country are 

recognized and accepted by the administration of another. According to CBP, the 

essential concept is that the Customs-Trade Partnership Against Terrorism and the 

foreign programs are compatible in both theory and practice so that one program 

may recognize the findings and validation results provided by another.

b

Source: GAO summary of information provided by DHS.

a

Container Status Messages report terminal container movements, such as loading and discharging 

the vessel, and report the change in the status of containers, such as if they are empty or full. The 

stow plan contains the position of each cargo container on a vessel. 

b

These mutual recognition arrangements were individually negotiated and signed in different years. 

CBP has signed arrangements with New Zealand (2007), Canada (2008), Jordan (2008), Japan 

(2009), Korea (2010), and the European Union (2012). 
Appendix II: Past Audit Findings and 

Recommendations to Improve the Targeting 

Process 

Page 30 GAO-13-9 Supply Chain Security

Since 2004, we have conducted audits of CBP’s targeting process and 

ATS. In particular, we published reports related to these topics in 

February 2004, August 2006, and September 2010.1 In addition, the 

Coast Guard and Maritime Transportation Act of 2004 requires the 

Department of Homeland Security’s (DHS) Office of Inspector General 

(OIG) to report annually on its evaluation of the current targeting system 

for international intermodal cargo containers.2 Our and DHS OIG’s audits 

have addressed, among other things, incorporating key elements of a risk 

management framework and recognized modeling practices, 

documenting the targeting rule development process, improving data 

collection and ATS’s use of data, and providing additional information 

outside of ATS to targeters. Collectively, these audits have made 

recommendations to DHS and CBP for improving the targeting process, 

and CBP has taken actions to implement them, although in some cases 

CBP’s implementation efforts have been slow, leaving CBP without the 

benefits of these improvements for several years. This appendix provides 

an overview of key findings and recommendations from these audits, as 

well as the status of actions taken to implement the recommendations.3

In February 2004, we reported that while CBP had taken steps to address 

the terrorism risks posed by maritime cargo containers, its targeting 

strategy did not incorporate all key elements of a risk management 

framework, and ATS was not consistent with certain recognized modeling 

practices. We recommended, among other things, that CBP improve the 

targeting strategy by incorporating key elements of a risk management 

framework and recognized modeling practices. These recommendations 

included specific steps CBP should take to help ensure it could achieve 

the objectives of its overall targeting strategy and better ensure that the 

tools it uses to protect against terrorism are working effectively at the 

 

1

GAO, Supply Chain Security: CBP Has Made Progress in Assisting the Trade Industry in 

Implementing the New Importer Security Filing Requirements, but Some Challenges 

Remain, GAO-10-841 (Washington, D.C.: Sept. 10, 2010). Because the reports we issued 

in February 2004 and August 2006 regarding CBP’s targeting practices contain sensitive 

information, they are not publicly available.

2

Pub. L. No. 108-293, § 809(g), 118 Stat. 1028, 1087.

3

Some recommendations from these products addressed other elements of CBP’s cargo 

container security strategy, such as policies and procedures for examining cargo 

containers. The scope of our review focused on ATS and CBP’s targeting efforts, and 

therefore the recommendations listed in this appendix are those that fall within that scope.

Appendix II: Past Audit Findings and 

Recommendations to Improve the Targeting 

Process 

Incorporating Key 

Elements of a Risk 

Management Framework 

and Recognized Modeling 

Practices
Appendix II: Past Audit Findings and 

Recommendations to Improve the Targeting 

Process 

Page 31 GAO-13-9 Supply Chain Security

nation’s ports. We later reported, in August 2006, that CBP had made 

progress addressing the recommendations but had not yet fully 

implemented the recommendations in our February 2004 report.4

Table 2: Status of GAO Recommendations from February 2004

Since 

that time, CBP has fully implemented the February 2004 

recommendations aimed at improving the targeting strategy by 

incorporating key elements of a risk management framework and 

recognized modeling practices. Table 2 provides more detail on our 

February 2004 recommendations and CBP’s efforts to implement them.

Recommended steps for incorporating 

key elements of a risk management 

framework and recognized modeling 

practices Status of the recommendation

Conduct and use a comprehensive set of 

threat, criticality, vulnerability, and risk 

assessments related to maritime cargo 

containers to determine the need for risk 

mitigation actions

Implemented

In August 2006, we reported that CBP was conducting risk assessments based on threat 

and intelligence information it was receiving. Although CBP was preparing threat 

assessments, it did not have a documented methodology in place to guide its personnel 

in identifying sources of threat information, what approach to take in conducting the 

assessment, and key elements to include that could help ensure consistency in the 

preparation of such assessments. As a result, we made an additional recommendation in 

August 2006 related to establishing and documenting a methodology for conducting 

threat assessments, which CBP subsequently implemented in October 2006.

Initiate an external peer review of ATS to 

evaluate, among other things, the types 

and sources of data being used and the 

appropriateness and weighting of targeting 

rules

Implemented

In response to this recommendation, CBP contracted with a consulting firm in 2005 and 

2006 to conduct a peer review and to develop performance measures, obtain additional 

insights into ATS’s performance, and determine whether ATS is more effective at 

targeting shipments than a random sampling approach.

 

4

In November 2006, the DHS OIG also reported CBP was in the process of developing 

national ATS performance measures in response to our February 2004 recommendations. 

The DHS OIG did not issue any related recommendations in that report. That DHS OIG 

report contains sensitive information and is therefore not publicly available.
Appendix II: Past Audit Findings and 

Recommendations to Improve the Targeting 

Process 

Page 32 GAO-13-9 Supply Chain Security

Recommended steps for incorporating 

key elements of a risk management 

framework and recognized modeling 

practices Status of the recommendation

Implement a mandatory random sampling 

program that cannot be waived, which 

would allow CBP to better compare the 

inspection results from its random sample 

program with those of ATS, and which may 

result in improvements to the targeting 

rules.

Implemented

a

CBP’s Compliance Measurement Program randomly selects shipments based on 

customs entry information submitted by the trade industry. We reviewed CBP’s sampling 

methodology and determined that its sampling techniques support a statistically valid 

random probability sample. In August 2006 we reported that CBP had made progress in 

conducting random sampling, but that at the time, CBP was unable to compare the 

examination results from its Compliance Measurement Program with ATS inspection 

results because CBP did not yet have an automated system in place to compare multiple 

sets of data—like results of random examinations with results of routine ATS inspections. 

Subsequently, in March 2008, CBP implemented the Cargo Enforcement Reporting and 

Tracking System (CERTS) within ATS to capture examination findings.b

Conduct simulated events to operationally 

test and validate the targeting strategy

As a result of its 

actions, CBP is better positioned to systematically analyze inspection results for 

adjusting ATS.

Implemented

We reported in August 2006 that CBP had not yet conducted simulated events such as 

covert tests and computer-generated simulations to test and validate the effectiveness of 

ATS in targeting oceangoing cargo shipped in containers that pose the highest risk of 

having links to terrorism. CBP has since taken actions to implement this 

recommendation by, among other things, developing and implementing a testing and 

simulation environment to conduct computer-generated tests of ATS. In June 2007, CBP 

reported to us that as of May 2007, it had tested approximately 26,000 mock cargo 

shipments and modified or updated information contained in ATS.

Monitor and evaluate the effectiveness of 

risk mitigation actions taken so that the 

targeting strategy may be amended as 

needed and responds to changes in risk

Implemented

CBP contracted a consulting firm to develop performance measures, and the consulting 

firm produced a report in April 2006 that outlined performance measures and a 

methodology for assessing the effectiveness of ATS. CBP has continued to use these 

performance measures, but as we have stated in this report, CBP’s assessments of 

ATS’s effectiveness have not occurred on a regular basis.

Source: GAO.

a

In addition to this recommendation, the DHS OIG made a recommendation in July 2005 that CBP 

use examination results to refine ATS targeting rules. See DHS OIG, Audit of Targeting Oceangoing 

Cargo Containers (Unclassified Summary), OIG-05-26 (Washington, D.C.: July 2005). 

b

The DHS OIG recommended in August 2007 that CBP develop systematic procedures to extract 

oceangoing container examination results information and begin using it to refine existing targeting 

rules and developing new rules. DHS OIG stated in its report that the intention of its recommendation 

was satisfied as CBP had plans for and had begun implementing CERTS. This August 2007 DHS 

OIG report contains sensitive information and is therefore not publicly available. The DHS OIG 

subsequently issued a report in June 2008 with recommendations related to the management and 

oversight of the development and implementation of CERTS. See DHS OIG, Targeting of Cargo 

Containers 2008: Review of CBP’s Cargo Enforcement Reporting and Tracking System, OIG-08-65 

(Washington, D.C.: Jun. 11, 2008). As noted above in the table, CBP fully implemented CERTS in 

May 2008. 

In August 2006, among other things, we reported on the status of 

recommendations issued in our February 2004 report and reiterated the 

importance of the recommendations. Additionally, with respect to the 

recommendation regarding risk assessments, at the time, CBP had 
Appendix II: Past Audit Findings and 

Recommendations to Improve the Targeting 

Process 

Page 33 GAO-13-9 Supply Chain Security

begun taking action to implement the recommendation by conducting and 

using risk assessments that incorporated discussions of potential threats 

and estimates of the relative importance of assets and vulnerabilities 

associated with the supply chain. We noted, though, that CBP did not 

have a methodology in place to guide its staff in identifying sources of 

threat information, such as agencies to contact, what approach to take in 

conducting the assessment, and key elements to include that would help 

ensure consistency in the preparation of threat assessments associated 

with the movement of cargo shipped in containers. As a result, we further 

recommended that CBP establish and document a methodology for 

conducting threat assessments associated with cargo shipped in 

containers to help ensure that CBP staff responsible for conducting threat 

assessments consult relevant information sources, prepare threat 

assessments consistently, and include key elements to effectively 

communicate risk to program managers.

In response to that recommendation, in October 2006, CBP issued a 

protocol to assist its intelligence research specialists in preparing port 

threat assessments in support of the Container Security Initiative. Under 

this initiative, CBP places staff at foreign seaports to work with foreign 

counterparts to inspect high-risk containers before they are shipped to the 

United States. The October 2006 protocol included a discussion of 

information sources for CBP staff to consult and the overall methodology 

to follow in making the port threat assessment. CBP also distributed a 

template that discussed the key elements its specialists should include 

when preparing port threat assessments. In addition, CBP developed a 

checklist for its specialists to use to help ensure that appropriate 

information sources are consulted in making port threat assessments. To 

assist in addressing threats related to global supply chain logistics and 

the movement of maritime containers carrying cargo arriving in the United 

States, CBP established an Office of Intelligence and Operations 

Coordination in October 2007, which has since been renamed the Office 

of Intelligence and Investigative Liaison. This office includes the Analysis 

and Targeting Division (A&T), which is composed of program managers 

with operational experience and intelligence analysts and is responsible 

for conducting risk assessments of countries (known as country risk 

profiles) that consider threats, vulnerabilities, and the associated criticality 

of related assets from which or through which cargo is shipped to the 

United States. In 2008, the A&T Division assessed risks in two primary 

mission areas: (1) terrorism and weapons of mass destruction and (2) 

narcotics. On the basis of these assessments, the A&T Division 

developed a risk assessment methodology to rank countries according to 

the level of risk associated with each mission area. The A&T Division 
Appendix II: Past Audit Findings and 

Recommendations to Improve the Targeting

Process 

Page 34 GAO-13-9 Supply Chain Security

teams translated these rankings into scores, and program managers 

integrated them into ATS. Specifically, CBP integrated the rankings for 

the terrorism mission into ATS in 2008 and for the narcotics mission in 

2009. Thus, CBP has fully addressed this recommendation and is better 

positioned to ensure consistency in the preparation of risk assessments 

associated with the movement of maritime cargo container shipments.

In January 2010, the DHS OIG reported on several aspects of CBP’s 

process for developing and updating targeting rules in ATS.5

In February 2004, in addition to making the recommendations discussed 

earlier, we also reported that CBP was relying on the manifest as its 

principal data input, and CBP did not mandate the transmission of 

additional information before a cargo’s risk level was assigned. We 

In this 

report, the DHS OIG stated that CBP could improve its process for 

changing or deleting targeting rules by, among other things, documenting 

(1) rule change decisions and (2) the testing and evaluation of rule 

changes. Specifically, one component of the rule update process involves 

the review of the proposed rule changes by subject matter experts, and 

the DHS OIG reported that CBP could improve the process by ensuring 

the rationale for changes implemented or not implemented are 

documented and recorded for future use. Furthermore, the DHS OIG 

reported that CBP tested new rules using actual data to determine how 

well the new rules are working, but the DHS OIG noted that this process 

for testing and evaluating the rules, and subsequent modifications of the 

new rules, was not documented. The DHS OIG recommended that CBP 

ensure it documents each stage of the process for analyzing and 

developing ATS rules, including the rationale for making changes and the 

details on tools used to improve application consistency and rule change 

standardization. According to the DHS OIG report, in response to this 

recommendation, CBP (1) developed a documentation process to capture 

and record information that includes the rationale for rule changes and 

the utilization of tools and (2) introduced more formality into the rules 

process by implementing a structure to guide national conferences, rule 

evaluation, targeting development, and process management, among 

other things. The DHS OIG stated in its report that it considered the

actions taken by CBP to be responsive to the recommendation.

 

5

DHS OIG, Cargo Targeting and Examinations, OIG-10-34.

Documenting the Targeting 

Rule Development Process

Improving Data Collection 

and ATS’s Use of Data
Appendix II: Past Audit Findings and 

Recommendations to Improve the Targeting 

Process 

Page 35 GAO-13-9 Supply Chain Security

reported that terrorism experts, members of the international trade 

community, and CBP inspectors at the ports we visited as part of that 

review characterized the ship’s manifest as one of the least reliable or 

useful types of information for targeting purposes. We reported that 

terrorism experts, trade community representatives, and some CBP 

inspectors at ports we visited told us that CBP should explore requiring 

more timely electronic transmittal of additional data elements for cargo 

container targeting purposes, such as stowage plans (a map of where 

each container aboard a ship is stored), container movement tracking 

data, and entry data. Although we did not analyze the feasibility or the 

costs and benefits of these suggestions, we reported that it could be 

useful for CBP to explore requiring appropriate parties in the supply 

chain, such as the importer, to provide additional data elements for use in 

ATS to perform more complex linkage analyses and identify potential 

anomalies in the shipping documents filed. Although we did not make a 

recommendation directly related to data collection because we did not 

analyze the feasibility or the costs and benefits of collecting additional 

data, the recommendations in our report focused generally on 

incorporating key elements of recognized modeling practices, and one of 

the recognized modeling practices applicable to ATS is enhancing the 

sources and types of information input into ATS. Furthermore, in July 

2005, the DHS OIG issued an unclassified summary of an audit regarding 

CBP’s targeting for maritime cargo containers, which concluded that 

improvements were needed in the data to which ATS targeting rules are 

applied.6

In January 2009, CBP initiated an effort to collect additional data through 

the Importer Security Filing and Additional Carrier Requirements, 

collectively known as the 10+2 rule. The rule requires importers and 

carriers to provide 10 data elements and 2 data elements, respectively, to 

CBP for improving CBP’s ability to identify high-risk cargo container 

shipments. Specifically, the 10+2 rule requires importers to submit 

information about the commodities being transported in a shipment and 

about entities involved in the supply chain. These additional data 

elements include information that we reported could be helpful in 

improving targeting efforts. In particular, the rule requires stowage plans 

and some elements of entry data. We reported in September 2010 that 

 

6

DHS OIG, Audit of Targeting Oceangoing Cargo Containers (Unclassified Summary), 

OIG-05-26.
Appendix II: Past Audit Findings and 

Recommendations to Improve the Targeting 

Process 

Page 36 GAO-13-9 Supply Chain Security

the 10+2 rule data elements were available for identifying high-risk cargo 

at that time, but that CBP had not yet updated ATS to fully incorporate the 

data into its targeting criteria.7 We recommended that CBP establish 

milestones and time frames for updating the targeting criteria. In 

December 2010, CBP provided us with a project plan for integrating the 

data into its targeting criteria, and in early 2011, CBP implemented the 

updated targeting criteria to address risk factors present in the Importer 

Security Filing data.

In November 2006, the DHS OIG reported on issues related to 

information that is available to targeters to conduct targeting activities.8

The DHS OIG found that CBP targeters did not always have access to a 

particular system that directly accesses a database containing billions of 

records on individuals and businesses. The DHS OIG recommended that 

CBP provide targeters with access to that system or a similar system that 

would allow targeters to access business records. In its report, the DHS 

OIG stated that it considered the recommendation to be implemented 

because CBP had acquired funding to grant personnel access to that 

system and had issued a memo stating that all personnel that were to be 

scheduled for targeting training should also have access to the system.9

 

7

The DHS OIG also found that it was unclear which personnel involved in 

targeting should have security clearances, and as a result, important 

information affecting container targeting and inspection decisions may not 

be available to the staff for making these decisions in a timely manner. 

The DHS OIG recommended that CBP increase the number of targeters 

with security clearances. In response to this recommendation, according 

to the DHS OIG, CBP presented a corrective action plan with an 

established completion date of June 30, 2007, for this recommendation. 

The DHS OIG further noted that in July 2006, CBP issued a memo for 

Security Clearance for Counter Terrorism Response Officers, directing 

CBP field offices to forward applications of port personnel that need to 

GAO-10-841. 

8 The DHS OIG report issued in November 2006 contains sensitive information and is 

therefore not publicly available.

9

The system discussed in the November 2006 DHS OIG report is the predecessor to the 

third-party database of public and proprietary records that is mentioned in this report as a 

tool that targeters use outside of ATS to research supply chain parties involved in a 

shipment and to inform their decision whether to target a shipment for examination.

Providing Additional 

Information Outside of 

ATS to Targeters
Appendix II: Past Audit Findings and 

Recommendations to Improve the Targeting 

Process 

Page 37 GAO-13-9 Supply Chain Security

have security clearances. On the basis of these efforts, the DHS OIG 

reported that CBP had satisfied the intent of this recommendation.
Appendix III: Comments from the Department 

of Homeland Security

Page 38 GAO-13-9 Supply Chain Security

Appendix III: Comments from the 

Department of Homeland Security
Appendix III: Comments from the Department 

of Homeland Security

Page 39 GAO-13-9 Supply Chain Security
Appendix IV: GAO Contact and Staff 

Acknowledgments

Page 40 GAO-13-9 Supply Chain Security

Stephen L. Caldwell, Director (202) 512-9610 or caldwells@gao.gov

In addition to the contact named above, Christopher Conrad (Assistant 

Director), Alana Finley, Richard Hung, Katie Mauldin, and Janay Sam 

made key contributions to this report. Also contributing to this report were 

Richard Brown, Frances Cook, Stanley Kostyla, and Lara Miklozek.

Appendix IV: GAO Contact and Staff 

Acknowledgments 

GAO Contact

Acknowledgments
Related GAO Products

Page 41 GAO-13-9 Supply Chain Security

Supply Chain Security: Container Security Programs Have Matured, but 

Uncertainty Persists over the Future of 100 Percent Scanning. 

GAO-12-422T. Washington, D.C.: February 7, 2012.

Maritime Security: Responses to Questions for the Record. 

GAO-11-140R. Washington, D.C.: October 22, 2010.

Supply Chain Security: CBP Has Made Progress in Assisting the Trade 

Industry in Implementing the New Importer Security Filing Requirements, 

but Some Challenges Remain. GAO-10-841. Washington, D.C.: 

September 10, 2010.

Supply Chain Security: Feasibility and Cost-Benefit Analysis Would Assist 

DHS and Congress in Assessing and Implementing the Requirement to 

Scan 100 Percent of U.S.-Bound Containers. GAO-10-12. Washington, 

D.C.: October 30, 2009.

Intellectual Property: Better Data Analysis and Integration Could Help 

U.S. Customs and Border Protection Improve Border Enforcement 

Efforts. GAO-07-735. Washington, D.C.: April 26, 2007.

Cargo Container Inspections: Preliminary Observations on the Status of 

Efforts to Improve the Automated Targeting System. GAO-06-591T. 

Washington, D.C.: March 30, 2006.

Homeland Security: Summary of Challenges Faced in Targeting 

Oceangoing Cargo Containers for Inspection. GAO-04-557T. 

Washington, D.C.: March 31, 2004.

Related GAO Products

(441011)
The Government Accountability Office, the audit, evaluation, and 

investigative arm of Congress, exists to support Congress in meeting its 

constitutional responsibilities and to help improve the performance and 

accountability of the federal government for the American people. GAO 

examines the use of public funds; evaluates federal programs and 

policies; and provides analyses, recommendations, and other assistance 

to help Congress make informed oversight, policy, and funding decisions. 

GAO’s commitment to good government is reflected in its core values of 

accountability, integrity, and reliability.

The fastest and easiest way to obtain copies of GAO documents at no 

cost is through GAO’s website (http://www.gao.gov). Each weekday 

afternoon, GAO posts on its website newly released reports, testimony, 

and correspondence. To have GAO e-mail you a list of newly posted 

products, go to http://www.gao.gov and select “E-mail Updates.”

The price of each GAO publication reflects GAO’s actual cost of 

production and distribution and depends on the number of pages in the 

publication and whether the publication is printed in color or black and 

white. Pricing and ordering information is posted on GAO’s website, 

http://www.gao.gov/ordering.htm. 

Place orders by calling (202) 512-6000, toll free (866) 801-7077, or 

TDD (202) 512-2537. 

Orders may be paid for using American Express, Discover Card, 

MasterCard, Visa, check, or money order. Call for additional information.

Connect with GAO on Facebook, Flickr, Twitter, and YouTube. 

Subscribe to our RSS Feeds or E-mail Updates. Listen to our Podcasts. 

Visit GAO on the web at www.gao.gov. 

Contact:

Website: http://www.gao.gov/fraudnet/fraudnet.htm

E-mail: fraudnet@gao.gov

Automated answering system: (800) 424-5454 or (202) 512-7470

Katherine Siggerud, Managing Director, siggerudk@gao.gov, (202) 512-

4400, U.S. Government Accountability Office, 441 G Street NW, Room 

7125, Washington, DC 20548

Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800

U.S. Government Accountability Office, 441 G Street NW, Room 7149 

Washington, DC 20548

GAO’s Mission

Obtaining Copies of 

GAO Reports and 

Testimony

Order by Phone

Connect with GAO

To Report Fraud, 

Waste, and Abuse in 

Federal Programs

Congressional 

Relations

Public Affairs

Please Print on Recycled Paper.