Supply Chain Security: CBP Needs to Conduct Regular Assessments of Its Cargo Targeting System
Published: 2012-10-25T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The U.S. economy is dependent on the expeditious flow of millions of tons of cargo each day. Cargo containers are an important instrument of global trade but also can present security concerns. CBP is responsible for administering container security programs, and its strategy for securing maritime cargo containers includes analyzing information to identify shipments that may contain terrorist weapons or other contraband. Because CBP has insufficient resources to examine every container, targeters use ATS to target which container shipments should be examined. GAO was asked to assess CBP's targeting efforts. This report addresses (1) how ATS supports CBP's targeting of maritime cargo container shipments for national security purposes and (2) the extent to which CBP assesses the effectiveness of ATS's national security targeting rules. GAO analyzed fiscal year 2011 CBP data on shipments and containers arriving at U.S. ports and containers scanned at these ports. GAO also visited six CBP units selected on the basis of the percentage of maritime shipments that were scored as high risk or medium risk for national security purposes at these locations in fiscal year 2011, among other factors. GAO also analyzed documents, such as CBP's ATS performance measures.
section_title: What GAO Found
paragraphs: U.S. Customs and Border Protection (CBP), within the Department of Homeland Security (DHS), employs a risk-based approach that uses the Automated Targeting System (ATS) and other tools to identify (target) maritime cargo shipments for further examination. ATS is a web-based enforcement and decision support system that includes a set of rules to assess the risk level for each arriving cargo shipment. This set of rules is referred to as the maritime national security weight set (weight set) because each rule in the set has a specific weighted value assigned to it. CBP classifies the risk scores from the weight set as low, medium, or high risk. CBP policy states that a shipment's risk score is to determine, in part, actions taken by CBP officers (targeters) at the ports. Specifically, targeters are generally required to review shipment data for all medium-risk and high-risk shipments and hold high-risk shipments for examination. The risk score, however, is not the sole factor that determines whether a targeter reviews the data for a shipment or whether CBP examines a shipment. In particular, targeters at each of the six ports GAO visited explained that they use the ATS risk score as a starting point for the targeting process but that their decisions regarding which shipments to examine are ultimately based on additional research. Targeters at the six ports GAO visited said they also use tools outside of ATS, such as web searches, to research shipments.
CBP efforts to assess the weight set's effectiveness in identifying the risk of shipments have been limited. CBP has performance measures--represented by the percentage of shipments targeted as high risk that contain a threat and the percentage of shipments targeted as high risk that do not contain a threat--that enable CBP to determine the accuracy of the weight set, given a particular workload or examination rate. However, CBP did not assess the weight set to verify its effectiveness when implementing an updated version in early 2011. Prior to implementing the updated version of the weight set, CBP assessed the potential impact of the update on CBP's workload but did not conduct an assessment to determine whether the updated version of the weight set would be more effective in identifying high-risk shipments than the previous version or other alternatives. Assessing the potential effectiveness of alternative versions of the weight set prior to selecting one for implementation could help CBP make more informed decisions about future updates. Doing so could also provide CBP reasonable assurance that the version it selects is the most effective of the alternatives and is more effective than the previous version it replaces. Furthermore, since implementing the updated version of the weight set in early 2011, CBP has not regularly assessed the weight set to monitor its performance and to help determine when changes are needed. For example, CBP conducted the first assessment of the current version of the weight set, using the performance measures, in the summer of 2012--18 months after the weight set's implementation in early 2011. Regular assessments of the weight set's effectiveness could help CBP determine when updates are needed in a timelier manner and ensure that targeters have the best information available to make targeting decisions. Moreover, CBP has not established targets for the performance measures so that it is not clear whether a particular change in the weight set's performance is significant enough to suggest that changes are needed to improve the effectiveness of the weight set.
section_title: What GAO Recommends
paragraphs: GAO recommends that CBP (1) ensure that future updates to the weight set are based on assessments of its performance and (2) establish targets for performance measures and use those measures to regularly assess effectiveness of the weight set. DHS concurred with these recommendations.

section_title: Letter
paragraphs: The economic well-being of the United States is dependent on the  expeditious flow of millions of tons of cargo each day. According to the  U.S. Department of Transportation, the majority of U.S. imports arrive by  ocean vessel, and much of that is transported in cargo containers. In  fiscal year 2011, for example, about 13.4 million cargo containers arrived  at U.S. seaports. Cargo containers are an important segment of the  global supply chain—the flow of goods from manufacturers to retailers— and can present significant security concerns. Within the federal  government, U.S. Customs and Border Protection (CBP), part of the  Department of Homeland Security (DHS), is responsible for administering  container security and reducing the vulnerabilities associated with the  supply chain. Balancing security concerns with the need to facilitate the  free flow of commerce, part of CBP’s mission, remains an ongoing  challenge for the public and private sectors alike. CBP officials believe  that the likelihood of terrorists smuggling weapons of mass destruction  (WMD) into the United States in cargo containers is relatively low;  however, the consequences of such an event could be catastrophic.
Although there have been no known incidents of cargo containers being  used to transport WMD, ensuring the security of cargo containers remains  an important role for CBP given that criminals have exploited containers  for other illegal purposes, such as smuggling weapons, people, and illicit  substances.
Since September 11, 2001, Congress has passed various laws to  address concerns about the security of cargo containers in the global  supply chain. The enactment of the Maritime Transportation Security Act  of 2002 called for the establishment of a program to evaluate and certify  secure systems of international intermodal transportation, including  standards and procedures for screening and evaluating cargo containers  prior to loading onto vessels and for securing and monitoring cargo while  in transit. In 2006, the Security and Accountability For Every Port Act  was enacted, which required, among other things, that pilot projects be  established at three ports to test the feasibility of scanning 100 percent of  U.S.-bound cargo containers at foreign ports. Subsequently, the  Implementing Recommendations of the 9/11 Commission Act of 2007  (9/11 Act) required, among other things, that by July 2012, 100 percent  of U.S.-bound cargo containers be scanned at foreign ports with both  radiation detection and nonintrusive inspection (NII) equipment before  being placed on U.S.-bound vessels. In May 2012, the Secretary of  Homeland Security authorized a 2-year extension—until July 2014—of  the deadline for implementing the requirement that containers not enter  the United States unless they were scanned at foreign ports prior to being  loaded on vessels.
We reported in October 2009 that CBP faced numerous challenges in  implementing 100 percent scanning at pilot ports, and on the basis of  work we have completed since then we know that CBP has not yet  achieved 100 percent scanning. among other things, that CBP conduct a feasibility analysis of  implementing the 100 percent scanning requirement and provide the  results to Congress along with any suggestions of cost-effective  alternatives to implementing the 100 percent scanning requirement, as  appropriate. DHS stated that CBP concurred with these  recommendations, but CBP has not yet taken action to address them. In  its report to Congress in May 2012 on the planned deadline extension,  DHS stated that it recognizes the need to proceed with container security  programs in a manner that maximizes the security of maritime cargo and  facilitates its movement and reported that it plans to continue working with  other federal agencies and international partners to develop technology  and enhance risk management processes, in addition to continuing  existing programs that enhance cargo security. However, given that the  feasibility of 100 percent scanning remains in doubt and DHS and CBP  have not identified alternatives that could achieve the same goals as 100  percent scanning, uncertainty persists regarding the scope of DHS’s and  CBP’s existing container security programs and how these programs will  collectively affect the movement of goods between trading partners.
GAO, Supply Chain Security: Container Security Programs Have Matured, but  Uncertainty Persists over the Future of 100 Percent Scanning, GAO-12-422T  (Washington, D.C.: Feb. 7, 2012), and Supply Chain Security: Feasibility and Cost-Benefit  Analysis Would Assist DHS and Congress in Assessing and Implementing the  Requirement to Scan 100 Percent of U.S.-Bound Containers, GAO-10-12 (Washington,  D.C.: Oct. 30, 2009).
CBP’s strategy for securing the maritime supply chain consists of  programs that intersect with key points in the supply chain. These  programs include, among other things, analyzing information to identify  shipments that may be at high risk of transporting WMD or other  contraband; working with foreign governments to examine U.S.-bound,  high-risk shipments at foreign ports; and examining U.S.-bound, high-risk  shipments that were not examined overseas upon their arrival at a  domestic port. To aid in this process, CBP uses the Automated  Targeting System (ATS), which is an intranet-based enforcement and  decision support system that compares traveler, cargo, and conveyance  information against intelligence and other enforcement data. Among other  things, ATS uses a set of rules that assess different factors in data  provided by supply chain parties, such as importers, to determine the risk  level for a shipment. CBP officers (targeters) use information in ATS to  identify (target) which shipments should be held for an examination,  which may include an NII scan or a physical inspection. Because CBP  does not scan 100 percent of U.S.-bound cargo containers, the  effectiveness of CBP’s security strategy depends on CBP’s ability to use  ATS, among other tools, to effectively target shipments in the supply  chain that pose the greatest security risks.
In response to your request, we analyzed certain aspects of CBP’s  maritime national security targeting efforts. Specifically, this report  addresses the following objectives:
How does ATS support CBP’s process for targeting maritime cargo  container shipments for national security purposes?
To what extent does CBP assess the effectiveness of the national  security targeting rules in ATS?
To address the first objective, we obtained data from CBP for each of the  115 U.S. seaports for fiscal year 2011 on the number of (1) shipments it  placed in each of three categories—high risk, medium risk, and low risk— arriving at each of the ports; (2) container arrivals at these ports; and (3)  containers scanned at these ports using NII equipment. We selected and  visited six CBP units responsible for targeting at domestic ports.  Specifically, these ports were selected from among the largest ports in  the United States using the following criteria: (1) the percentage of  maritime shipments that were scored as high risk or medium risk for  national security purposes, (2) the percentage of cargo containers that  were examined using NII equipment, and (3) whether a CBP official from  the port participated in the most recent CBP conference to discuss  changes to ATS cargo targeting rules. The six targeting units we visited  were responsible for targeting efforts at 15 ports that collectively received  about 60 percent of the maritime shipments that arrived in the United  States in fiscal year 2011. As part of these site visits, we observed port  operations, including the scanning of containers. At each location, we  also interviewed CBP targeters, including those who participated in the  most recent CBP conference to discuss changes to cargo targeting rules,  and observed their use of ATS and other tools to conduct cargo targeting  activities. The results from our visits to these six targeting units cannot be  generalized to ports nationwide; however, visits to these locations allowed  us to directly observe the targeting process and provided insights into  how ATS assigns risk scores to maritime shipments and how CBP  integrates the scores into its targeting process. We synthesized the  information from these site visits to describe the targeting process and  also analyzed CBP policies and guidance, such as CBP’s National  Maritime Targeting Policy and course materials from CBP’s Sea Cargo  Targeting Training. We also visited CBP’s National Targeting Center– Cargo (NTC-C) to interview targeters responsible for conducting national- level targeting and to observe their targeting activities.
DHS Office of Inspector General, Cargo Targeting and Examinations, OIG-10-34  (Washington, D.C.: Jan. 6, 2010). Because the reports we issued in February 2004 and  August 2006 regarding CBP’s targeting practices contain sensitive information, they are  not publicly available. review of relevant documentation and was sufficient to address the issue  of documentation of changes to the ATS rules that we present in this  report. We analyzed documentation of CBP’s most recent update to the  national security targeting rules, which CBP implemented in early 2011,  and the extent to which this documentation addressed effectiveness.  Specifically, we evaluated CBP’s impact assessments, which provide  information on the number of shipments the updated rules would assess  as high risk, thereby affecting CBP’s examination workload. In addition,  we evaluated the extent to which CBP used its methodology to assess  the current national security targeting rules and analyzed the results of  CBP’s assessments. To assess the reliability of the results of CBP’s  assessments, we reviewed documentation on the methodology created  for CBP by a consulting firm in 2006. We interviewed knowledgeable CBP  officials about any adjustments to this methodology since the contract  expired and CBP analysts began conducting the performance  assessments. On the basis of this information we determined that the  results of the assessments are sufficiently reliable for the purposes of our  report. To determine the extent to which CBP conducts such  assessments on a regular basis, we analyzed documentation of recent  assessments of the national security targeting rules conducted in spring  2011 and summer 2012. We compared this information with key elements  for a risk management approach and Standards for Internal Control in the  Federal Government. We also reviewed our prior work on risk  management practices and compared our analysis of CBP’s actions and  assessments with those practices. Finally, we interviewed officials at CBP  headquarters who are responsible for maintaining and updating ATS to  obtain information on past efforts to assess and update the national  security targeting rules.
DHS deemed some of the information in a draft version of this report as  sensitive, and therefore, this report omits sensitive details regarding  specific information available in ATS, examples of how targeters may use  that information, and specific dates associated with changes to CBP’s  targeting criteria. These omissions did not affect the presentation of the  key information and findings that support our conclusions and  recommendations.
We conducted our work from October 2011 through October 2012 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
subsections: section_title: CBP’s Maritime Cargo Container Security Strategy
paragraphs: CBP’s security strategy for maritime cargo uses a risk-based approach to  focus limited resources on targeting and examining cargo shipments that  pose a potential risk while allowing other cargo shipments to proceed  without unduly disrupting commerce into or out of the United States. The  strategy is based, in part, on obtaining advance cargo information. In  particular, through what is referred to as the 24-hour rule, CBP generally  requires vessel carriers to electronically transmit cargo manifests to CBP  24 hours before cargo is loaded onto U.S.-bound vessels at foreign  ports. Through the Importer Security Filing and Additional Carrier  Requirements (known as the 10+2 rule), CBP requires importers and  vessel carriers to provide data elements for improved identification of  containerized cargo shipments that may pose a risk for terrorism.
Importers are responsible for supplying CBP with 10 shipping data  elements—such as country of origin—24 hours prior to loading, while  vessel carriers are required to provide 2 data elements—container status  messages and stow plansOther aspects of CBP’s maritime cargo container security strategy include  using technology, such as radiation detection equipment, to detect  potential threats and partnering with foreign governments and the trade  industry to examine containers prior to their arrival in the United States  and implement security measures throughout the supply chain process,  respectively. A brief description of the core programs that constitute  CBP’s security strategy for cargo containers is provided in appendix I.  Throughout the supply chain process, and underlying many of the  programs within CBP’s security strategy for cargo containers, CBP  assesses the national security risks posed by shipments throughout the  supply chain process, as shown in figure 1.  —that are not required by the 24-hour rule.
section_title: CBP Uses ATS and Other Tools to Target Maritime Cargo Shipments for National Security Purposes
paragraphs: ATS is the primary system that CBP targeters use to review maritime  cargo shipments for national security purposes, and targeters we spoke  with were generally satisfied with how ATS and its weight set of national  security rules have assisted in their targeting efforts. For example,  targeters at one ATU we visited said that because ATS filters information  and presents key information to the targeters, the targeters are able to  better focus their targeting efforts than before they had ATS. Those  targeters as well as the ATU supervisor at another ATU noted that, in  particular, the risk scores that the current version of the weight set  produces are helpful in balancing their targeting workload. The risk score,  however, is not the sole factor that determines whether a targeter reviews  the data for a shipment or whether the shipment is selected for a security  examination. In particular, targeters at each of the six ATUs we visited  explained that they use the ATS risk score as a starting point for the  targeting process, but that their decisions are ultimately based on  additional research. To conduct this research, they may use information  within ATS or other tools and information outside of ATS. On the basis of  the ATS risk score and the research conducted, targeters make a  qualitative assessment of the risk and determine whether to hold a  shipment for examination. Targeters at one of the ATUs we visited  emphasized the important role that targeters’ expertise and experience  play in the risk assessment process, stating that although they believe  ATS’s capabilities are helpful, they believe that there could be negative  effects from further automating the targeting process, such as decreased  use of targeters’ expertise regarding the different types of shipments that  arrive at their respective ports.
According to CBP policy, the risk scores assigned by the weight set in  ATS determine, in part, what actions CBP officers at the ports are to take  to address potential threats. Targeters at ATUs are required to review  data in ATS for all medium-risk and high-risk shipments that arrive at their  respective ports. For example, a targeter may review individual data  elements, such as the name of the importer or other supply chain parties.  A targeter may also review the rules that detected potential threats and,   ATU targeters  therefore, contributed to the calculation of the risk score. are also required to hold high-risk shipments for examination unless they  can mitigate the risk through additional research or analysis of available  information.demonstrated how they implement this policy at their respective ports. In  addition to actions targeters take in accordance with CBP policy, targeters  have discretion over which low-risk and medium-risk shipments to select   CBP targeters at each of the six ATUs we visited  for security examinations, and CBP expects targeters to select shipments  based on discretionary factors.
Targeters use various features within ATS to assist them in their research  into shipments of interest. Officials at each of the six ATUs we visited  discussed or demonstrated the following features and how targeters use  these features when targeting:
Queries: Through ATS’s querying capabilities, targeters are able to  search for shipments that meet specific criteria, such as shipments  from a particular country. For example, targeters at each ATU we  visited use queries to identify shipments for review, although the set of  queries used varied at each ATU. Five of the six ATUs we visited use  queries to ensure that all shipments, regardless of risk score, are  reviewed prior to arrival. For example, targeters at one ATU run a  query for each arriving vessel to ensure that all shipments on the  vessel have been reviewed. Targeters at another ATU have a query  for each risk level (high, medium, or low) and targeters reviewing the  results of each query are to ensure that all shipments of a particular  risk level have been reviewed. In addition, targeters at five of the six  ATUs we visited said that they run additional queries of interest for  discretionary targeting after completing their assigned duties. Such  discretionary targeting could be for national security purposes or for  other efforts, such as counternarcotics. For example, targeters at one  ATU may independently create queries to identify items of interest,  such as all shipments of a particular commodity or those coming from  a particular country of origin. In addition to queries that targeters at  the ATUs run, NTC-C targeters run nationwide queries daily to identify  shipments with the potential for containing chemical, biological,  radiological, nuclear, or conventional weapons, among other things.
Targeters’ notes: ATS has a feature that allows a targeter to  annotate a shipment with the targeter’s conclusion based on research  regarding whether the shipment is considered a potential threat. The  notes feature within ATS facilitates the sharing of research findings  with CBP targeters at other locations.
Targeters’ reviews: ATS also indicates whether the shipment data  have been reviewed by a targeter at the targeter’s own location or at  another CBP targeting location, such as a Container Security Initiative  port or NTC-C.
CBP targeters also use tools outside of ATS to conduct research. During  our interviews at the six domestic port ATUs we visited, targeters  explained that they use web-based and other research tools to aid in their  assessments of shipments. Such tools include web searches, which  targeters use to find general information on a company or address; a  third-party database of public and proprietary records, which targeters  use to research business names and associated information such as a  business’s locations, officers, and assets (e.g., registered vehicles); and  the State Department’s Consular Consolidated Database, which targeters  may use to obtain visa and passport information for foreign individuals  involved in a shipment. Targeters review and analyze all of the  information collected to make a decision as to whether a shipment should  be examined. On the basis of such research and analysis, a targeter  could select a low-risk shipment for examination. A targeter could also  determine that an examination is not necessary for a medium-risk  shipment—for example, the weight set may assign a medium-risk score  to a shipment based on the data available, but the targeter could  determine through research that the score is based on a clerical error in  the data provided.
Targeters’ experience may also inform targeting decisions. For example,  targeters at ATUs may have information about recent seizures and can  look for recurring patterns to identify future shipments that may be part of  a trend of illegal shipments. Targeters may also share such information  with other targeting units to help inform targeting decisions. Also,  targeters at all six of the ATUs we visited said they communicate regularly  with targeters at NTC-C regarding shipments of interest.
section_title: CBP’s Efforts to Assess the Weight Set Have Been Limited
subsections: section_title: CBP Developed Performance Measures to Assess the Weight Set but Is Continuing to Update the Methodology
paragraphs: We have previously reported that ensuring controls to assess ATS’s  effectiveness in identifying high-risk shipments was important for  providing CBP with the best information to inform its targeting efforts. In  2005, in response to our work and an external peer review of ATS  conducted in 2005, CBP contracted with a consulting firm to develop  performance measures and a methodology to determine the effectiveness  of the weight set in identifying high-risk shipments. The resulting  performance measures and methodology, which the consulting firm  provided to CBP in April 2006, balanced targeting accuracy with  examination workload and enabled CBP to compare the weight set’s  performance with the effectiveness of examinations conducted through a  random selection program. We then reported in August 2006 that the  performance measures and methodology developed by the consulting  firm were sufficient to assess the performance of the weight set and  provide a baseline against which future assessments may be  conducted; however, we also reported that data limitations and  uncertainties existed, and we noted that CBP must interpret the  evaluations cautiously. We also reported in August 2006 that CBP  planned to continue using the consulting firm’s methodology in making  future adjustments to the weight set. For more information about GAO’s  past audit findings and recommendations to improve the targeting  process, see appendix II.
Currently, CBP assesses the performance of the weight set using the  following performance measures:
True positive rate (TPR) which reflects the percentage of maritime  shipments that ATS assessed as high risk within the population of  shipments in which CBP identified a threat during an examination.
False positive rate (FPR) which reflects the percentage of maritime  shipments that ATS assessed as high risk within the population of  shipments in which CBP did not identify a threat during an  examination.
Taken in combination, the TPR and FPR measures enable CBP to  determine the effectiveness of the weight set by providing information  about the accuracy of the weight set and its impact on examination  workload. The TPR enables CBP to determine the accuracy of the weight  set in identifying high-risk shipments. The TPR and FPR measures also  enable CBP to determine the workload or examination rate for ports  based on the results of the weight set scores. For example, a high FPR  would unnecessarily increase the workload or number of examinations at  ports because officials would be required to examine a higher number of  shipments that do not contain an actual threat. Using data from the  version of the weight set CBP was using in 2005, the consulting firm used  its methodology to conduct a performance assessment, which involved  calculating the TPR and FPR to ultimately indicate the effectiveness of  the weight set at that time. Since the contract with the firm ended in 2011,  CBP has taken on the role of assessing the effectiveness of the weight  set. In its project plan for the most recent update to the weight set  implemented in early 2011, as it had previously done, CBP planned to  assess the performance using these measures to compare the targeting  effectiveness of the weight set with other measurements.
CBP officials stated that they face ongoing challenges with the  performance measures and the methodology by which they are  calculated. In particular, the FPR and TPR may not accurately reflect the  weight set’s performance in identifying national security threats because  they rely on indirect measures given that no true security threat has been  found in a cargo container. We reported in 2006 that CBP planned to take  steps to improve the process for assessing ATS performance. Since the  most recent update to the weight set in early 2011, CBP formed working  groups and has begun taking steps to ensure that the methodology it  uses to approximate threats accurately reflects what CBP considers to be  a national security threat (to the extent possible). For example, these  working groups plan to (1) create a new definition of “national security” for  the purposes of clarifying what the weight set should target and (2) revisit  the current definitions of threats in containers to provide consistency with  the new definition of national security. CBP officials stated that they  expect the working groups’ activities to be completed by April 2013.
section_title: CBP Does Not Have Reasonable Assurance That the Updated Weight Set Is More Effective than Alternative Versions or the Version It Replaced
paragraphs: Prior to implementing the current version of the weight set in early 2011,  CBP did not conduct an assessment to determine whether the updated  version of the weight set would be more effective than the previous  version of the weight set or other alternatives that were considered during  the update process. For the 2011 update, CBP developed and evaluated  five alternative versions of the weight set. CBP’s consideration for which  alternative to select focused on two of the five versions because,  according to CBP officials, these two versions incorporated the newly  required 10+2 data and reflected current threat information about  countries of interest. CBP ultimately implemented one of these two  versions of the weight set, but CBP could not provide any documentation  to demonstrate that the version selected was more effective than either  the other alternative or the version it was to replace.
CBP’s process for updating the weight set involves assessing the impact  of alternative versions of the weight set. For example, for the most recent  update to the weight set, CBP’s impact assessment provides information  on how many shipments would be assessed as high risk under each  alternative version of the weight set and would, therefore, affect CBP  targeters’ workload at ports of arrival because such high-risk shipments,  under CBP policy, are to be held for examination, for example, through  the use of NII equipment. CBP officials stated that they believed the  impact assessment that CBP conducted during the update process  indicates the reasons for selecting the chosen version of the weight set.
While, according to CBP officials, the impact assessment provides CBP’s  reason for replacing the prior version of the weight set, we found that the  impact assessment primarily evaluates how the chosen alternative  version of the weight set could affect targeter workload and does not  address measures of accuracy in identifying high-risk shipments.  Therefore, the impact assessment does not fully account for the  effectiveness of each alternative of the weight set. Although managing  resources is an element of risk management, effectiveness in reducing  risks is also an important consideration when evaluating alternatives to  manage risk. CBP’s impact assessment does not address the balance  between targeting accuracy and workload.
Assessing the potential effectiveness of alternative versions of the weight  set prior to selecting one for implementation would provide CBP with  more information to make an informed decision. In January 2010, the  DHS Office of Inspector General recommended that CBP enhance its  documentation efforts to ensure that each stage of the process for  analyzing and developing ATS rules is documented, and CBP concurred  with this recommendation. As part of this recommendation, the DHS  Office of Inspector General recommended documenting the rationale for  making changes to ATS rules but did not specify what types of analyses  could demonstrate or support the rationale for making changes to the  rules. On the basis of our analysis, the rationale for updates to the weight  set could be further strengthened through assessments of effectiveness  beyond workload. For example, determining the expected TPR and FPR  for an alternative version of the weight set and comparing these  measures against the TPR and FPR for the existing version of the weight  set could enable CBP to determine if the alternative version of the weight  set could be expected to result in improved effectiveness, based on these  performance measures. This would enable CBP to quantitatively compare  the effectiveness of the alternative versions of the weight set being  considered prior to selecting one for implementation. Doing so, in addition  to the impact assessment, would provide CBP with reasonable assurance  that the version of the weight set it selects for implementation is the most  effective of the alternatives considered after taking into account any  resource constraints. Furthermore, assessing the alternative versions of  the weight set in the future would provide CBP with better assurance that  the version it selects for implementation is more effective than the  previous version of the weight set. CBP officials stated that they plan to  calculate and document measures of effectiveness during the planned  update to the weight set that will begin in the fall of 2012.
section_title: CBP Has Not Regularly Assessed the Weight Set against Performance Targets to Determine when Updates Are Needed
paragraphs: Since implementing the current version of the weight set in early 2011,  CBP has not regularly assessed the weight set against established  performance targets to monitor its performance and obtain information to  determine when updates to the weight set are necessary. We reported in  August 2006 that CBP intended to establish targets for the performance  measures to assess future performance of ATS, but CBP did not  establish such targets for those measures.
Targets could help CBP determine when updates are needed to improve  targeting effectiveness. For example, according to CBP’s analysis, the  TPR for summer 2011 through spring 2012 shows that, among shipments  CBP found to contain a potential threat during an examination, the weight  set accurately identified 6.3 percent as high risk, meaning that the weight  set classified 93.7 percent of shipments that carried a potential threat as  either medium risk or low risk. Furthermore, the FPR for that time period  shows that, among all the arriving shipments that CBP examined during  that time that did not pose a threat, the weight set identified 3.6 percent  as high risk. However, because CBP did not establish targets for either  TPR or FPR, it is not clear whether 6.3 percent for the TPR is sufficiently  low or 3.6 percent for the FPR is sufficiently high to suggest that changes  are needed to improve the performance of the weight set.
CBP’s project plan calls for conducting periodic performance  assessments by determining recurring measures of TPR and FPR.  Furthermore, according to CBP officials, the performance assessments  are to be conducted as part of quarterly reporting responsibilities.  However, CBP did not calculate these measures at the end of each  quarter, but instead calculated them as part of a single assessment in the  summer of 2012 and divided the results into quarters. Accordingly, CBP  was not aware of the ongoing performance of the weight set from its  implementation in early 2011 through spring 2012, and CBP was  therefore unable to determine for 18 months whether the weight set was  performing at a level that could require changes or updates to improve its  effectiveness. According to CBP officials, the summer 2012 assessment  was conducted at that time in preparation for a conference to discuss  updates to the weight set planned for the fall of 2012. CBP had decided to  hold this conference before CBP conducted the assessment, meaning its  decision for when to update the weight set was not based on information  about the weight set’s effectiveness from ongoing monitoring of CBP’s  performance measures for the weight set.
Ongoing monitoring is a key element of a risk management approach,  and CBP’s project plan calls for such periodic performance monitoring to  determine targeting effectiveness. In addition, standard practices for  internal control indicate that (1) ongoing monitoring should occur in the  course of normal operations and can be accomplished by periodic review  of performance measures and (2) in the process of ongoing performance  monitoring, actions should include continuous comparison of performance  data against planned targets and analysis of any differences to take  corrective actions as necessary.
CBP officials stated that personnel have not been consistently available  to conduct performance assessments since the initial contract with the  consulting firm ended in July 2011 and that resource concerns, such as  funding, the availability of subject matter experts, and the availability of  programmers may affect the timing of weight set updates. Nevertheless,  given the importance of the weight set to CBP’s process for targeting  cargo containers, regular performance assessments of the weight set that  include evaluating results against established performance targets could  help CBP determine when updates are needed in a timelier manner and  help it better prioritize the resources it needs to complete the updates.Furthermore, CBP officials stated that they intend to continue adjusting  the methodology for calculating the performance measures to mitigate  data limitations and more accurately reflect the performance of the weight  set. Such steps could help CBP ensure that its targeters have the best  information available regarding the risk of maritime cargo container  shipments arriving in the United States.
section_title: Conclusions
paragraphs: CBP recognizes the importance of and challenges to ensuring the  security of the global supply chain while facilitating the flow of legitimate  commerce. Although no events have occurred to date, terror-related  attacks on the supply chain could have devastating effects on the nation’s  security and economic well-being, and it is imperative that CBP use the  best information and tools available to continually mitigate potential  threats and address vulnerabilities. DHS and CBP face difficulties in  achieving 100 percent scanning of cargo containers prior to loading at  foreign ports and have, instead, advocated a risk-based approach to  target and scan those cargo containers that pose the highest risk. Given  the critical role that ATS plays as part of this risk-based approach, it is  important to ensure that ATS is performing effectively.
CBP plans to continue enhancing risk management processes, including  the use of ATS and its associated targeting rules. CBP’s determination of  which containerized shipments to review or to hold for examination is  based, in part, on the risk score. Thus, updating the weight set in ATS  that calculates this risk score is important for ensuring that targeters are  using the most effective tools in making targeting decisions. CBP has  assessed workload impacts when making updates to the weight set, but it  did not fully assess the weight set’s effectiveness as part of the most  recent update. As a result, CBP does not have reasonable assurance that  the implemented version is the most effective. Further, CBP did not  conduct periodic assessments as part of ongoing monitoring efforts.  Specifically, CBP did not conduct an assessment of the weight set until  18 months after CBP implemented the new weight set. We believe it is  important that CBP more regularly assess the performance of the weight  set in ATS that produces the risk scores and compare the results of this  assessment against established performance targets. Such steps could  help CBP determine when changes may be needed and ensure that its  targeters have the best information available regarding the risk of  maritime cargo container shipments arriving in the United States.
section_title: Recommendations for Executive Action
paragraphs: To enhance its targeting of maritime cargo containers and better position  CBP to provide reasonable assurance of the effectiveness of ATS, we  recommend that the Commissioner of CBP take the following two actions: ensure that future updates to the weight set are based on results of  assessments that demonstrate that the chosen version of the weight  set is more effective than other alternatives, including the existing  version, and establish targets for CBP’s performance measures and use those  measures to assess the effectiveness of the weight set on a regular  basis to better determine when updates to the weight set are needed.
section_title: Agency Comments
paragraphs: On October 17, 2012, DHS provided written comments on a draft of this  report, which are reprinted in appendix III. DHS concurred with the two  recommendations. Specifically, DHS concurred with the recommendation  to ensure that future updates to the weight set are based on the results of  assessments and stated that CBP plans to conduct analyses to ensure  that future versions of the weight set result in increased effectiveness.  DHS also noted that CBP is to conduct these analyses during the  development and deployment of future versions of the weight set.  According to DHS, these analyses would include performance measures,  subject matter expert input, current threat information, and other  intelligence. DHS stated that it expects these actions to be completed by  April 2013. Such actions should address the intent of the  recommendation to ensure improvements in the effectiveness of future  versions of the weight set. DHS also concurred with the recommendation  to establish targets for CBP’s performance measures and stated that CBP  is working to improve the current performance measures methodology.  DHS stated that, following approval of this methodology, CBP plans to  conduct quarterly reviews of the weight set to inform decision making.  DHS stated that it expects these actions to be completed by September  2013. If CBP takes these steps as planned and includes targets for any  performance measures that are part of the updated methodology, this  should address the intent of our recommendation. DHS also provided  technical comments, which we incorporated as appropriate.
As agreed with your office, unless you publicly announce its contents  earlier, we plan no further distribution of this report until 30 days after its  issue date. At that time, we will send copies of this report to the Secretary  of Homeland Security, appropriate congressional committees, and other  interested parties. In addition, the report will be available at no charge on  the GAO Web site at http://www.gao.gov.
If you or your staffs have any questions on this report, please contact me  at (202) 512-9610 or caldwells@gao.gov. Contact points for our Offices of  Congressional Relations and Public Affairs may be found on the last page  of this report. Staff acknowledgments are provided in appendix IV.
section_title: Appendix I: Federal Strategy for Ensuring the Security of Maritime Cargo Container Shipments
paragraphs: This appendix describes the core programs related to U.S. Customs and  Border Protection’s (CBP) strategy for ensuring the security of maritime  cargo container shipments. CBP has developed this strategy to mitigate  the risk of weapons of mass destruction, terrorist-related material, or  other contraband being smuggled into the United States in cargo  containers. CBP’s strategy is based on related programs that attempt to  focus resources on potentially risky cargo shipped in containers while  allowing other cargo containers to proceed without unduly disrupting  commerce into the United States. The strategy includes obtaining  advanced cargo information to identify high-risk containers, using  technology to inspect cargo containers, and partnering with foreign  governments and the trade industry. Table 1 provides a brief description  of the core programs that compose this security strategy.
section_title: Appendix II: Past Audit Findings and Recommendations to Improve the Targeting Process
paragraphs: Since 2004, we have conducted audits of CBP’s targeting process and  ATS. In particular, we published reports related to these topics in  February 2004, August 2006, and September 2010. In addition, the  Coast Guard and Maritime Transportation Act of 2004 requires the  Department of Homeland Security’s (DHS) Office of Inspector General  (OIG) to report annually on its evaluation of the current targeting system  for international intermodal cargo containers. Our and DHS OIG’s audits  have addressed, among other things, incorporating key elements of a risk  management framework and recognized modeling practices,  documenting the targeting rule development process, improving data  collection and ATS’s use of data, and providing additional information  outside of ATS to targeters. Collectively, these audits have made  recommendations to DHS and CBP for improving the targeting process,  and CBP has taken actions to implement them, although in some cases  CBP’s implementation efforts have been slow, leaving CBP without the  benefits of these improvements for several years. This appendix provides  an overview of key findings and recommendations from these audits, as  well as the status of actions taken to implement the recommendations.
subsections: section_title: Incorporating Key Elements of a Risk Management Framework and Recognized Modeling Practices
paragraphs: In February 2004, we reported that while CBP had taken steps to address  the terrorism risks posed by maritime cargo containers, its targeting  strategy did not incorporate all key elements of a risk management  framework, and ATS was not consistent with certain recognized modeling  practices. We recommended, among other things, that CBP improve the  targeting strategy by incorporating key elements of a risk management  framework and recognized modeling practices. These recommendations  included specific steps CBP should take to help ensure it could achieve  the objectives of its overall targeting strategy and better ensure that the  tools it uses to protect against terrorism are working effectively at the  nation’s ports. We later reported, in August 2006, that CBP had made  progress addressing the recommendations but had not yet fully  implemented the recommendations in our February 2004 report. Since  that time, CBP has fully implemented the February 2004  recommendations aimed at improving the targeting strategy by  incorporating key elements of a risk management framework and  recognized modeling practices. Table 2 provides more detail on our  February 2004 recommendations and CBP’s efforts to implement them.
In August 2006, among other things, we reported on the status of  recommendations issued in our February 2004 report and reiterated the  importance of the recommendations. Additionally, with respect to the  recommendation regarding risk assessments, at the time, CBP had  begun taking action to implement the recommendation by conducting and  using risk assessments that incorporated discussions of potential threats  and estimates of the relative importance of assets and vulnerabilities  associated with the supply chain. We noted, though, that CBP did not  have a methodology in place to guide its staff in identifying sources of  threat information, such as agencies to contact, what approach to take in  conducting the assessment, and key elements to include that would help  ensure consistency in the preparation of threat assessments associated  with the movement of cargo shipped in containers. As a result, we further  recommended that CBP establish and document a methodology for  conducting threat assessments associated with cargo shipped in  containers to help ensure that CBP staff responsible for conducting threat  assessments consult relevant information sources, prepare threat  assessments consistently, and include key elements to effectively  communicate risk to program managers.
In response to that recommendation, in October 2006, CBP issued a  protocol to assist its intelligence research specialists in preparing port  threat assessments in support of the Container Security Initiative. Under  this initiative, CBP places staff at foreign seaports to work with foreign  counterparts to inspect high-risk containers before they are shipped to the  United States. The October 2006 protocol included a discussion of  information sources for CBP staff to consult and the overall methodology  to follow in making the port threat assessment. CBP also distributed a  template that discussed the key elements its specialists should include  when preparing port threat assessments. In addition, CBP developed a  checklist for its specialists to use to help ensure that appropriate  information sources are consulted in making port threat assessments. To  assist in addressing threats related to global supply chain logistics and  the movement of maritime containers carrying cargo arriving in the United  States, CBP established an Office of Intelligence and Operations  Coordination in October 2007, which has since been renamed the Office  of Intelligence and Investigative Liaison. This office includes the Analysis  and Targeting Division (A&T), which is composed of program managers  with operational experience and intelligence analysts and is responsible  for conducting risk assessments of countries (known as country risk  profiles) that consider threats, vulnerabilities, and the associated criticality  of related assets from which or through which cargo is shipped to the  United States. In 2008, the A&T Division assessed risks in two primary  mission areas: (1) terrorism and weapons of mass destruction and (2)  narcotics. On the basis of these assessments, the A&T Division  developed a risk assessment methodology to rank countries according to  the level of risk associated with each mission area. The A&T Division  teams translated these rankings into scores, and program managers  integrated them into ATS. Specifically, CBP integrated the rankings for  the terrorism mission into ATS in 2008 and for the narcotics mission in  2009. Thus, CBP has fully addressed this recommendation and is better  positioned to ensure consistency in the preparation of risk assessments  associated with the movement of maritime cargo container shipments.
section_title: Documenting the Targeting Rule Development Process
paragraphs: In January 2010, the DHS OIG reported on several aspects of CBP’s  process for developing and updating targeting rules in ATS. In this  report, the DHS OIG stated that CBP could improve its process for  changing or deleting targeting rules by, among other things, documenting  (1) rule change decisions and (2) the testing and evaluation of rule  changes. Specifically, one component of the rule update process involves  the review of the proposed rule changes by subject matter experts, and  the DHS OIG reported that CBP could improve the process by ensuring  the rationale for changes implemented or not implemented are  documented and recorded for future use. Furthermore, the DHS OIG  reported that CBP tested new rules using actual data to determine how  well the new rules are working, but the DHS OIG noted that this process  for testing and evaluating the rules, and subsequent modifications of the  new rules, was not documented. The DHS OIG recommended that CBP  ensure it documents each stage of the process for analyzing and  developing ATS rules, including the rationale for making changes and the  details on tools used to improve application consistency and rule change  standardization. According to the DHS OIG report, in response to this  recommendation, CBP (1) developed a documentation process to capture  and record information that includes the rationale for rule changes and  the utilization of tools and (2) introduced more formality into the rules  process by implementing a structure to guide national conferences, rule  evaluation, targeting development, and process management, among  other things. The DHS OIG stated in its report that it considered the  actions taken by CBP to be responsive to the recommendation.
section_title: Improving Data Collection and ATS’s Use of Data
paragraphs: In February 2004, in addition to making the recommendations discussed  earlier, we also reported that CBP was relying on the manifest as its  principal data input, and CBP did not mandate the transmission of  additional information before a cargo’s risk level was assigned. We  reported that terrorism experts, members of the international trade  community, and CBP inspectors at the ports we visited as part of that  review characterized the ship’s manifest as one of the least reliable or  useful types of information for targeting purposes. We reported that  terrorism experts, trade community representatives, and some CBP  inspectors at ports we visited told us that CBP should explore requiring  more timely electronic transmittal of additional data elements for cargo  container targeting purposes, such as stowage plans (a map of where  each container aboard a ship is stored), container movement tracking  data, and entry data. Although we did not analyze the feasibility or the  costs and benefits of these suggestions, we reported that it could be  useful for CBP to explore requiring appropriate parties in the supply  chain, such as the importer, to provide additional data elements for use in  ATS to perform more complex linkage analyses and identify potential  anomalies in the shipping documents filed. Although we did not make a  recommendation directly related to data collection because we did not  analyze the feasibility or the costs and benefits of collecting additional  data, the recommendations in our report focused generally on  incorporating key elements of recognized modeling practices, and one of  the recognized modeling practices applicable to ATS is enhancing the  sources and types of information input into ATS. Furthermore, in July  2005, the DHS OIG issued an unclassified summary of an audit regarding  CBP’s targeting for maritime cargo containers, which concluded that  improvements were needed in the data to which ATS targeting rules are  applied.
In January 2009, CBP initiated an effort to collect additional data through  the Importer Security Filing and Additional Carrier Requirements,  collectively known as the 10+2 rule. The rule requires importers and  carriers to provide 10 data elements and 2 data elements, respectively, to  CBP for improving CBP’s ability to identify high-risk cargo container  shipments. Specifically, the 10+2 rule requires importers to submit  information about the commodities being transported in a shipment and  about entities involved in the supply chain. These additional data  elements include information that we reported could be helpful in  improving targeting efforts. In particular, the rule requires stowage plans  and some elements of entry data. We reported in September 2010 that  the 10+2 rule data elements were available for identifying high-risk cargo  at that time, but that CBP had not yet updated ATS to fully incorporate the  data into its targeting criteria. We recommended that CBP establish  milestones and time frames for updating the targeting criteria. In  December 2010, CBP provided us with a project plan for integrating the  data into its targeting criteria, and in early 2011, CBP implemented the  updated targeting criteria to address risk factors present in the Importer  Security Filing data.
section_title: Providing Additional Information Outside of ATS to Targeters
paragraphs: In November 2006, the DHS OIG reported on issues related to  information that is available to targeters to conduct targeting activities.  The DHS OIG found that CBP targeters did not always have access to a  particular system that directly accesses a database containing billions of  records on individuals and businesses. The DHS OIG recommended that  CBP provide targeters with access to that system or a similar system that  would allow targeters to access business records. In its report, the DHS  OIG stated that it considered the recommendation to be implemented  because CBP had acquired funding to grant personnel access to that  system and had issued a memo stating that all personnel that were to be  scheduled for targeting training should also have access to the system.
The DHS OIG also found that it was unclear which personnel involved in  targeting should have security clearances, and as a result, important  information affecting container targeting and inspection decisions may not  be available to the staff for making these decisions in a timely manner.  The DHS OIG recommended that CBP increase the number of targeters  with security clearances. In response to this recommendation, according  to the DHS OIG, CBP presented a corrective action plan with an  established completion date of June 30, 2007, for this recommendation.  The DHS OIG further noted that in July 2006, CBP issued a memo for  Security Clearance for Counter Terrorism Response Officers, directing  CBP field offices to forward applications of port personnel that need to  have security clearances. On the basis of these efforts, the DHS OIG  reported that CBP had satisfied the intent of this recommendation.
section_title: Appendix III: Comments from the Department of Homeland Security
section_title: Appendix IV: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Acknowledgments
paragraphs: In addition to the contact named above, Christopher Conrad (Assistant  Director), Alana Finley, Richard Hung, Katie Mauldin, and Janay Sam  made key contributions to this report. Also contributing to this report were  Richard Brown, Frances Cook, Stanley Kostyla, and Lara Miklozek.
section_title: Related GAO Products
paragraphs: Supply Chain Security: Container Security Programs Have Matured, but  Uncertainty Persists over the Future of 100 Percent Scanning.  GAO-12-422T. Washington, D.C.: February 7, 2012.
Maritime Security: Responses to Questions for the Record.  GAO-11-140R. Washington, D.C.: October 22, 2010.
Supply Chain Security: CBP Has Made Progress in Assisting the Trade  Industry in Implementing the New Importer Security Filing Requirements,  but Some Challenges Remain. GAO-10-841. Washington, D.C.:  September 10, 2010.
Supply Chain Security: Feasibility and Cost-Benefit Analysis Would Assist  DHS and Congress in Assessing and Implementing the Requirement to  Scan 100 Percent of U.S.-Bound Containers. GAO-10-12. Washington,  D.C.: October 30, 2009.
Intellectual Property: Better Data Analysis and Integration Could Help  U.S. Customs and Border Protection Improve Border Enforcement  Efforts. GAO-07-735. Washington, D.C.: April 26, 2007.
Cargo Container Inspections: Preliminary Observations on the Status of  Efforts to Improve the Automated Targeting System. GAO-06-591T.  Washington, D.C.: March 30, 2006.
Homeland Security: Summary of Challenges Faced in Targeting  Oceangoing Cargo Containers for Inspection. GAO-04-557T.  Washington, D.C.: March 31, 2004.