Cargo Container Inspections: Preliminary Observations on the Status of Efforts to Improve the Automated Targeting System
Released: 2006-03-30T00:00:00

Letter

Background

Oceangoing cargo containers have an important role in the movement of cargo between global trading partners. Approximately 90 percent of the world’s trade is transported in cargo containers. In the United States almost half of incoming trade (by value) arrives by containers aboard ships. If terrorists smuggled a weapon of mass destruction into the nation using a cargo container and detonated such a weapon at a seaport, the incident could cause widespread death and damage to the immediate area, perhaps shut down seaports nationwide, cost the U.S. economy billions of dollars, and seriously hamper international trade.

The Department of Homeland Security and CBP are responsible for addressing the threat posed by terrorist smuggling of weapons in oceangoing containers. To carry out this responsibility, CBP uses a layered security strategy. One key element of this strategy is ATS. CBP uses ATS to review documentation, including electronic manifest information submitted by the ocean carriers on all arriving shipments, to help identify containers for additional inspection. CBP requires the carriers to submit manifest information 24 hours prior to a United States-bound sea container being loaded onto a vessel in a foreign port. ATS is a complex mathematical model that uses weighted rules that assign a risk score to each arriving shipment in a container based on manifest information. As previously discussed, CBP officers use these scores to help them make decisions on the extent of documentary review or physical inspection to be conducted.

ATS is an important part of other layers in the security strategy. Under its CSI program, CBP places staff at designated foreign seaports to work with foreign counterparts to identify and inspect high-risk containers for weapons of mass destruction before they are shipped to the United States. At these foreign seaports, CBP officials use ATS to help target shipments for inspection by foreign customs officials prior to departing for the United States. Approximately 73 percent of cargo containers destined for the United States originate in or go through CSI ports.

ATS is also an important factor in the Customs-Trade Partnership Against Terrorism (C-TPAT) program. C-TPAT is a cooperative program linking CBP and members of the international trade community in which private companies agree to improve the security of their supply chains in return for a reduced likelihood that their containers will be inspected. Specifically, C-TPAT members receive a range of benefits, some of which could change the ATS risk characterization of their shipments, thereby reducing the probability of extensive documentary and physical inspection.

CBP Currently Does Not Have Reasonable Assurance That ATS Is Effective

CBP does not yet have key controls in place to provide reasonable assurance that ATS is effective at targeting oceangoing cargo containers with the highest risk of containing smuggled weapons of mass destruction. To address this shortcoming, CBP is (1) developing and implementing performance metrics to measure the effectives of ATS, (2) planning to compare the results of randomly conducted inspections with the results of its ATS inspections, (3) developing and implementing a simulation and testing environment, and (4) addressing recommendations contained in a 2005 peer review. To date, none of these control activities have been fully completed or implemented. Thus, CBP does not yet have key internal controls in place to be reasonably certain that ATS is providing the best available information to allocate resources for targeting and inspecting containers that are the highest risk and thus not overlook inspecting containers that pose a high threat to the nation.

CBP Does Not Yet Have Performance Measures to Gauge the Effectiveness of ATS in Targeting Cargo Containers, But is Working to Develop Them

CBP does not yet have performance measures in place to help it determine the effectiveness of ATS at targeting oceangoing cargo containers with the highest risk of smuggled weapons of mass destruction. The Comptroller General’s internal control standards include the establishment and review of performance measures as one example of a control activity to help an entity ensure it is achieving effective results. In July 2005, CBP contracted with a consulting firm to develop such performance metrics. CBP officials and personnel from this consulting firm told us that the firm’s personnel analyzed shipment information in ATS over a 2-year period to obtain additional insights into ATS’s performance and to determine whether ATS is more effective at targeting cargo containers for terrorism related risk than a random sampling inspection approach. CBP officials told us that the consulting firm’s personnel prepared a draft of the results of their analyses and that, as of March 21, 2006, CBP officials are reviewing these analyses. They also said that the consulting firm’s personnel are documenting the methodology for their analyses and related performance measures that CBP can use in the future. CBP officials expect to receive this methodology and the performance measures in April 2006, and told us that they expect to begin using the measures in June 2006. CBP officials also told us that they initially planned to have performance measures developed by August 31, 2005, but that this process has taken longer than expected because of delays in (1) obtaining security clearances for the consulting firm’s personnel, (2) obtaining workspace for the firm’s staff, and (3) arranging for the appropriate levels of access to CBP’s information systems.

CBP Is Not Yet Using the Results of Random Inspections to Assess ATS Effectiveness

Currently, CBP is not using the results of its random sampling program to assess the effectiveness of ATS. As part of its Compliance Measurement Program, CBP plans to randomly select 30,000 shipments based on entry information submitted by the trade community and examine those shipments to ensure compliance with supply chain security during fiscal year 2006.

At this time, CBP is unable to compare the examination results from its random sampling program with its ATS inspection results, as we recommended in our 2004, report because CBP does not yet have an integrated, comprehensive system in place to compare multiple sets of data—like results of random inspections with results of routine ATS inspections that were triggered by ATS scores and other operational circumstances. Such a comparison would allow examination of if and why the outcomes of ATS’s weighted rule sets are not consistent with the expected outcomes possible in the universe of cargo containers, based on sample projections. Furthermore, the Comptroller General’s standards for internal control state that information should be recorded and communicated to management and others within the entity who need it in a form that enables them to carry out their responsibilities.

CBP Has Not Yet Tested the Effectiveness of ATS in Targeting Cargo Containers for Inspection but Has Plans to Do So

Currently, CBP does not conduct simulated events (e.g., covert tests and computer-generated simulations)—a key control activity—to test and validate the effectiveness of ATS in targeting oceangoing cargo containers with the highest risk of containing smuggled weapons of mass destruction and has not yet implemented a dedicated simulation and testing environment. Without testing and validation, CBP lacks a vital mechanism for evaluating ATS’s ability to identify high-risk containers.

In July 2005, CBP contracted with a consulting firm to obtain assistance in the development of a computer-generated simulation and testing environment. CBP officials report that they have the simulation environment infrastructure in place and have processed mock manifest data to simulate cargo linked to terrorism in the new environment. CBP is currently reviewing the results of this test. Further, CBP officials told us that the consulting firm is continuing to work with CBP to develop system requirements so that officers can effectively use the simulation environment. CBP expects to receive the consulting firm’s final input for the simulation and testing environment by June 2006. CBP officials said that they cannot estimate when this simulation and testing environment will be fully operational until CBP receives the consulting firm’s final product. As with the development of performance measures, CBP officials also told us that this process has taken longer than expected because of delays in (1) obtaining security clearances for the consulting firm’s personnel, (2) obtaining workspace for the firm’s staff, and (3) arranging for the appropriate levels of access to CBP’s information systems.

As we reported in 2004, terrorism experts suggested that testing ATS by covertly simulating a realistic event using probable methods of attack would give CBP an opportunity to examine how ATS would perform in an actual terrorist situation. CBP officials told us that although they are considering implementing this kind of practice, they do not currently have a program in place to conduct such tests. The Director of CBP’s Management Inspections and Integrity Assurance office told us that in mid-April 2006, his office will be presenting a proposal to the Acting Commissioner and other senior management to request initiation of a program to conduct testing of the CSI program that will include testing ATS to help ensure that it is appropriately targeting the highest-risk cargo in the CSI program.

CBP Is Working to Address Peer Review Recommendations

In response to our 2004 recommendation that CBP initiate an external peer review of ATS, CBP contracted with a consulting firm to evaluate CBP’s targeting methodology and recommend improvements. Specifically, the contractor identified strengths of the CBP targeting methodology and compared ATS with other targeting methodologies. However, the peer review did not evaluate the overall effectiveness of ATS because CBP did not have the systems in place to allow the contractor to do so.

The contractor’s final report, issued in April 2005, identified many strengths in the ATS targeting methodology, such as a very capable and highly dedicated team and the application of a layered approach to targeting. It also made several recommendations to improve the targeting methodology that included control activities, such as (1) the development of performance measures, (2) the development of a simulation and testing environment, (3) the development and implementation of a structured plan for continual rules enhancement, and (4) an evaluation and determination of the effectiveness of the ATS targeting rules, several of which reinforced the recommendations we made in our 2004 report.

CBP issued a detailed plan, which projected delivery dates, for responding to the recommendations made in the contractor’s final report. However, about half of these dates have not been met. For example, CBP projected that it would have its testing and simulation environment in place by September 30, 2005. Although CBP has been working on this effort, the environment has not yet been implemented. As previously discussed, CBP officials said that they cannot provide a current estimate of when this simulation and testing environment will be fully operational.

Although CBP Strives to Refine ATS for Intelligence Information and Officer Feedback, It Is Not Yet Positioned to Use Inspection Results

CBP strives to refine ATS to include intelligence information it acquires and feedback it receives from its targeting officers at the seaports, but it is not able to systematically adjust ATS for inspection results. CBP does not have a comprehensive, integrated system in place to report details on security inspections nationwide that will allow management to analyze those inspections and refine ATS. CBP officials said that they are developing a system that will allow them to do so but did not know when it will be fully operational. CBP officials cautioned that because an inspection does not identify any contraband or a weapon of mass destruction or its components, it may not necessarily indicate that a particular rule is not operating as intended. They noted that terrorist incidents may happen infrequently, and the rule therefore might operate only when weapons, materials, or other dangerous contraband is actually shipped. However, without analyzing and using security inspection results to adjust ATS, CBP is limited in refining ATS, a fact that could hinder the effectiveness of CBP’s overall targeting strategy.

CBP Adjusts ATS for Targeting Cargo Containers for Inspection Based on Intelligence

CBP adjusts ATS’s rules and weights for targeting cargo containers for inspection in response to intelligence received on an ongoing basis. CBP’s Office of Intelligence (OINT) is responsible for acquiring, reviewing, analyzing, and disseminating intelligence. OINT officials told us they receive information from the intelligence community, which includes federal agencies such as the Central Intelligence Agency and the Federal Bureau of Investigation. According to OINT officials, OINT disseminates information to CBP’s offices at the seaports to, among other things, support these offices’ targeting efforts related to cargo containers. For example, the targeting officers may use information provided by OINT to search ATS for information about shipments and containers. OINT officials said they also disseminate information to CBP’s senior management to inform them about risks associated with cargo containers. CBP uses intelligence information to refine its targeting of cargo containers for inspection by incorporating the intelligence information into ATS to readily identify containers whose manifest information may match or be similar to data contained in the intelligence information.

CBP documentation and our observations showed that CBP headquarters personnel incorporate intelligence information into ATS by adjusting ATS’s existing rules and weights and creating new rules and weights that result in a higher risk score being assigned to a container whose manifest information may match or be similar to data contained in the intelligence information. CBP officers can also conduct queries or create lookouts in ATS that will search all manifest data in the system to identify those containers whose manifest information may match or be similar to data contained in the intelligence information. Once ATS identifies these containers, CBP officers are to then designate these containers for inspection. When CBP receives credible intelligence information that requires immediate action, CBP officials also report that they can initiate a special operation to address specific concerns identified in the intelligence data. CBP officials at the six seaports we visited reported that they sometimes receive intelligence information from local sources such as state and local law enforcement. Officials at five of these seaports reported that they will use such information to help them make decisions regarding targeting efforts. Additionally, officials at five of the six seaports we visited said that if the information they receive has national implications, they will notify CBP headquarters personnel, who will make a determination regarding potential adjustments to ATS.

CBP Targeting Officers at the Seaports Have Provided Few Suggestions for Adjusting ATS

In the late summer of 2005, CBP headquarters initiated a process to formally track its targeting officers’ suggestions to enhance ATS for targeting cargo containers for inspection. Targeting officers at all six seaports we have visited are aware of the process for providing suggestions to CBP headquarters. According to documentation maintained by headquarters, CBP officers at the seaports have provided few suggestions to date.

CBP headquarters officials said that although they have received few suggestions for modifying ATS, they do not believe this is an indication of ATS’s effectiveness. These officials stated that overall the feedback they have received from CBP targeting officers at the seaports related to the operation and usefulness of ATS has been positive.

We reviewed the report CBP uses to track these suggestions and found that since it was established, CBP headquarters has received 20 suggestions for enhancing the ATS component responsible for targeting oceangoing cargo containers for inspection. Some of these suggestions relate to modifying ATS’s rules, while others focused on other aspects of ATS such as enhancing the organization and presentation of ATS screens by changing the size of an icon and the fonts or text used.

CBP Is Not Using Inspection Results to Systematically Adjust ATS, but It Is Developing a System to Allow it to Do So

CBP is not using inspection results to systematically adjust ATS for targeting cargo containers for inspection because CBP does not yet have a comprehensive, integrated system in place that can report sufficient details for analyzing inspection results. CBP officials said that although they can analyze inspection results on a case-by-case basis to identify opportunities to refine ATS, such as when an inspection results in a seizure of some type of contraband, they currently do not have a reporting mechanism in place that will allow them to view inspection results nationwide to identify patterns for systematically adjusting ATS. CBP is developing the Cargo Enforcement Reporting Tracking System (CERTS) to document, among other things, all cargo examinations so that documentation substantiating the examinations will be available for analysis by management to adjust ATS. CBP officials said they will begin testing CERTS in the spring of 2006. CBP officials told us that once testing of CERTS is complete, they will be in a better position to estimate when CERTS can be fully implemented.

CBP officials cautioned that because an inspection does not identify any contraband or a weapon of mass destruction or its components, it may not necessarily indicate that a particular rule is not operating as intended. They noted that terrorist incidents may happen infrequently and the rule therefore might operate only when weapons, materials, or other dangerous contraband is actually shipped. However, without using inspection results to adjust ATS, CBP may not be targeting and inspecting containers with the highest risk of containing smuggled weapons of mass destruction.

CBP Has Taken Steps to Better Implement the Targeting Strategy at the Seaports

CBP has implemented a testing and certification process for its officers who complete the Sea Cargo Targeting Course that should provide better assurance of effective targeting practices. CBP has also made a good faith effort to address longshoremen’s safety concerns regarding radiation emitted by nonintrusive inspection equipment. Nevertheless, it has not been able to persuade one longshoremen’s union to permit changes in the procedure for staging containers to increase inspection efficiency.

CPB Has Implemented a Testing and Certification Process for Officers Who Target Cargo Containers for Inspection

In our 2004 report, we recommended that CBP establish a testing and certification process for CBP staff who complete the national targeting training to provide reasonable assurance that they have sufficient expertise to perform targeting work. CBP has implemented such a testing and certification process.

CBP conducted two evaluations that assessed its targeting training program—a job performance assessment and a job task analysis. With the results of these evaluations, CBP concluded that a certification component should be added to the training program and the Sea Cargo Targeting Training course content should remain unchanged. CBP officials then updated the course materials to encompass the inclusion of the certification component. In October 2004, CBP began certifying officers who successfully completed the Sea Cargo Targeting Training course. Since the establishment of the testing and certification component for the Sea Cargo Targeting Training course, CBP data indicate that it has trained and certified 278 of its officers responsible for targeting cargo as of March 24, 2006.

While CBP has conducted a job performance assessment prior to the incorporation of a certification program for Sea Cargo Targeting Training, it has not yet formally assessed the impact that revised training and certification has had on officers’ targeting of oceangoing cargo containers. However, a CBP official said that CBP has recently initiated planning efforts to begin such an evaluation and expects to complete the evaluation in May 2006. Nevertheless, supervisory officers from five of the six CBP offices at the seaports we visited said that the mandatory training and certification program has been beneficial. These supervisory officers told us that the training and certification improves the confidence of targeters, provides the ability for officers to improve their targeting productivity, and provides an opportunity for officers to gain a broader perspective into the targeting environment by examining passenger and outbound targeting.

Despite CBP Action to Address Longshoremen’s Safety Concerns, Efficiency Concerns Remain on the West Coast

In our 2004 report, we discussed concerns that longshoremen had regarding the safety of driving cargo containers through the gamma ray imaging system, one type of nonintrusive inspection equipment used to examine containers to detect potential contraband or weapons of mass destruction. Because this equipment emits radiation as it takes images of the inside of cargo containers, some longshoremen expressed concerns about the health effects of this radiation. As a result of these safety concerns, the longshoremen’s union representing West Coast longshoremen established a policy that prevents its members from driving containers through the gamma ray imaging system. In response, CBP altered its procedures at ports affected by this policy. For example, at some West Coast ports, CBP allows longshoremen to stage cargo containers away from the dock, in rows at port terminals, so that CBP officers can then drive the gamma ray imaging system over a group of containers.

However, this procedure can be space-intensive and time-consuming compared to the procedure utilized at East and Gulf Coast ports, whereby the gamma ray imaging system machinery is operated by a CBP officer and parked in place while longshoremen drive the cargo containers through the machinery. At other West Coast ports, the longshoremen get out of the trucks after transporting the cargo containers so that CBP officials can drive the gamma ray imaging system cargo over the container. This is also time-consuming compared to the procedure utilized at the East and Gulf Coast ports.

In response to our recommendation that CBP work with longshoremen to address their safety concerns, CBP engaged in two efforts: (1) establishing CBP’s radiation threshold in accordance with the Nuclear Regulatory Commission’s (NRC) federal guidelines for public radiation exposure and advertising this threshold to longshoremen through the unions, and (2) working with longshoremen’s unions and other maritime organizations to develop public radiation tests on nonintrusive inspection equipment. Officials from the West Coast union that prohibits its members from driving through the gamma ray imaging system told us that the union is satisfied with CBP efforts to operate the gamma ray imaging system in an alternative format, to comply with the union’s policy of receiving no amount of man-made radiation. Despite CBP efforts to assure this union that the amount of radiation emitted by the gamma ray imaging system is within safe levels, a union representative told us that CBP will not convince the union to change its policy unless it eliminates radiation emission from inspection equipment. - - - - - In closing, ATS is an integral part of CBP’s layered security strategy. A well-functioning ATS is crucial to the effective screening of cargo containers at domestic and CSI foreign ports, as well as cargo shipped by the trade community participating in C-TPAT. While CBP is working to make improvements to ATS, our ongoing work indicates that it is not yet in a position to gauge the effectiveness of ATS. We are continuing to review CBP’s plans and actions to improve ATS and will report to this subcommittee and the other requesters later this year.

GAO Contacts and Acknowledgments

Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. For further information about this testimony, please contact me at 202-512-8777 or at stanar@gao.gov. Debra Sebastian, Assistant Director; Chan-My J. Battcher; Lisa L. Berardi; Wayne A. Ekblad; and Jessica A. Evans made key contributions to this report. Additional assistance was provided by Frances Cook, Kathryn E. Godfrey, Nancy A. Hess, Arthur L. James, Jr., Stanley J. Kostyla, and Vanessa R. Taylor.

Appendix I: Scope and Methodology

To address each of our objectives, we met with U.S. Customs and Border Protection (CBP) officials in headquarters and six seaports including, Baltimore, Charleston, Los Angeles-Long Beach, Miami, New York- Newark, and Savannah. These seaports were selected based on the number of cargo containers arriving at the seaport and their geographic dispersion as reported by the U.S. Department of Transportation. At these locations, we also observed targeting and inspection operations. Because we did not select a random, probability sample of ports to visit, the results from these visits cannot be generalized to ports nationwide. We also spoke with CBP’s contractor responsible for conducting CBP’s peer review and longshoremen’s union representatives.

To evaluate how CBP provides assurance that the Automated Targeting System (ATS) targets the highest-risk oceangoing cargo containers for inspection, we reviewed CBP documentation and prior GAO work on performance measures. Additionally, we reviewed CBP’s peer review report. To gain an understanding of CBP’s random sampling program, we met with CBP officials responsible for this program and reviewed and analyzed CBP documentation, including procedures for examining the randomly selected shipments and documenting the results of the inspections completed for those shipments. We did not independently validate the reliability of CBP’s targeting results.

To assess how CBP adjusts ATS to respond to findings that occur during the course of its operational activities, we met with CBP officials responsible for gathering and disseminating intelligence and for incorporating intelligence into CBP’s targeting operations. Further, we reviewed CBP policies and procedures on intelligence gathering and disseminating as well as intelligence received and resulting changes to ATS rules and weights. We did not assess the quality of intelligence received or the appropriateness of adjusted rules and weights. To determine how targeting officers’ feedback and inspection results are used to adjust ATS rules and weights, we met with CBP officials responsible for collecting and maintaining data on suggestions provided by targeting officers and reviewed CBP data on the suggestions received over a 7 month period. Regarding inspection results, we reviewed CBP’s policies and procedures for documenting inspection results. Additionally, we reviewed CBP’s manuals identifying the specific details of an inspection completed and observed officers entering inspection results into the ATS findings module during our site visits. Further, during these visits, we discussed how CBP offices at the seaports may use inspection results to enhance their targeting efforts. Last, we met with CBP officials and reviewed CBP documentation on its current and planned findings module.

To determine the status of recommendations from GAO’s February 2004 report to (1) establish a testing and certification process for CBP staff who complete the national targeting training to provide assurance that they have sufficient expertise to perform targeting work and (2) work with longshoremen’s unions to address fully their safety concerns so that the noninstrusive inspection equipment can be used to conduct inspections efficiently and safely, we reviewed and analyzed data on the number of officers trained and certified in sea cargo targeting. We also reviewed CBP’s Sea Cargo Training Manual as well as CBP evaluations assessing the quality of its Sea Cargo Training course. We did not assess the quality of this training. Regarding longshoremen’s union concerns, we reviewed scientific literature related to radiation safety and the Nuclear Regulatory Commission guidelines on radiation levels. We also spoke with longshoremen’s representatives to discuss whether CBP had addressed their concerns since we issued our 2004 report. Last, we also met with CBP’s Radiation Safety Officer to gain a further understanding of the potential risks associated with CBP’s inspection equipment and actions he took to address longshoremen’s concerns. We did not assess the appropriateness of radiation safety levels used by CBP.

We conducted our work from October 2005 through March 2006 in accordance with generally accepted government auditing standards.

This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.