Supply Chain Security: Examinations of High-Risk Cargo at Foreign Seaports Have Increased, but Improved Data Collection and Performance Measures Are Needed
Released: 2008-01-25T00:00:00

Letter

Background

States. We disagree. As discussed in this report, a core element of the CSI program, specifically the extent to which U.S.-bound containers carrying high-risk cargo are examined at CSI seaports, is not addressed through CBP’s performance measures.

Vulnerabilities of Containers in the International Supply Chain

Seaports are critical gateways for the movement of commerce through the international supply chain. The facilities, vessels, and infrastructure within seaports, and the cargo passing through them, all have vulnerabilities that terrorists could exploit. The containers carrying goods that are shipped in oceangoing vessels are of particular concern because they can be filled overseas at many different locations and are transported through complex logistics networks before reaching U.S. seaports.

In addition, transporting such a shipping container from its international point of origin to its final destination involves many different participants and many points of transfer. The materials in a container can be affected not only by the manufacturer or supplier of the material being shipped, but also by carriers who are responsible for getting the material to a port and by personnel who load containers onto the ships. Others who interact with the cargo or have access to the records of the goods being shipped include, among others, exporters who make arrangements for shipping and loading, freight consolidators who package disparate cargo into containers, and forwarders who manage and process the information about what is being loaded onto the ship. Figure 1 illustrates many of the key participants and points of transfer involved from the time that a container is loaded for shipping to its arrival at the destination seaport and ultimately the importer.

Several studies on maritime security conducted by federal, academic, nonprofit, and business organizations have concluded that the movement of oceangoing cargo in containers is vulnerable to some form of terrorist action, largely because of the movement of cargo throughout the supply chain. Every time responsibility for cargo in containers changes hands along the supply chain there is the potential for a security breach, and thus, vulnerabilities exist that terrorists could take advantage of by placing a WMD into a container for shipment to the United States. While there have been no known incidents of containers being used to transport WMDs, criminals have exploited containers for other illegal purposes, such as smuggling weapons, people, and illicit substances, according to CBP officials. Finally, while CBP has noted that the likelihood of terrorists smuggling WMD into the United States in cargo containers is low, the nation’s vulnerability to this activity and the consequences of such an attack are potentially high. In 2002, Booz Allen Hamilton sponsored a simulated scenario in which the detonation of weapons hidden in cargo containers shut down all U.S. seaports over a period of 12 days. The results of the simulation estimated that the port closure could result in a loss of $58 billion in revenue to the U. S. economy, along with significant disruptions to the movement of trade.

Efforts to Secure Containers in the International Supply Chain

The federal government has taken many steps to secure the supply chain, including the cargo in containers destined for the United States. While CBP officials at domestic seaports continue efforts to identify and examine high-risk imports arriving in containers, CBP’s post-September 11 strategy also involves focusing security efforts beyond U.S. borders to target and examine high-risk cargo before it enters U.S. seaports. CBP’s strategy is based on a layered approach of related initiatives that attempt to focus resources on potentially risky cargo shipped in containers while allowing other containers carrying cargo to proceed without unduly disrupting commerce into the United States. CBP has initiated most of these efforts, shown in table 1. However, the Department of Energy (DOE) has led U.S. efforts to detect radiation in cargo containers originating at foreign seaports.

CBP uses this computerized decision support tool to review documentation, including electronic manifest information submitted by ocean carriers on all cargo destined for the United States to help identify shipments requiring additional scrutiny. ATS utilizes complex mathematical models with weighted rules that assign a risk score to each shipment based on manifested information. CBP officers review the rule firings that support the ATS score to help them make decisions on the extent of documentary review or examination to be conducted.

CBP generally requires ocean carriers to electronically transmit cargo manifests to CBP’s Automated Manifest System 24 hours before the U.S.-bound cargo is loaded onto a vessel at a foreign seaport. Carriers and importers are to provide information to CBP that is used to strengthen how ATS assigns risk scores. The cargo manifest information is submitted by ocean carriers on all arriving cargo shipments, and entry data (more detailed information about the cargo) are submitted by brokers.

CSI places staff at participating foreign seaports to work with host country customs officials to target and examine high-risk cargo to be shipped in containers for weapons of mass destruction before they are shipped to the United States. CBP officials identify the high-risk containers and request that their foreign counterparts examine the contents of the containers.

CBP develops voluntary partnerships with members of the international trade community comprised of importers; customs brokers; forwarders; air, sea, and land carriers; and contract logistics providers. Private companies agree to improve the security of their supply chains in return for various benefits, such as a reduced likelihood that their containers will be examined.

DOE installs radiation detection equipment at key foreign seaports, enabling foreign government personnel to use radiation detection equipment to screen shipping containers entering and leaving these seaports, regardless of the containers’ destination, for nuclear and other radioactive material that could be used against the United States and its allies.

Pilot program at selected CSI seaports to scan 100 percent of U.S.-bound cargo containers for nuclear and radiological materials overseas using integrated examination systems that couple nonintrusive inspection equipment and radiation detection equipment.

In January 2002, CBP began CSI to target container cargo at overseas seaports so that high-risk cargo could be examined prior to departure for the United States. More recently, Congress passed legislation affecting the CSI program, including (1) the SAFE Port Act enacted in October 2006 that established a statutory framework for CSI and, among other things, required a pilot program, now known as the Secure Freight Initiative, to determine the feasibility of 100 percent scanning of U.S.-bound cargo containers at foreign seaports; and (2) the 9/11 Act enacted in August 2007, that, among other things, requires by 2012, the scanning of all U.S.-bound containers at foreign seaports with potential exceptions if a seaport cannot meet that deadline.

For the CSI program, CBP officials stated that DHS expended about $138 million and $143 million, respectively for fiscal years 2006 and 2007. The President’s budget for fiscal year 2008 requested $156 million for CSI. CSI is now operating at 58 seaports in 33 foreign countries, as shown in figure 2. Appendix III lists the specific CSI seaports.

Core Elements and Security Activities of CSI Program

According to CBP, the three core elements of CSI include (1) CBP identifying high-risk containers; (2) CBP requesting, where necessary, that host governments examine high-risk containers before they are shipped; and (3) host governments conducting examinations of high-risk containers. To integrate these elements into CSI operations, CBP negotiated and entered into bilateral, nonbinding arrangements with foreign governments, specifying the placement of CBP officials at foreign seaports and the exchange of information between CBP and foreign customs administrations. To participate in CSI, a host nation must meet several criteria developed by CBP. The host nation must utilize (a) a seaport that has regular, direct, and substantial container traffic to seaports in the United States; (b) customs staff with the capability of examining cargo originating in or transiting through its country; and (c) nonintrusive inspection equipment with gamma or X-ray capabilities and radiation detection equipment. Additionally, each potential CSI port must indicate a commitment to (d) establish an automated risk management system for identifying potentially high-risk container cargo; (e) share critical data, intelligence, and risk management information with CBP officials; (f) conduct a seaport assessment to ascertain vulnerable links in a port’s infrastructure and commit to resolving those vulnerabilities; and (g) maintain a program to prevent, identify, and combat breaches in employee integrity.

As part of the arrangements with foreign governments participating in CSI, CBP most often stations teams of CBP officers at each foreign seaport to conduct CSI activities in collaboration with host government customs officials. While the number of CBP officers stationed at CSI seaports varies by location, typically a CSI team consists of (1) a CSI team leader, who manages the team and monitors the relationship with the host country; (2) CBP officers, who target high-risk cargo and observe (where possible) the host government’s examination of containers carrying the cargo; (3) an intelligence research specialist, who assimilates data to support timely and accurate targeting of containers; and (4) a special agent responsible for CSI-related investigations at the seaport. According to CBP, it is ideal for the CSI team to be located in close physical proximity with host government customs counterparts to facilitate collaboration and information sharing. However, CBP officials also stated that the agency uses CBP officers stationed at the NTCC as needed to support the CBP officers located at the CSI seaports. The CBP officials at NTCC assist the CSI teams at high-volume seaports to ensure all containers that pass through CSI seaports are targeted to identify high-risk container cargo; carry out CSI targeting responsibilities for CSI seaports that do not have CBP officials stationed there; and, according to CBP officials, conduct targeting for U.S.-bound container cargo that does not pass through CSI seaports using national sweeps to identify high-risk container cargo.

At CSI seaports, CBP officers share responsibilities with host governments’ customs officials to target and examine high-risk container cargo. Figure 3 describes the activities carried out by CBP officers and host government customs officials, respectively, to target and examine high-risk container cargo at CSI seaports.

CBP Collaborated on the DHS Strategy to Enhance International Supply Chain Security, and Met Goals for CSI Expansion and Increased Container Examination

CBP has undertaken strategic planning to guide efforts to secure the international supply chain and, more specifically, to manage the CSI program. CBP contributed to an international supply chain security strategy DHS recently issued that builds on DHS’s existing strategic framework for maritime security. In 2006 CBP enhanced its strategic plan for CSI by including three key elements missing from the plan’s previous iteration, and has achieved two performance goals by expanding CSI locations and increasing the percentage of total U.S.-bound containers that pass through CSI seaports. Concurrently, CBP reported an increase in the number of high-risk containers examined by host governments participating in CSI.

Recently Issued International Supply Chain Security Strategy Builds on DHS’s Existing Strategic Framework for Maritime Security

When it published the Strategy to Enhance International Supply Chain Security in July 2007, DHS filled a gap that had existed between broad national strategies and program-specific plans in the federal government’s strategic planning framework for maritime security. Over the last 5 years, DHS has made progress in developing a multilayered strategic framework for securing the maritime domain, including the international supply chain. This framework consists of high-level national strategies, such as the National Strategy for Maritime Security and the Maritime Commerce Security Plan, which describe the federal government’s broad approach to maritime security. These plans are supplemented by a related hierarchy of documents that includes the DHS strategic plan, the CBP strategic plan, and the CSI program’s own strategic plan.

Prior to July 2007, the federal government’s maritime security framework touched on many specific aspects of maritime trade and commerce, such as how the CSI program contributes to securing containers bound for U.S. seaports. However, it did not provide a detailed description of how federal, state, and local authorities were to collaborate on supply chain security specifically. In addition, Congress included a provision in the SAFE Port Act of 2006 requiring DHS to develop a strategic plan to enhance the security of the international supply chain. Moreover, the DHS fiscal year 2007 appropriation act withheld $5 million from DHS until a comprehensive strategic plan for port, cargo, and container security, which included specific elements, had been submitted to specified congressional committees. In response, CBP contributed to the Strategy to Enhance the International Supply Chain Security, which DHS developed and issued in July 2007. According to DHS, the supply chain security strategy is not meant to replace other strategic planning documents, but seeks to harmonize the goals of the various plans and programs into a multilayered, unified approach that can be further developed by DHS components, including CBP.

This new strategic planning document for supply chain security delineates the supply chain security roles, responsibilities, and authorities of federal, state, local, and private sector entities. The strategy seeks to build on the current multilayered strategic framework for maritime security by establishing an overarching framework for the secure flow of cargo through the supply chain—from point of origin to final destination. The strategy describes how CBP’s portfolio of supply chain security initiatives—including CSI, C-TPAT, cargo screening using ATS, the 24-hour rule, and the use of nonintrusive inspection equipment to examine containers—addresses the various stages in the supply chain. In addition, the strategy provides details on how other organizations’ programs or efforts—such as DOE’s Megaports initiative, which places radiation detection equipment at foreign seaports—contribute to different aspects of supply chain security. Figure 4 describes the major components of the supply chain and the CBP initiatives that operate to secure them.

CBP Added Key Elements to the CSI Strategic Plan in Response to our 2005 Recommendation

At the program level, CBP has revised its CSI strategic plan, an important component of the DHS strategic framework described above, incorporating three critical elements that were absent from the plan’s previous iteration. In our April 2005 report on CSI, we reported that the CSI strategic plan lacked three of the six key elements identified by the Government Performance and Results Act of 1993 for an agency strategic plan, including descriptions of 1. how performance goals and measures are related to program objectives, 2. the external factors beyond the control of CBP that could affect the achievement of program objectives, and 3. the evaluations that CBP conducts to monitor CSI.

We noted that, given the importance of having an effective strategic plan for the program, we would continue to monitor CBP’s progress in refining the plan. CBP has subsequently taken steps to address our concerns. In the most recent version of the plan, released in August 2006, CBP included information in three areas, as we had previously recommended. First, the CSI strategic plan links each performance measure to the strategic goal it supports. In addition, the plan describes how some performance measures were designed to act as proxies for program objectives that can be difficult to measure. Second, the CSI strategic plan also lists a variety of external factors that have the potential to influence CSI operations, including regional conflicts, organized crime, and changes in the political administration of a foreign government participating in CSI. Finally, the revised plan provides an explanation of the CSI team evaluation process, thus addressing the third issue identified in our April 2005 report. We discuss performance measure outcomes, other external factors, and CBP’s evaluation process in greater detail later in this report.

CBP Met Performance Goals to Expand Number of CSI Seaports and to Increase Proportion of Total U.S.-bound Containers Passing Through CSI Seaports

The August 2006 CSI strategic plan set specific goals for expanding the number of seaports participating in CSI, and set targets for related increases in the percentage of total U.S.-bound containers that pass through CSI seaports. As of September 2007, CBP reported meeting its goals in both of these areas. Specifically, the plan called for CBP to expand CSI program operations from 40 to 50 seaports by the end of fiscal year 2006, and to 58 seaports by the end of fiscal year 2007 (see appendix III for a complete list of participating seaports). Having reached its goal of 58 CSI seaports, CBP officials reported it currently does not have plans to add other CSI seaports, as the costs associated with expanding the program further would outweigh the potential benefits. In addition, the plan set a performance target that by 2010, 86 percent of all U.S.-bound container cargo was to pass through CSI seaports. According to CBP, when U.S.- bound containers pass through CSI seaports there is an opportunity for high-risk cargo to be examined at the foreign seaport by the host governments participating in CSI, rather than upon arriving at a U.S. seaport. CBP reported that about 73 percent and about 80 percent of total U.S.-bound container cargo passed through CSI seaports in fiscal years 2005 and 2006, respectively, and that it reached its 2010 goal early by reaching approximately 86 percent by the end of fiscal year 2007. Figure 5 shows that as the number of operational CSI seaports expanded from 2002 to 2007, the proportion of total U.S.-bound container cargo passing through CSI seaports also continued to increase.

In implementing the CSI program and reaching its goal of 58 operational CSI seaports, CBP selected foreign seaports to participate in the program in three phases. CBP officials reported using the following general selection criteria for each phase as follows: Most of the 23 phase I seaports were selected because they shipped the highest volume of U.S.-bound container cargo.

The 19 phase II seaports were selected based on factors such as cargo volume, strategic threat factors and the foreign government’s level of interest in CSI.

The 16 phase III seaports were selected using the phase II criteria as well as diplomatic or political considerations, such as the requests of foreign governments already participating in CSI.

As CBP expanded the number of CSI seaports and increased the proportion of total U.S.-bound container cargo passing through CSI seaports, the agency also achieved increases in security activities that occur at CSI seaports—targeting (CBP screens container cargo with ATS to produce risk scores and conducts additional review or research to ascertain risk levels) and examining high-risk container cargo (host government officials examine high-risk containers by scanning with nonintrusive inspection equipment or by physically searching the container). As of September 2007 CBP reported fully targeting 100 percent of all U.S.-bound container cargo to identify high-risk cargo as required by the SAFE Port Act. In addition, foreign governments participating in CSI have examined an increasing amount of high-risk container cargo as a growing proportion of total U.S.-bound containers pass through CSI seaports. In keeping with the CSI program’s risk-based approach, CBP currently does not request that the host governments examine all U.S.- bound containers passing through the CSI seaports, just those that CBP officers have determined to be high-risk. In fiscal year 2006, the number of high-risk containers examined by host government officials at CSI seaports increased by 77 percent from the previous year to almost 71,000 containers. In fiscal year 2007, examinations continued to increase, reaching almost 137,000 containers. Moreover, in fiscal year 2007 CBP reported that host government officials examined approximately 96 percent of the container cargo referred for examination. CBP reported that about 4 percent of the referrals did not lead to examinations (about 5,600 requests) because (1) logistical difficulties arose, such as the container had already been loaded on the shipping vessel (about 5,200 requests),or (2) the host government denied the request (fewer than 400 requests).

To Strengthen CSI Operations, CBP Has Taken Steps to Address Human Capital Challenges and Enhance Host Government Relations, but Operational Challenges Remain

CBP has made various operational improvements to CSI, though challenges remain. First, CBP has revised its human capital plan and added permanent staff at CSI seaports, though it reports difficulties in hiring and deploying qualified staff. Second, CBP’s relations with CSI host governments we spoke to that conduct cargo examinations have improved over time, though access to key examination-related information and processes is limited by host governments at some CSI seaports. And finally, CBP’s ability to conduct CSI program activities involves logistical challenges that are inherent to many seaport environments, such as those that are densely packed with equipment and personnel.

CBP Has Increased Permanent Staffing Levels at CSI Seaports, but Has Yet to Determine Optimum Distribution of Staff to Ensure All Critical Operations Are Performed

The ability of the CSI program to operate in accordance with its mission and objectives depends, in part, on the success of its human capital strategy—and CBP’s ability to manage and deploy staff in a way that ensures that critical security functions are performed. Our April 2005 report on CSI noted that although CBP’s goal is to target all U.S.-bound cargo shipped in containers at CSI seaports before they depart for the United States, the agency had not been able to place enough officers at some CSI seaports to do so. Specifically, CBP had developed a CSI staffing allocation model to determine the staff needed to target container cargo. However, at some CSI seaports CBP had been unable to staff the CSI teams at the levels called for in the CSI staffing model. We noted that CBP’s staffing model had not, at the time, considered whether some of the targeting functions could be performed in the United States. We recommended that CBP revise its staffing model to consider what functions need to be performed at CSI seaports and what functions can be performed in the United States, optimum levels of staff at CSI ports, and the cost of locating CBP targeters overseas at CSI seaports instead of the United States.

CBP has subsequently taken several steps to increase the number of CSI officers and to implement our 2005 recommendations. For example, in response to our concerns about staffing imbalances across seaports and shortages at the highest-volume seaports, CBP has increased staffing levels, bringing them closer to those called for in its staffing model— resulting in a parallel increase in the volume of container cargo that is targeted. Also, CBP has added 15 staff to CSI targeting duty at the NTCC since 2005, composed of temporary and permanent officers. In addition, in fiscal year 2007 CBP deployed an additional 125 permanent and 68 temporary officers to CSI seaports. Considering the officers at both CSI seaports and the NTCC, as of November 2007, CBP had deployed 209 CSI officers, which exceeds the 203 called for in the CSI staffing model. As a result of these efforts, CBP officials told us that they had increased their targeting of U.S.-bound container cargo from 65 percent in April 2005 to 100 percent in September 2007.

The agency also developed cost estimates for placing a mix of permanent and temporary staff at CSI seaports (with permanent staff costing about $330,000 per year and temporary staff about $275,000 per year) in response to our recommendation. CBP reported that the advantages of placing officers at CSI seaports on a permanent rather than a temporary basis include greater opportunities for enhanced communication and coordination with host governments, and less disruption due to fewer rotations into and out of the country. At one CSI port that we visited, host government customs officials told us that the presence of permanent staff facilitated increased information sharing, which over time could lead to a decrease in unnecessary examinations.

Despite the progress it has made, CBP continues to face staffing challenges. CBP officials told us, for example, they continue to face challenges in obtaining sufficient numbers of qualified officers to be permanently deployed at CSI seaports. For example, CBP officials reported that only 9 qualified applicants applied for 40 permanent positions at CSI seaports. Officials told us that CSI must compete for staff with targeting or seaport experience with other CBP programs or positions, such as C-TPAT or other programs that operate at the NTCC. To fill open positions at CSI seaports, CBP officials reported that in some instances officers have been deployed who have not received all of the required training. In addition, CBP evaluation data we reviewed showed examples of CBP officers at CSI seaports lacking key skills, such as the ability to target proficiently or communicate in the local language.

In addition, CBP has taken action to enhance its human capital planning process for CSI, but has not yet included important factors in its staffing allocation model. As we reported in 2005, one of the features of the CSI staffing model that may contribute to staffing imbalances was its reliance on placing officers overseas at CSI seaports. It did not consider what functions could be done in the United States. In May 2006, in response to our recommendations, CBP issued a human capital plan that did not specify that CSI targeting positions be located at CSI seaports, thus recognizing that officers could support CSI seaports from the NTCC in the United States. CBP officers assigned to the NTCC perform many of the same roles as officers at CSI seaports, including reviewing bills of lading. CBP officers at the NTCC review bills of lading for high-volume seaports where the placement of the number of CSI officers required to review all bills of lading is unfeasible. In addition, according to CBP officials, CBP officers at the NTCC review bills of lading for U.S.-bound cargo from CSI seaports where no CBP officers are stationed. Though CBP’s 2006 human capital plan generally recognizes that some CSI functions can be performed at either a CSI seaport or at the NTCC, the staffing allocation model used to calculate the number of targeters necessary to review bills of lading for each CSI port does not include factors that specify where these positions should be located.

In addition, CBP’s staffing allocation model does not take into account activities other than targeting—such as witnessing host government examinations—that CSI officers perform at CSI seaports. According to CBP, the agency stations as many of the total officers needed as possible at the CSI seaports, but if the number of officers needed is higher than the number of officers allowed by the host government or available to be stationed in the seaport, then the remainder of the officers target from the NTCC. However, we found that CBP has still not systematically determined the optimal number of officers that need to be physically located on-site at CSI seaports to carry out duties that require an overseas presence (such as coordinating with host government officials or witnessing the examinations they conduct) as opposed to other duties that could be performed off-site in the United States (such as reviewing bills of lading and databases). Also, CBP’s revised CSI human capital plan does not include costs related to placing temporary staff at the NTCC and thus does not have the data needed to conduct a cost-benefit analysis for determining the optimal location for its CSI officers. As we noted in our 2002 report on a staffing framework for use at U.S. embassies, federal agencies should consider factors such as cost and physical security of foreign operations and consider options such as relocating staff to the United States, as part of their framework for determining the right number of staff to be placed overseas. Determining optimal staffing levels is particularly important in light of ongoing challenges CBP reports facing to identify sufficient numbers of qualified individuals to hire for the program, and in light of the program’s recent expansion to additional seaports around the world.

While CBP has taken steps to implement the recommendations from our April 2005 report, further action is needed regarding the staffing allocation model. Specifically, as we recommended in 2005, the model should be revised to consider (1) what functions need to be performed at CSI seaports and what functions can be performed in the United States, (2) the optimum levels of staff needed at CSI seaports to maximize the benefits of targeting and examination activities in conjunction with host nation customs officials, and (3) the cost of locating targeting positions overseas at CSI seaports instead of in the United States.

Level of Collaboration between U.S. and Host Customs Officials Has Improved, but Challenges Remain at Some CSI Seaports

CSI’s strategic plan emphasizes the importance of CBP’s continued efforts to foster partnerships with foreign customs officials at CSI seaports to improve CSI operations. Specifically, according to CBP headquarters officials, when CSI teams stationed at foreign seaports develop strong interpersonal relations with foreign government officials, it leads to increased trust and information sharing and thus improved targeting and examination of high-risk cargo. While the extent of cooperation across all of the 58 CSI seaports now operating is difficult to quantify, our observations at 6 CSI seaports and our review of select CSI team evaluations provide examples of how collaboration can benefit the CSI program, and conversely, how the lack thereof can hinder progress.

At all 6 CSI seaports we visited, CBP officers or host government officials told us that the relationship between the CSI team and the host government has been positive or has improved over time. CBP and host government officials we spoke with at all of the seaports we visited reported that establishing trust and collegiality has led to increased information sharing, resulting in more effective targeting and examination of high-risk container cargo. For example, CBP officers noted instances in which host customs officials would occasionally notify them of container cargo they thought could be high-risk, so that CBP could take a closer look at the information available in ATS related to the container cargo. In addition, a few CBP officers or host government officials stated that the presence of CSI teams at foreign seaports has in many instances helped to prevent unnecessary examinations because information provided by host government customs officials has led to lower risk profiles for certain container cargo.

Moreover, CBP officials reported that strengthened relationships with host government officials and the trade community have led host governments to bolster their customs and port security practices. CBP officials we spoke to emphasized that, like the United States, most foreign customs administrations have traditionally focused on revenue collection and the seizure of contraband, rather than security concerns. During our visits to CSI seaports, the CBP and host government officials we spoke with reported several examples of how the presence of CSI teams at seaports has helped to expand the focus of the efforts of these foreign customs administrations and the trade community to include enhanced security practices. For example, one country developed databases with trade information to achieve its customs goals and to assist CSI after seeing how gathering historical data benefited CBP. Furthermore, at a couple of the CSI seaports we visited, the CSI team or host government officials arranged outreach meetings with the trade community to raise companies’ awareness of security practices and the benefits of providing correct and complete data about their cargo.

During our visits to CSI seaports and our review of data CBP collected during its evaluations of CSI teams, we also identified instances where cooperation between CSI teams and their counterparts in the host government could be improved—though, as CBP officials noted, some of the factors involved are beyond CBP’s ability to control directly. For example, in some locations, CBP officials reported that a country may have laws that hinder the collaboration of host government officials with CSI teams. We identified the following issues during our observations at 6 CSI seaports as well as from our review of CBP data collected in fiscal year 2007 at an additional 12 CSI seaports (for a total of 18 CSI seaports): At 9 CSI seaports, the CSI teams there reported that they only interacted infrequently with their host government counterparts or the host government officials did not readily share information that would benefit CSI, such as knowledge about potentially suspicious container cargo. In one instance the lack of interaction was attributed to the host government’s competing priorities.

At 6 CSI seaports, host governments restricted CSI teams from viewing nonintrusive inspection equipment examinations conducted by host customs authorities or the resulting images of the container’s contents, which is one of the key purposes for staffing CBP officers at CSI seaports.

At 4 CSI seaports, host governments prohibited the use of hand-held radiation detection devices by CBP officials, which is considered by CBP to be an important way to identify a potential anomaly in a high- risk container. According to CBP officials, a few of the countries prohibit the equipment due to safety and health concerns about the use of the equipment.

At 3 of the CSI seaports, host customs officials lacked access to technical equipment, such as computers or nonintrusive inspection equipment that worked properly, which CBP believes could limit their ability to share customs-related information with CSI team members or efficiently conduct examinations. According to CBP officials, sometimes host governments lack resources to meet these technological needs.

At 6 CSI seaports in 2 countries, CBP officers at the seaport reported that host customs administrations did not provide a sufficient number of staff to assist CSI teams or the host government officials were often unavailable, which, according to CSI teams, can sometimes lead to delays in examining high-risk containers.

At 3 CSI seaports, there was evidence of challenges to effective communication, such as some CSI teams having limited proficiency in the local language.

These examples are not intended to represent the CSI program as a whole, but are included to illustrate the types of challenges that CSI teams at the seaports and CBP program managers face. CBP officials responsible for managing the CSI program have reported that overall there has been a high level of cooperation at CSI seaports, though they acknowledged that the degree of involvement and participation that CBP officers have with foreign customs officials during the examination of high-risk cargo varies by country. It is also important to note that while CBP negotiates a written, nonbinding arrangement stating expectations for inclusion in the CSI program with the participating foreign governments, the agency cannot compel foreign governments to offer information for the purposes of CSI or to examine high-risk containers. Later in this report, we describe the processes CBP has in place to address difficulties that may be identified at the CSI seaports as part of its program oversight and monitoring efforts.

Seaport Environment and Logistics Present Challenges to CSI Operations

Another factor that can affect CBP’s ability to conduct CSI program operations involves logistical challenges that are inherent to many seaport environments. For example, as illustrated in figure 6, foreign government officials we spoke with at CSI seaports reported that many seaports are densely packed with equipment and personnel, which can make it difficult for host government customs officials to examine container cargo.

According to CBP, open space to place scanning equipment or to conduct physical searches of containers can be scarce at some CSI seaports. For example, in two of the CSI locations we visited, scanning equipment and examination sites were placed several miles from where container cargo is unloaded, loaded, or stored. According to the CBP officials we spoke with, this adds to the costs and time required for examination and may result in logistical difficulties in having high-risk U.S.-bound containers examined before being loaded onto the shipping vessel. In addition, at one port we visited, the host government limited the number of containers it would examine, in part to limit the cost of examination and the amount of delay caused by moving these containers, according to the CSI team we spoke with. CBP officials reported that despite this limit to examine no more than 250 containers (out of the over 115,000 container cargo shipments to the United States from this seaport in fiscal year 2007), the country has not denied many examination requests—only two in fiscal year 2007. However, this ceiling was not based on risk factors, and an increase in denied requests could lead to additional containers with high-risk cargo departing for the United States without being examined.

Finally, CBP officials stated that containers at seaports are generally stored in a container yard before they are loaded onto the shipping vessel. These container yards may be very large, and containers in these yards are often stacked to minimize the time required to load container vessels. As shown in figure 7, containers on a vessel may be stacked several layers deep. Accordingly, CBP and host government officials we spoke to at a few CSI seaports reported it can sometimes be challenging to access a container for examination. CBP officials noted that any examinations requested but not conducted in the CSI seaport would occur at a U.S. seaport upon arrival.

CBP Has Enhanced Its CSI Evaluations at CSI Seaports and Performance Measures but Still Does Not Capture Critical Information about Host Government Examination Systems

CBP has enhanced how it collects CSI data by strengthening its approach to conducting periodic evaluations of CSI officers at CSI seaports through on-site evaluations of performance. However, weaknesses remain in how CBP conducts evaluations, the information collected regarding host government examination systems, and performance measurement of the program as a whole. For example, CBP does not systematically collect information on the equipment, people, and processes that are part of the host government’s overall examination system. Also, while CBP has refined and updated its performance measures, we identified remaining limitations, such as the omission of measures for all core program elements and several performance targets.

CBP Significantly Improved Its CSI Evaluations to Assess Program Operations at CSI Seaports, but Weaknesses Remain

CBP conducts evaluations at CSI seaports to determine the effectiveness of the program. Specifically, CBP uses these on-site evaluations to assess CSI team operations and capabilities, such as how well CSI team members use ATS to determine the risk levels associated with U.S.-bound containers passing through CSI seaports. CBP’s CSI strategic plan states that these periodic reviews are intended both to ensure that deployed CSI teams are adhering to standard operating procedures as well as to evaluate the relationships between the teams and the host customs administrations. In fiscal years 2006 and 2007, CBP reported conducting 42 and 45 evaluations, respectively. Since the program’s inception in 2002, the agency reported conducting a total of 202 evaluations.

In November 2006, CBP significantly changed the way it conducts CSI team evaluations. Prior to that time, CBP officials reported that its evaluators relied on self-reported information from CSI team members on how proficiently they performed CSI program activities. CBP’s current approach to conducting CSI team evaluations seeks to provide a more thorough review of CSI team performance. According to CBP officials, the agency now requires the CSI team members under review to demonstrate their targeting competence to an evaluator, such as by physically showing the evaluator how they review information about container cargo to determine its risk level. To better assess the deployed CSI team’s performance, CBP augmented its evaluation teams with officers who have expertise in areas such as targeting and intelligence gathering.

Also, CBP has developed a new software tool that enables evaluators to record evaluation data electronically, using laptop computers to conduct the on-site evaluations. This tool, CSI Team Evaluation (CSITE), consists of a series of yes or no questions that cover the various areas of CSI team performance, including whether all of the container cargo that the CSI team designated high-risk were examined and whether these actions were properly documented. The CSITE tool also provides guidance on each question and prompts evaluators as they conduct their review by, for example, directing them to ensure that the CSI team is using the correct settings in ATS. In addition, employing CSITE, CBP reported it can now aggregate the results of some or all of its evaluations, a capability it previously lacked, and can conduct statistical analyses of the results of the evaluations. The agency can determine, for example, what percentage of CSI team members successfully demonstrated proficiency in targeting high-risk containers. According to CBP officials, CSITE will eventually allow the agency to make comparisons of CSI performance across seaports. Moreover, CBP now retains the information it collects at CSI seaports and the resultant evaluation reports in a more systematic fashion. CBP officials acknowledged that the agency did not always store this data effectively prior to the implementation of the new evaluation system and could not provide us with documentation of all of the evaluations it had conducted since the program’s inception.

While these efforts should help to strengthen the CSI team evaluation process, CBP is still not consistently collecting all available data to aid in its analysis of CSI team performance, and we identified instances in which the agency did not reconcile contradictory information it had collected. Based on our review of CBP’s documentation associated with 34 evaluations to assess the information the agency collected and its methods for doing so, we found that evaluators do not always answer all of the questions contained in CSITE. For example, the software tool instructs the CBP evaluation team to collect information on whether recommendations made in prior evaluations have been implemented. This information could allow CBP to determine whether past problems have been addressed, but it is not always provided by the evaluation team. We also identified discrepancies between (a) the CSITE checklist of questions that the evaluation team completes during the onsite evaluation, and (b) the resulting evaluation report produced by CBP headquarters officials for 2 of the 14 locations for which we had both documents to compare. At one seaport, for example, the CBP evaluation team indicated in the CSITE checklist that the CSI team did not have all of the data systems it needed to effectively target outbound shipments, whereas the evaluation report stated the team had access to all of the appropriate targeting tools and databases. With more complete information, collected in a consistent manner, CBP may be better able to determine how well CSI teams are performing, what corrective actions may be needed to improve the program, or whether the CSI program is achieving its security goals.

CBP Lacks a Process for Systematically Gathering Information on Host Government Examination Systems, Which Include Equipment, People, and Processes

Host Government Examination Systems— Equipment

In April 2005, we recommended that CBP establish minimum technical criteria required for the capabilities of nonintrusive inspection equipment at CSI seaports, while considering sovereignty issues with participating countries. CBP agreed to evaluate the feasibility of establishing such criteria. In 2006, section 205(e) of the SAFE Port Act required DHS to establish minimum technical capability criteria for the use of nonintrusive inspection equipment and nuclear and radiological detection systems in conjunction with CSI, but noted that these criteria should not be designed to conflict with the sovereignty of host countries. In 2007, the 9/11 Act also required the Secretary of DHS to develop technological standards for scanning systems that will be used to conduct 100 percent scanning at foreign seaports in the future and to ensure that these and other actions implementing the act’s 100 percent scanning provisions do not violate international trade obligations and are consistent with the World Customs Organization framework or other international obligations of the United States. CSI host governments, which are responsible for conducting examinations of container cargo, purchase and operate nonintrusive inspection equipment, though as of November 2007, 13 CSI seaports use equipment on loan from the United States. The capabilities of this inspection equipment vary by manufacturer and model. The equipment may differ, for example, in its ability to penetrate steel shielding in order to generate an image of container contents, or may scan containers at different rates. Appendix IV describes the capabilities of this equipment in greater detail. As of November 2007, CBP had not yet implemented our prior recommendation or taken actions to meet the SAFE Port and 9/11 Acts requirements for setting minimum technical criteria. CBP officials stated that the reason for this is that they do not consider the agency to be a standard-setting organization. While CBP refers host governments to the World Customs Organization’ SAFE Framework regarding the procurement of inspection equipment, this document does not include specific technical criteria or standards. Moreover, they added that it is important to acknowledge the inherent challenges involved in efforts to ascertain the capabilities of nonintrusive inspection equipment that is owned and operated by CSI host governments.

In May 2005, however, CBP put forth minimum technical criteria to evaluate the quality and performance of nonintrusive imaging inspection equipment being considered for use at U.S. seaports. These domestic standards set baseline performance requirements for penetration, contrast sensitivity, throughput, image quality, and scan size. To determine whether certain types of nonintrusive inspection equipment were acceptable for use at domestic seaports—and could meet the criteria that had been set— CBP conducted tests comparing the capabilities of nonintrusive imaging inspection equipment provided by seven manufacturers with its technical operating standards. On the basis of the test results, CBP recommended the inspection equipment from five of the seven manufacturers for use at domestic seaports, while equipment from two manufacturers was not recommended. CBP officials stated that there are no plans to systematically compare the capabilities of inspection equipment at CSI seaports against these criteria for domestic equipment due to sovereignty concerns.

CBP collects limited information on certain characteristics of the inspection equipment installed at CSI seaports, such as manufacturer; however, information related to capabilities and performance is not generally obtained. Officials in CBP’s Office of Technology stated that they have information on the capabilities of equipment that the United States loans to other countries for 16 CSI seaports, and that only this equipment can be assured of meeting the CBP domestic requirements. However, these CBP officials said that they had neither determined which other CSI seaports use the inspection equipment that was assessed as part of CBP’s test and recommended for use at domestic seaports, nor systematically determined the specific capabilities of the equipment used at those CSI seaports. Host government officials in the countries we visited stated that they followed their country’s acquisition procedures, which included reviewing equipment capabilities and performance, among other things, for the purchase of nonintrusive imaging inspection equipment. However, CBP does not have documentation on the testing used by the host countries or the manufacturers to determine the basis for the equipment’s stated performance or whether this stated performance is less than, meets, or exceeds the criteria CBP established for equipment used at domestic seaports.

According to CBP officials, the capabilities of nonintrusive inspection equipment are vetted during an assessment phase of the CSI program, when CBP is determining whether a seaport is prepared to operate within CSI. While, as part of the assessment phase, CSI officials stated that they collect descriptive technical information about the type of nonintrusive inspection equipment to be used at seaports, we did not find—in our review of CBP’s checklist used to guide its assessment teams as they examine prospective CSI seaports—questions covering inspection equipment other than general direction to ascertain whether some type of this equipment was in place. Also, through our review of CBP’s assessments of 10 CSI seaports—through which approximately 55 percent of all U.S.-bound containers passed in fiscal year 2007—we did not find any assessments that described the performance capabilities of the equipment or judgments about the proficiency of host government officials in operating these systems. CBP officials stated that the agency has never prohibited a seaport from participating in CSI on the basis of its inspection equipment, and CBP documents show that participation in the program requires only that some type of nonintrusive inspection equipment be available at or near the potential CSI port.

Host Government Examination Systems—People and Processes

The SAFE Port Act also directed DHS to (1) establish standard operating procedures for the use of nonintrusive inspection equipment at CSI seaports and (2) require CSI seaports to operate the equipment in accordance with the criteria and operating procedures established by DHS. Also, the 9/11 Act required DHS to develop operational standards for scanning systems that will be used to conduct 100 percent scanning at foreign seaports in the future. CBP officials stated that they recognize that the capabilities of nonintrusive inspection equipment are only one element for determining the effectiveness of examinations that take place at CSI seaports. It is better, in their view, to make assessments of the whole examination system, which includes nonintrusive inspection equipment, personnel, and processes. However, CBP acknowledged it does not systematically collect information on host governments’ use of examination systems and has not developed general guidelines or criteria that could provide CBP with the means to determine the quality of examinations of high-risk container cargo bound for the United States. CBP officials stated that they rely on CSI teams to notify headquarters if they have concerns about the host government customs or examination practices. Specifically, each CSI team leader is to meet weekly—usually via teleconference—with a CSI manager located at CBP headquarters to discuss ongoing CSI operations. However, CBP officials acknowledged that equipment, capabilities, and examinations practices of host government customs personnel are not routinely discussed.

CBP officials also reported that CSI team members witness most examinations of high-risk U.S.-bound containers, and their presence at the examinations would allow them to make judgments about aspects of the host government’s examination system. However some host governments specifically prohibit CSI team members from witnessing examinations. Also we found that CBP officials did not routinely observe inspections at one CSI seaport we visited, and were not always able to be present for inspections at two other CSI seaports because those inspections were scheduled and conducted when CBP officials were not available.

CBP officials told us that their CSI team evaluations are also a means of capturing some information on various aspects of the host government’s examination system. In order to participate in CSI, CBP requires that, among other things, host governments have customs staff capable of examining cargo originating in or transiting through its country and maintain a program to prevent breaches in employee integrity. However, the 15 CSI team evaluations we reviewed, which CBP had conducted since the agency revised its evaluation process in November 2006, showed limited coverage of whether host government customs personnel have been trained to use nonintrusive inspection equipment or are using it properly, the sufficiency of host staffing levels, and host government efforts to ensure the integrity of their customs administration. Specifically, 6 of the 15 CSI team evaluations discussed whether equipment was used properly, 1 discussed host staffing levels, and none discussed host integrity programs.

CBP’s Lack of Information on Host Government Examination Systems Potentially Limits Assurance That Examinations of High- Risk Container Cargo are Effective

CBP’s lack of a systematic way to collect information on host governments’ examination systems—including their equipment, people, and processes—potentially limits CBP’s ability to ensure that examinations of high-risk container cargo at CSI seaports can detect and identify WMD. Without information on host governments’ examination systems, CBP management may not be able to determine the reliability of the host government’s inspections of high-risk U.S.-bound container cargo. This is of particular concern since, according to CBP officials, most high- risk cargo that has already been examined at a CSI seaport, is generally not reexamined once it arrives at a U.S. seaport. CBP officials stated that if problems are found in the examination process at a CSI seaport, then high-risk container cargo would be reexamined upon arrival in the United States.

As already noted, CBP must respect participating countries’ sovereignty. CBP cannot require that a country use specific equipment. However, if a high-risk container was examined using an examination system found by CBP to be less capable than established criteria, the agency could require that the container be reexamined upon arrival at a U.S. seaport. CBP officials stated that they believe that in general the equipment used by participating governments meets or exceeds the capabilities of the nonintrusive inspection equipment used at U.S. seaports. However, because CBP has not set minimum technical criteria for nonintrusive inspection equipment at CSI seaports, and the agency does not systematically review the operations of the host government examination systems at CSI seaports, CBP potentially has limited assurance that their inspection equipment is capable of detecting and identifying potential WMDs. In light of the new 9/11 Act requirement that 100 percent of U.S.- bound container cargo be scanned in the future with nonintrusive inspection equipment at foreign seaports before leaving for the United States, it is important that CBP have processes in place to gather the information necessary to ensure that cargo container examinations—and the equipment used as part of the examination process—are reliable, regardless of the point of origin.

CBP Made Efforts to Refine CSI Performance Measures, but Did Not Fully Address our Previous Recommendation

While CBP has taken steps to strengthen performance measures for the CSI program, we identified areas that did not fully address our April 2005 recommendation to develop outcome-based performance measures or proxy measures of program functions—if program outcomes could not be captured—and performance targets to track the program’s progress in meeting its objectives. Whereas CBP’s CSI team evaluations and program monitoring activities help to evaluate CSI operations at the seaport level, CBP uses performance measures to gauge the effectiveness of the overall program in meeting its broader strategic objectives for CSI across seaports. By definition, performance measures are a particular value or characteristic used to quantify a program’s outputs—which describe the products and services delivered over a period of time—or outcomes— which describe the intended result of carrying out the program. A performance target is a quantifiable characteristic that establishes a goal for each measure; agencies can determine the program’s progress, in part, by comparing the program’s measures against the targets. For example, the target of one of CBP’s performance measures—the “number of operational CSI seaports”—was to have 58 CSI seaports operating in fiscal year 2007, which the agency achieved as described previously in this report. The Government Performance and Results Act of 1993 incorporated performance measurement as one of its most important features, and the establishment and review of performance measures are a key element of the standards for internal control within the federal government. As discussed in the Government Performance and Results Act of 1993 and as we reported in 1996, measuring performance allows organizations to track progress being made toward specific goals and provides managers crucial information upon which to base their organizational and management decisions. In addition, leading organizations recognize that performance measures can create powerful incentives to influence organizational and individual behavior.

In the past 2 years, CBP has made efforts to refine and modify its performance measures as the CSI program has matured. Since 2005, for example, CBP has eliminated five performance measures that it had used to track the implementation of seaports participating in CSI, measures that CBP determined were no longer needed because CSI operations were under way at the majority of planned CSI seaports. Also, in our April 2005 review of CSI, we identified a CSI performance measure that was calculated inappropriately, and in response, CBP modified how the measure was calculated to address our concerns. Specifically, for the CSI measure that tracks the number of container examinations waived because they are determined to be unnecessary, CBP began excluding inappropriate data that made the results of the performance measure misleading. This was an important modification because, as we reported in November 2002, measures that are defined inconsistently with how they are calculated can be confusing and create the impression that performance is better or worse than it actually is.

CBP has made efforts to enhance CSI performance measures, but we identified limitations in the information available for CSI program managers to assess the program. In the past, we and the Office of Management and Budget have encouraged federal departments and agencies to measure whether programs are achieving their intended outcomes, such as CSI’s purpose of protecting global trade from being exploited by international terrorists. However, we and the Office of Management and Budget have acknowledged the difficulty in developing outcome measures for programs that aim to deter or prevent specific behaviors. In such an instance, we have reported that proxy measures should be designed to assess the effectiveness of program functions. CBP officials reported the agency has not been able to develop a way to measure the deterrence effect of the program, as CSI is designed to support the CBP mission to prevent and deter terrorists and terrorist weapons from entering the United States. Examples of CSI program functions include targeting and examining high-risk container shipments before they are loaded on vessels bound for the United States, and in our 2005 review of CSI we provided guidance on an alternative method of developing proxy measures to evaluate program performance. Further, according to the Office of Management and Budget, proxy measures should be closely tied to the intended program outcome, and it may be necessary to have a number of proxy measures to help ensure sufficient safeguards are in place to account for performance results. According to CBP officials the following three of its existing performance measures were proxies for program outcomes.(1) The percentage of worldwide U.S.-bound containers passing through CSI seaports—since these containers are to be targeted and, if determined high-risk, may be examined by host government officials, this is a measure of the program goal to detect and prevent WMDs headed to U.S. seaports from leaving foreign seaports. (2) The number of foreign mitigated examinations (that is, examinations determined to be unnecessary due to information provided by host government officials and thus waived) by category—developed to quantify whether collocating CBP officials at CSI seaports increases information sharing and collaboration. (3) The number of intelligence reports based on CSI foreign sources— intended to measure whether having CBP officials located at foreign seaports leads to increased collaboration with foreign customs officials.

The Office of Management and Budget has stated that performance measures should capture the most important aspects of a program’s mission and priorities. However, CBP does not have a measure that tracks the extent to which U.S.-bound containers carrying high-risk cargo are examined at CSI seaports, despite the fact that this activity is a core element of the CSI program.

CBP has taken other actions to address our April 2005 recommendation that includes ways to improve CSI performance measures, but we found additional weaknesses as well. The strategic plan demonstrated how each performance measure corresponds to the three strategic goals of CSI, which include (1) securing U.S. borders, (2) building a robust CSI cargo security system, and (3) protecting and facilitating trade. This marked an improvement, as this linkage had not been made previously. In addition, CBP addressed an additional aspect of our prior recommendation by establishing performance targets for four of the six CSI performance measures currently used. However, only one measure had a target for multiple years. In addition, since issuing the CSI strategic plan the agency has not updated its performance targets for fiscal year 2008 or beyond for any of its measures. Without this information about the performance targets, it may be difficult for CBP to determine whether the results were more positive or negative than expected.

Also, we identified a weakness in how some CSI performance measures are calculated. As we noted earlier in this report, as the number of CSI seaports has increased in recent years, program activities have increased as well. However, CBP does not appropriately control for this program growth in how it calculates three of its six performance measures. For example, since the “number of foreign mitigated examinations by category”—the number of container examinations determined to be unnecessary due to information provided by host government officials—is not calculated on a per-container basis (i.e., per 10,000 containers), it may be difficult to determine whether fluctuation in the numbers across years is due to (1) increased collaboration with foreign government officials or (2) simply an increase in the number of containers reviewed and considered for examination at the increasing number of CSI seaports. Similarly, the number of intelligence reports and the number of investigative cases initiated may be due to an increase in the number of operational CSI seaports, not increased collaboration with host government officials. Without controlling for program growth, CBP’s calculation of results for its performance measures may be misleading or confusing to CBP and DHS program managers or the Congress, who provide program oversight.

Conclusions

Since we began reporting on the CSI program in 2003, CBP has made significant progress in expanding and developing the program. However, CBP continues to face several management and operational challenges, which may limit CBP’s ability to ensure that the CSI program provides the intended level of security for U.S.-bound container cargo moving through the international supply chain. Also, balancing security concerns with the need to facilitate the free flow of commerce remains an ongoing challenge for CBP.

Recognizing that program evaluation data are important for program managers to understand why results occur and what value a program adds, CBP has taken actions to enhance its evaluation of CSI team activities. The revised evaluation program has increased the information available to make policy and programmatic decisions regarding the operations at the CSI seaports. However, limitations that remain in CBP’s evaluation process affect the accuracy and completeness of the program information available for making sound management decisions about the CSI program as a whole. Specifically, when CBP’s evaluation teams do not complete the evaluation tools or resolve contradictory information, program managers may receive limited or inaccurate information. Further, when the data collected using the CSI evaluation tool during the evaluations are not reliable and readily available for assessment, CBP’s planned programwide trend analyses of the CSI program may be misleading.

In assessing CSI performance, CBP lacks information about a very important aspect of the program—the overall examination systems used by the host governments to examine high-risk cargo shipped in containers as requested by CBP. CBP’s efforts have led to the successful participation of a wide array of foreign governments in the CSI program, and CBP has established many cooperative relationships with its foreign partners. While we acknowledge the agency cannot force security requirements upon foreign governments, the lack of information systematically gathered about the examination systems used by participating governments is problematic. Data about the equipment, people and processes involved in the examination system are vital for determining whether high-risk U.S.- bound containers have been properly examined or should be examined or reexamined upon arrival at a U.S. seaport. CBP lacks guidelines and criteria for most of the equipment and the people and processes used by host government examination systems—as required, in some instances, by the SAFE Port and 9/11 Acts—for evaluating CSI seaport operations and determining overall program effectiveness. In light of the new 9/11 Act requirement that 100 percent of U.S.-bound container cargo be scanned in the future at foreign seaports before leaving for the United States, it is important that that CBP have programs in place to gather the information necessary to ensure that cargo container examinations—and the equipment used as part of the examination process—are reliable, regardless of the point of origin.

Program evaluations are just one source of information that managers need to make decisions, and evaluation data must often be coupled with performance measurement to assess overall program results. Measuring the overall impact of the CSI program remains difficult due to the challenges involved in creating effective performance measures, and because of great difficulty in measuring the deterrent effect of the program. As we and the Office of Management and Budget have reported, performance measurement can be very valuable to program managers, as the process can indicate what a program is accomplishing and whether intended results are being achieved. Measuring program performance encourages managers to focus on the key goals of a program and helps them by providing information on how resources and efforts are best allocated to ensure effectiveness. Though CBP identified performance measures it considers proxies for program outcomes (given the difficulty in assessing the deterrent effect of CSI), these measures do not cover a key core program function, for example a performance measure for the number of high-risk U.S.-bound containers examined at CSI seaports. Finally, without clearly developed performance targets for each of its measures, program managers, Congress, and the public lack information needed to determine the extent to which the CSI program is performing as intended. Taken as a whole, the lack of clearly articulated performance measures and accurate and reliable evaluative data may hinder CBP’s ability to ensure that the resources it expends for CSI effectively achieve its goal of helping to secure U.S. borders against terrorists and terrorist weapons.

Recommendations for Executive Action

To help ensure that CBP has the information needed to assess its achievement of CSI program goals to help enhance supply chain security—while at the same time balancing security concerns with the need to facilitate the free flow of commerce—we recommend that the Secretary of Homeland Security direct the Commissioner of U. S. Customs and Border Protection to take the following actions in three areas: Strengthen CBP’s process for evaluating CSI teams at overseas ports by (a) systematically capturing and maintaining all relevant evaluation data and documentation so that it can be used by CBP management to guide operating decisions, monitor program performance, and inform resource allocation decisions; (b) ensuring that CSI evaluation teams follow established evaluation procedures; and (c) monitoring the completion, within established time frames, of recommendations made in previous evaluations.

In collaboration with host government officials, improve the information gathered about the host governments’ examination systems—which includes people, processes, and equipment—at each CSI port by (a) establishing general guidelines and technical criteria regarding the minimal capability and operating procedures for an examination system that can provide CBP with a basis for determining the reliability of examinations and related CSI activities; (b) systematically collecting data for that purpose; and (c) analyzing the data against the guidelines and technical criteria to determine what, if any, mitigating actions or incentives CBP should take to help ensure the desired level of security.

Enhance CSI performance measures to better assess CSI performance overall by (a) developing measures for all core CSI program functions designed to have a deterrent effect, (b) establishing annual performance targets—based on explicit assumptions—for all performance measures, and (c) revising how performance measures are calculated to take into account CSI program growth.

Agency Comments and Our Evaluation

We provided a draft of this report to the Department of State and the Department of Homeland Security for their review and comment. The Department of State did not provide written comments but provided technical comments, which have been incorporated into the report as appropriate. DHS provided written comments—incorporating comments from CBP—on December 20, 2007, which are presented in Appendix II. In commenting on a draft of this report, DHS noted that it concurred with one recommendation and partially concurred with the remaining two recommendations.

In its written comments, DHS and CBP concurred with our recommendation on strengthening its process for evaluating CSI teams at overseas locations. Specifically, CBP noted that by June 2008, it planned to establish a database that would contain all recommendations and action plans as a result of CSI port evaluations as well as due dates for implementing recommendations and actions taken. To ensure that CSI evaluation teams follow procedures, CBP indicated that it would make it mandatory that the teams complete all database fields. Furthermore, CBP reported that it would assign values to questions in its evaluation tool on the basis of the criticality of the activity evaluated in each question to CSI’s mission as a whole.

DHS commented that CBP partially concurred with our second recommendation to improve information gathered about host governments’ examination systems by (a) establishing general guidelines and technical criteria regarding the minimal capability and operating procedures; (b) systematically collecting data for that purpose; and (c) analyzing the data against the guidelines and technical criteria. CBP agreed on the importance of an accepted examination process and noted it continues to take steps in addressing improvements in the information gathered about host government’s examination systems at CSI seaports by working directly with host government counterparts, through the World Customs Organization, and providing capacity building training and technical assistance. While CBP does engage in capacity building, it does so with only 5 of the 33 countries with CSI ports. CBP also stated that it will continue to use the WCO through its SAFE Framework of Standards to address a uniform customs process and technical standards for equipment. However, the SAFE Framework mentions no specific technical capability criteria for inspection equipment. Additionally, CBP does not systematically collect or assess information on the people, processes, or technology used by these host governments to examine high- risk U.S.-bound containers. CBP also noted in its comments to this report, that equipment used for inspection of containers in foreign countries is equal to or better than the equipment used by CBP at its domestic ports. While CBP has performance information for the 16 seaports that have inspection equipment on loan from CBP, it is not in a position to assess the performance of equipment used at the remaining 42 CSI seaports. Although we repeatedly requested systematic information regarding the equipment technical capabilities in these other ports, CBP officials were unable to provide it to us. In response to our 2005 report, CBP stated that it would evaluate the feasibility of technical requirements for nonintrusive inspection equipment, but a legal issue may exist regarding CBP’s ability to impose such requirements. While we understand CBP’s position, it could still gather information on such equipment’s technical capabilities. Because the CSI inspection might be the only inspection of a container before it enters the United States, it is important that information on the people, processes, and equipment used as part of CSI be obtained and assessed to provide some level of assurance of the likelihood that the examination system could detect the presence of WMD. If a port’s examination system were determined to be insufficient, CBP could take mitigating actions, such as re-examining container cargo upon its arrival at a domestic seaport.

Finally, DHS commented that CBP partially concurred with our third recommendation to enhance CSI performance measures to better assess CSI performance overall. CBP stated that it believes its current measures address core program functions of targeting and collaboration with host governments to mitigate or substantiate the risk of a maritime container destined for the United States. We disagree. As discussed earlier in this report, a core element of the CSI program, specifically the extent to which U.S.-bound containers carrying high-risk cargo are examined at CSI seaports, is not addressed through CBP’s performance measures. In its comments, CBP stated that its outcome performance indicator captures the number of foreign mitigated examinations by category, however CBP did not respond to our requests for more information regarding these categories, including whether risk was a category. Although it considers action on this recommendation completed, CBP noted its intention to continue to refine, evaluate, and implement measures to track progress toward meeting CSI objectives. As previously stated, since issuing the CSI strategic plan, CBP has not updated its performance targets for fiscal year 2008 or beyond for any of its measures. Thus, we believe additional action is warranted. Establishing annual targets for performance measures is important, as agencies can determine the program’s progress, in part, by comparing the performance measures against the targets. In addition, CBP did not address whether it plans to reconsider how it calculates some of its performance measures to control for CSI program growth. Without doing so, CBP’s calculation of results for its performance measures may be misleading or confusing to CBP and DHS program managers, or the Congress, who provide program oversight.

DHS and CBP also provided technical comments, which have been incorporated into the report as appropriate.

If you or your staff have any questions about this report, please contact me at (202) 512-9610 or at caldwells@gao.gov. Key contributors to this report are listed in appendix VI. This report will also be available at no charge on the GAO Web site at http://www.gao.gov.

Appendix I: Objectives, Scope, and Methodology

Objectives

We addressed the following issues regarding the U.S. Customs and Border Protection’s (CBP) Container Security Initiative (CSI): How has CBP contributed to strategic planning for supply chain security efforts and the CSI program in particular, and what progress has been made in achieving CSI performance goals?

How has CBP strengthened CSI operations in response to our 2005 review, and what challenges, if any, remain?

How does CBP evaluate CSI port operations and assess program performance overall, and how has this process changed over time?

Scope and Methodology

To address our first objective, we reviewed the strategic plans of the Department of Homeland Security (DHS), CBP, and CSI as well as national strategies like the National Maritime Security Strategy and the Strategy to Enhance International Supply Chain Security. We also analyzed the CSI strategic plan to determine whether it includes all of the key elements included in the Government Performance and Results Act. In addition, to measure CSI’s progress in meeting its performance goals, we reviewed and analyzed CBP data related to the number of CSI seaports, the cargo CBP targeted and referred to the host government to examine, and the number of cargo containers that were (and were not) examined by host government officials at the CSI seaports. We also met with CBP officials responsible for managing the CSI program, from the CSI Strategic Planning and Evaluation Branch, and from CBP’s Office of Field Operations and Office of International Affairs and Trade Relations, not only to gather information about CSI strategic planning and performance goals, but to discuss all of the issues within the scope of this review.

To examine CBP’s efforts to enhance CSI operations and the operational challenges that remain at CSI seaports, we reviewed GAO’s previous assessments of the CSI program and examined CBP’s efforts to implement our three prior recommendations. We also reviewed the CSI human capital plan and spoke to CBP officials about actions the agency has taken to ensure that CSI human resources are appropriately allocated. As part of that process, we met with officials at CBP headquarters and at the National Targeting Center - Cargo (NTCC) in Virginia to discuss the agency’s decision to conduct some targeting of high-risk containers from the NTCC rather than at CSI seaports. In addition, we spoke to CBP officials at three domestic seaports, selected according to geographical location and container volume. We also visited six CSI seaports located overseas, and selected the locations based on geographic and strategic significance, container volume to the United States from the seaports, when the seaports began conducting CSI operations, and whether the seaport was involved in CBP’s Secure Freight Initiative. At the CSI seaports, we also interviewed host government officials and CSI teams to discuss the frequency and level of collaboration involved in their interactions with each other, circumstances at seaport facilities that affect CSI operations, and financial cost issues associated with examinations. The results from our visits to seaports provided examples of CBP and host government operations but cannot be generalized beyond the seaports visited because we did not use statistical sampling techniques in selecting the seaports.

To determine what progress CBP has made in strengthening its tools for monitoring and measuring the progress of the CSI program, we reviewed the performance measures presented in the CSI strategic plan against criteria developed by the Office of Management and Budget and GAO. In addition, to appraise CBP’s efforts to strengthen its methods to evaluate CSI teams and to learn about operations at CSI seaports, we analyzed a sample of evaluation documents. Our nonrepresentative sample consisted of evaluations for all 40 seaports for which we had documentation at the time of our review, including (1) the 15 evaluations conducted between November 2006 (when CBP revised its evaluation process and began using the Container Security Initiative Team Evaluation software tool) and May 2007 (when we conducted our analysis), (2) the 7 available evaluations that directly preceded them chronologically and were conducted using CBP’s previous evaluation methodology (for the purpose of comparison), and (3) the most recent evaluations conducted at each of the additional locations for which documentation had been provided by CBP. Thus, we reviewed a total of 34 evaluations (covering 40 CSI seaports) out of the 114 evaluations that GAO had obtained from CBP as of May 2007. For each of the evaluations reviewed, we assessed any available materials, which could include a narrative report and/or a checklist of yes or no responses. While our sample covered various aspects of CBP’s evaluations, our sample was not selected using statistical sampling techniques. Thus, the results from our review of CBP evaluation data provide illustrative examples about CSI team evaluation methods and program operations at CSI seaports—and generally corroborated our seaport site visit observations—but cannot be generalized to the all 58 seaports conducting CSI operations.

We also met with CBP officials managing the CSI program to assess the agency’s efforts to collect information about the equipment, people, and processes involved in the host governments’ examinations of U.S.-bound container cargo, including the capabilities of examination equipment operating at CSI seaports and the proficiency of host customs administrations using the equipment. In addition, we selected and analyzed a nonrepresentative sample of 10 port assessments among those that CBP conducted at each port prior to its admission into the CSI program—the sample was composed of the 6 seaports we visited plus the 4 highest-volume locations as of January 2007. As of that date, approximately 55 percent of containers bound for the United States passed through these 10 seaports. Thus, our findings from our review of the assessments provide examples about the type of information collected as part of the process, but cannot be generalized to all 58 seaports in the program.

We conducted this performance audit from May 2006 through January 2008 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Data Reliability

We met with CBP officials to discuss the agency’s efforts to ensure CSI data on the number of cargo shipments and containers subject to targeting and examination are reliable. In our 2005 review of the program, we found the data to be sufficiently reliable to support our findings. Since that time, CBP has further enhanced the way in which it collects and aggregates information about CSI program activities at foreign seaports, including the targeting and examination of high-risk container cargo. Specifically, CSI teams now utilize improved technology, eliminating the need for transmitting data to CBP headquarters via e-mail and thereby reducing the opportunity for human error in manually entering and aggregating data for the program. CBP officials at headquarters can now directly access the data entered at each CSI port as soon as they are entered into the shared system and can monitor the data on a daily basis to identify errors in or mischaracterization of the data. While we did not directly test the reliability of 2006 data, the recent CBP initiatives to improve reliability, combined with GAO's previous assessment of the 2005 data, gave us confidence in using CSI targeting and examination data to provide descriptive, background information regarding the extent to which high- risk container cargo is targeted by CBP and examined by foreign governments participating in CSI.

Appendix II: Comments from the Department of Homeland Security

Appendix III: Container Security Initiative Seaports

This appendix provides information on the 58 foreign seaports participating in CBP’s Container Security Initiative (CSI). According to CBP, CSI was operating in 58 foreign seaports by the end of September 2007. Table 2 lists the CSI seaports according to the date when the seaports began conducting CSI operations, shows the phase (I, II, or III) in which specific seaports were selected for participation in CSI, the volume of U.S.-bound shipments passing through the seaport in fiscal year 2007, and specifies which seaports are participating in the Department of Energy’s (DOE) Megaports Initiative and in CBP’s Secure Freight Initiative.

Appendix IV: CSI Activities and Equipment

Targeting and Examining High-risk Containers

This appendix provides a detailed description of activities and equipment used at CSI seaports to target and examine container cargo. CBP targets all of the U.S.-bound containers that pass through CSI seaports to identify and, where feasible, examine high-risk container cargo. The container targeting and examination activities conducted at the foreign seaports for U.S.-bound cargo (exports) are very similar to activities CBP conducts at domestic seaports for arriving containers (imports). Figure 8 illustrates the various steps and decision points involved in targeting and examining high-risk U.S.-bound containers at CSI seaports.

Under CSI, the targeting of cargo can include the targeters’ review of the Automated Targeting System (ATS) score and the information on which it is based, the bills of lading—which include data about the cargo—and additional information provided by host government officials. CBP targeters at CSI seaports are to access bills of lading through ATS, a system that automatically uses its hundreds of rules to check available data for every container arriving in the United States and assigns a risk score to each cargo shipment. Targeters review the bill of lading, making a cursory check for discrepancies and anomalies in the name and address of the importer, the commodity, the cargo description and other data elements. On the basis of the initial review of the bill of lading, CBP officials are to either (1) categorize the cargo as low risk, in which case, the container holding the cargo is loaded onto the departing vessel without being examined, or (2) conduct further research in order to properly characterize the risk level of the cargo. Further research entails targeters using automated resources, such as the Treasury Enforcement Communication System or AutoTrack, as well as nonautomated resources, such as information provided by host government officials, to obtain applicable information to determine the validity of the shipment. Further research may also be conducted by the team’s intelligence research specialist. After further research is completed, CBP officials are to characterize the cargo as either (1) low risk, in which case it is loaded onto the departing vessel without being examined, or (2) high-risk, in which case it is referred to host government officials for concurrence to examine. Since CBP officials do not have the legal authority to examine U.S.-bound containers in foreign seaports, the host government customs officials conduct the examinations.

Host government officials can respond to the referrals for examination in one of three ways—cargo is examined or the request is either waived or denied. After receiving a referral from CSI teams, host customs officials are to review the bill of lading and the reasons for the referrals to determine whether or not to examine the container cargo. Some host governments collect intelligence information on U.S.-bound cargo independent of CSI, which host officials also consider in decisions of whether to examine the referred cargo. If host government officials agree that the cargo is high-risk, they will proceed with an examination. According to CBP, in general, CSI team members are to observe the examinations and review and document the results. On the basis of the results of a nonintrusive examination, such as if an anomaly is apparent in the image of the container, the host government and CBP officials must decide whether the host government will conduct a physical examination of the a container.

Alternatively, the CSI team may waive an examination referral if (1) host government officials provide the CSI team with additional information that lowers the risk level of the cargo or (2) logistics prohibit an examination, such as if the cargo container were already loaded on the departing vessel. Finally, if the host government officials determine, on the basis of their review, that the cargo is not high-risk, they will deny examination of the cargo. For any high-risk cargo for which an examination is waived or denied, CSI teams are to place a domestic hold on the cargo, so that an examination will be conducted upon arriving in the United States. However, if CSI team members are adamant that a cargo container poses an imminent risk to the carrier or U.S. seaport of arrival but cannot otherwise convince the host officials to examine the container, CSI team members are to contact and coordinate with the NTCC to issue a do-not- load order for national security. According to CBP officials, this order advises the carrier that the specified container will not be permitted to be unloaded in the United States until a time when any associated imminent risk to the cargo container is neutralized. Once the risk is neutralized, the container is to be loaded back onto the carrier and placed on hold for a domestic examination. According to CBP officials, this type of do-not-load order for national security has been implemented six times since the inception of CSI.

Equipment Used to Conduct Examinations of Cargo Containers

There are generally two types of CSI cargo container examinations— scanning with nonintrusive inspection equipment and physical searches. For scanning cargo containers, there are two basic types of nonintrusive inspection equipment currently used at CSI seaports: (1) radiation detection equipment and (2) imaging inspection equipment, which may use X-rays or gamma rays. Radiation detection equipment, such as a radiation portal monitor (RPM) and radiation isotope identifier devices (RIID) detects the presence of radioactive material that may originate from a container. However, only the RIID can determine whether the type of radiation emitted by the material actually poses a threat or is a normal emission of radiation, such as that found in ceramic tile. We observed at a domestic and a foreign seaport that generally if radioactive emissions are detected from a cargo container, customs officials will use a RIID (shown in fig. 9), to determine whether the radiation being emitted poses a threat.

The second type of equipment, referred to as imaging equipment, uses X- ray or gamma ray technology to scan a container and create images of the container’s contents without opening the container. CBP officials, along with host government officials, may review the images produced with the X-ray or gamma ray equipment to detect anomalies that may indicate the presence of WMD. Figure 10 shows a sample image produced by this type of equipment.

The capabilities of nonintrusive imaging inspection equipment vary by manufacturer and model. In May 2005, CBP defined minimum performance capabilities to evaluate the quality and performance of the nonintrusive imaging inspection equipment being considered for use at domestic seaports. The domestic standards set baseline performance requirements for such things as the ability of nonintrusive inspection equipment to identify images through steel shielding (referred to as penetration) or the ability to scan an amount of containers in a given time (referred to as throughput).

Appendix V: CSI Performance Measures

This appendix provides information on the CSI performance measures used by CBP. Table 3 describes the performance measures CBP is currently using to report the overall performance of the CSI program, the linkage between performance measures and CSI goals, the performance targets established, and the recent results collected for each measure. In addition, since our 2005 report, CBP has identified performance measures one, two and three below as its proxy measures used in place of a measure for program outcomes, given the difficulty in measuring the deterrent effect of the program.

Appendix VI: GAO Contact and Staff Acknowledgments

Stephen L. Caldwell, Director, Homeland Security and Justice Issues, (202) 512-9610, caldwells@gao.gov.

Staff Acknowledgments

This report was prepared under the direction of Christine A. Fossett, Assistant Director, Homeland Security and Justice Issues. Key contributions to this report also included Amy Bernstein, Fredrick Berry, Yecenia Camarillo, Frances Cook, Christopher Conrad, Wendy Dye, Kathryn Godfrey, Valerie Kasindi, Stanley Kostyla, Matthew Lee, Frederick Lyles, Robert Rivas, and Leslie Sarapu.

Related GAO Products

Maritime Security: The SAFE Port Act: Status and Implementation One Year Later. GAO-08-126T. Washington, D.C.: October 30, 2007.

Maritime Security: One Year Later: A Progress Report on the SAFE Port Act. GAO-08-171T. Washington, D.C.: October 16, 2007.

Maritime Security: The SAFE Port Act and Efforts to Secure Our Nation’s Seaports. GAO-08-86T. Washington, D.C.: October 4, 2007.

Combating Nuclear Smuggling: Additional Actions Needed to Ensure Adequate Testing of Next Generation Radiation Detection Equipment. GAO-07-1247T. Washington, D.C.: September 18, 2007.

Maritime Security: Observations on Selected Aspects of the SAFE Port Act. GAO-07-754T. April 26, 2007.

Customs Revenue: Customs and Border Protection Needs to Improve Workforce Planning and Accountability. GAO-07-529. Washington, D.C.: April 12, 2007.

Cargo Container Inspections: Preliminary Observations on the Status of Efforts to Improve the Automated Targeting System. GAO-06-591T. Washington, D.C.: March 30, 2006.

Combating Nuclear Smuggling: Efforts to Deploy Radiation Detection Equipment in the United States and in Other Countries. GAO-05-840T. Washington, D.C.: June 21, 2005.

Container Security: A Flexible Staffing Model and Minimum Equipment Requirements Would Improve Overseas Targeting and Inspection Efforts. GAO-05-557. Washington, D.C.: April 26, 2005.

Homeland Security: Key Cargo Security Programs Can Be Improved. GAO-05-466T. Washington, D.C.: May 26, 2005.

Maritime Security: Enhancements Made, but Implementation and Sustainability Remain Key Challenges. GAO-05-448T. Washington, D.C.: May 17, 2005.

Cargo Security: Partnership Program Grants Importers Reduced Scrutiny with Limited Assurance of Improved Security. GAO-05-404. Washington, D.C.: March 11, 2005.

Preventing Nuclear Smuggling: DOE Has Made Limited Progress in Installing Radiation Detection Equipment at Highest Priority Foreign Seaports. GAO-05-375. Washington, D.C.: March 31, 2005.

Homeland Security: Process for Reporting Lessons Learned from Seaport Exercises Needs Further Attention. GAO-05-170. Washington, D.C.: January 14, 2005.

Port Security: Better Planning Needed to Develop and Operate Maritime Worker Identification Card Program. GAO-05-106. Washington, D.C.: December 10, 2004.

Maritime Security: Substantial Work Remains to Translate New Planning Requirements into Effective Port Security. GAO-04-838. Washington, D.C.: June 30, 2004.

Homeland Security: Summary of Challenges Faced in Targeting Oceangoing Cargo Containers for Inspection. GAO-04-557T. Washington, D.C.: March 31, 2004.

Container Security: Expansion of Key Customs Programs Will Require Greater Attention to Critical Success Factors. GAO-03-770. Washington, D.C.: July 25, 2003.

This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.