Supply Chain Security: CBP Needs to Enforce Compliance and Assess the Effectiveness of the Importer Security Filing and Additional Carrier Requirements
Released: 2017-07-20T00:00:00

Letter

Background

Entities Involved in the Global Supply Chain

A number of entities are involved in the global supply chain, including the following: Importers: Bring cargo from a foreign source into a domestic market. Importers are responsible for submitting ISF data, but an importer may designate an authorized agent to file the ISF on its behalf.

Vessel carriers: Transport cargo from a foreign port to a U.S. port.

For foreign cargo remaining on board (FROB), the carrier is considered the importer and is required to submit the ISF for the shipment.

Licensed customs brokers: Assist in clearing cargo through customs by preparing and filing proper entry forms, advising importers on duties to be paid, and arranging for delivery of imported goods to the destination. They also may act as the designated agent for importers in submitting their ISFs.

Shippers: Supply or own the commodities that are being shipped.

Non-vessel operating common carriers: Buy shipping space on a vessel, through a special arrangement with a vessel carrier, and resell the space to individual shippers.

ISF Rule Requirements for Importers and Carriers

Importers are responsible for submitting the ISF, and the required ISF data elements differ depending on the cargo’s destination. For cargo bound for the United States as the final destination, the rule requires importers to submit an ISF-10 to CBP 24 hours prior to vessel loading. For cargo transiting the United States, but for which the United States is not the final destination, the rule requires importers to submit an ISF-5 to CBP prior to loading. See table 1 for further details on the ISF-10 and ISF-5 required data elements.

Carriers transporting containers are to submit the Additional Carrier Requirements, which include the following:

Vessel stow plan: No later than 48 hours after departure from the last foreign port, carriers are to submit vessel stow plans to CBP, to include the vessel operator, voyage number, the stow position of each container, hazardous material code (if applicable), and the port of discharge. For a voyage of less than 48 hours (short haul), CBP requires that the stow plan be provided any time prior to arrival at the first U.S. port. See figure 1 for an example of a vessel stow plan.

Container status messages: Carriers create CSMs to monitor terminal container movements, such as loading and discharging of vessels; as well as changes in the status of containers, such as if they are empty or full. A carrier is to submit CSMs to CBP no later than 24 hours after the message is entered into the carrier’s equipment tracking system.

Targeting High-Risk Shipments Using ISF Rule Data

According to the rule, ISF data are intended to improve CBP’s ability to identify (target) high-risk shipments. The data elements are processed and provided to CBP’s Automated Targeting System (ATS), which is a decision support system that compares cargo and conveyance information against intelligence and other law enforcement data. ATS consolidates data from various sources to create a single, comprehensive record for each U.S.-bound cargo shipment. Among other things, ATS uses a set of rules that assess different factors in the data to determine the risk of a shipment for particular threats, such as national security threats or illegal drug trafficking. For example, one set of rules within ATS, collectively referred to as the maritime national security weight set, is programmed to check for information or patterns that could be indicative of suspicious or terrorist activity.

As we have previously reported, the effectiveness of CBP’s security strategy depends on CBP’s ability to use ATS and other tools to effectively target those shipments that pose the greatest security risks. CBP officials (targeters) use information in ATS to identify which shipments to examine, which may include a non-intrusive inspection (NII) scan or a physical inspection. The ATS risk score, however, is not the sole factor that determines whether a CBP targeter reviews the data for a shipment or whether the shipment is selected for a security examination. CBP targeters we spoke with told us that they use the ATS risk score as a starting point for the targeting process, but their decisions are ultimately also based on additional research.

CBP Advance Targeting Units

CBP targeters are assigned to ATUs located at or near selected domestic ports across the United States. Targeters at the ATUs are to review the information associated with shipments destined for ports within their respective regions to identify those shipments that may be at risk for containing terrorist weapons or other contraband. An ATU may be responsible for targeting shipments arriving at one port or multiple ports within its region. For example, targeters at the Houston ATU are also responsible for targeting shipments that are bound for ports in Freeport and Galveston. CBP targeters at ATUs can review data as soon as carriers and importers submit the required data (in accordance with the 24-hour rule and the ISF rule) and the data are available in ATS. Once a shipment is loaded onto a U.S.-bound vessel, CBP targeters are to continue to review shipment data in ATS because the data can be updated or amended while the shipment is in transit to the U.S. port, resulting in risk score changes.

According to CBP policy, targeters at ATUs are required to review data in ATS for all medium-risk and high-risk shipments that are destined to arrive at their respective ports. For example, a targeter may review individual data elements, such as the name of the importer or other supply chain parties, for these shipments. A targeter may also review the weight set rules that detected potential threats and, therefore, contributed to the calculation of the risk score. ATU targeters are also required to hold high-risk shipments for examination unless they can mitigate the risk through additional research or analysis of available information. Targeters may conduct discretionary targeting by running queries of interest for national security purposes or for other efforts, such as counternarcotics. For example, targeters may independently create queries to identify items of interest, such as all shipments of a particular commodity or those coming from a particular country of origin. Targeters also have responsibility for enforcing the ISF rule, and ATUs have discretion in conducting enforcement activities based on the individual characteristics—such as volume of shipments or length of voyage for arriving cargo— of the ports that the ATUs oversee.

Submission Rates for ISFs Vary, While Carrier Requirement Rates Could Not Be Determined; CBP Generally Monitors Compliance

Submission Rates for ISF- 10s Have Generally Been High, and CBP is Taking Steps to Increase ISF-5 Submission Rates

Submission rates for ISF-10s have generally been high and CBP is taking steps to increase the ISF-5 submission rate. In particular, submission rates for shipments requiring an ISF-10 increased from approximately 95 percent in 2012 to 99 percent in 2015 (see figure 2). According to CBP officials, from the ISF program’s beginning in January 2009, the submission rates for ISF-10s generally rose as CBP gave importers time to adjust to the new requirements. From January 2012 through June 2013, however, the submission rates remained at approximately 96 percent, on average. CBP officials told us that they suspect ISF submission rates did not increase during that time period because some importers had become complacent given that CBP had not yet increased its enforcement actions. After CBP began taking greater enforcement actions, beginning in July 2013, the submission rates increased to approximately 98 percent by the end of 2013, and generally continued to rise to 99 percent by the end of 2015.

Representatives from two of the three importers we interviewed, and an association representing customs brokers, told us that the biggest challenge in complying with the ISF rule is depending on third parties to provide the information for the required ISF data elements. A representative for one importer told us his company stationed 23 representatives abroad to educate vendors on the ISF requirements and the penalties associated with filing late and added that he believes these actions have helped increase his company’s ISF submission rate.

ISF-5 submission rates were lower than ISF-10 rates during the same time period, ranging from approximately 68 percent in 2012 to 80 percent in 2015 (see figure 3). According to CBP officials, ISF-5 submission rates were lower because, as we previously reported in September 2010, the ISF rule lacked clarity regarding the party responsible for submitting the ISF-5. Specifically, CBP determined that in some cases the rule designated a party as the ISF Importer even though that party had limited access to the ISF data. As a result, CBP determined that it would not be appropriate to enforce the ISF-5 requirement.

In July 2016, CBP published a Notice of Proposed Rulemaking, which seeks to address the ISF-5 issue by expanding the definition of ISF Importer to ensure that the party that has the best access to the required information will be responsible for filing the ISF. According to the notice, CBP also proposes expanding the definition of the ISF Importer to include non-vessel operating common carriers for FROB shipments, because when a party uses a non-vessel operating common carrier to book space on a vessel, the vessel carrier frequently does not have access to the required ISF data elements. CBP is also proposing to expand the definition of ISF Importer for immediate exportation shipments, transportation and exportation shipments, and for shipments to be delivered to a foreign trade zone to include the goods’ owner, purchaser, consignee, or agent, such as a licensed customs broker. According to the Notice of Proposed Rulemaking, by broadening the definition to include these parties, the responsibility to file the ISF will be with the party causing the goods to arrive in the United States that will most likely have access to the required ISF information. CBP estimates it will publish the final rule in December 2017.

Submission Rates for Vessel Stow Plans and Container Status Messages Could Not Be Determined

We were not able to determine submission rates for the two additional carrier requirements—vessel stow plans and CSMs—for 2012 through 2015. CBP provided us data on vessels that arrived in the United States with vessel stow plans on file during this time period, but the data did not include vessels that arrived in the United States and did not submit vessel stow plans. As a result, we were not able to determine carriers’ compliance with the requirement to submit vessel stow plans. CBP provided examples of daily reports it produced calculating the acceptance rate of vessel stow plans submitted, but it has not comprehensively calculated submission rates over time. According to CBP officials, carriers’ overall compliance overall with stow plan submissions is likely nearly 100 percent given that targeters at ATUs follow up with carriers prior to vessel arrival if they have not yet submitted the vessel stow plan. Similar to vessel stow plans, CBP provided us data on the number of CSMs it receives, but is not able to produce data on the number of CSMs it should have received. Carriers generate CSMs in their individual data systems to capture movements and status changes and CBP officials told us they do not have direct access to carriers’ private data systems to know if a CSM has been created and is, therefore, required to be submitted. As a result, we were not able to determine carriers’ compliance with the requirement to submit CSMs.

CBP Monitors Compliance with ISF and Vessel Stow Plan Submissions, but not CSMs

CBP has processes for monitoring daily whether importers and carriers have submitted required ISFs and vessel stow plans, but not CSMs. In particular, CBP headquarters officials told us they review daily reports on ISF-10 and ISF-5 submission rates at each U.S. port to monitor the overall level of compliance with the ISF requirement. For all shipments scheduled to arrive at U.S. ports in approximately 2 days, CBP calculates the percentage of shipments that have ISFs. For example, for shipments scheduled to arrive in the United States on September 20, 2015, CBP generated a report on September 18, 2015, that indicated that 21,114 shipments out of 21,593 shipments (about 98 percent) requiring ISF-10s had an ISF. Additionally, four of the five ATUs we visited conduct queries in ATS to identify shipments arriving in the near future without ISF-10s. Similar to ISFs, CBP generates daily reports on vessels scheduled to arrive in the United States without vessel stow plans on file. Also, all five ATUs we visited have a process to identify arriving vessels with missing stow plans and coordinate with the responsible carriers to obtain those stow plans prior to the vessels arriving at their first U.S. port.

CBP officials stated that they are not able to comprehensively monitor CSM submissions because, as previously discussed, CBP does not have access to carriers’ private data systems to know if a CSM has been created and if it was provided to CBP within 24 hours of being entered in the carrier’s system. However, as we observed during our ATU visits, targeters can identify if CSMs were not sent to CBP based on their current knowledge of a container’s location when reviewing other sources of information.

CBP Has Taken Actions to Enforce ISF and Stow Plan Submissions, but Could Do More to Enforce CSMs and Assess the Effectiveness of its ISF Enforcement Actions

CBP Has Taken Actions to Enforce Submission of ISF-10s and Stow Plans, but not CSMs

CBP primarily uses two types of enforcement actions—ISF holds and liquidated damages claims—to enforce compliance with the ISF rule among importers and carriers. An ISF hold can prevent a shipment from leaving the U.S. port of arrival, and an LDC is similar to a monetary fine or penalty. Upon implementation of the ISF rule in January 2009, CBP delayed enforcement for 1 year to give the trade community time to adjust to the rule’s requirements. In January 2010, CBP extended the period of delayed enforcement while beginning to take some limited enforcement actions against noncompliant importers by placing their shipments on hold. In July 2013, CBP began full enforcement of the ISF rule by authorizing ATUs to issue LDCs. The use of ISF holds also increased at that time. Figure 4 shows key changes in CBP’s enforcement of the ISF rule over time.

ISF Holds: In June 2010, CBP authorized ATUs to hold all shipments with no ISF-10 on file. Depending on an ATU’s individual enforcement policy, the shipment could remain on hold until an ISF is filed, be scanned by NII equipment, or be physically inspected. For example, two of the five ATUs that we visited do not remove an ISF hold from a noncompliant shipment until the ISF-10 is submitted. Another ATU that we visited sends shipments for physical inspection if an ISF-10 has not been submitted within 96 hours of a shipment’s arrival. On the basis of our analysis of CBP data, from 2012 through 2015, ATUs placed approximately 181,000 shipments on ISF hold, representing about 20 percent of shipments arriving at U.S. ports without an accepted ISF-10. Figure 5 shows the number of shipments ATUs placed on hold from 2012 through 2015.

Liquidated Damages Claims: In July 2013, CBP authorized ATUs to issue LDCs to noncompliant importers and carriers for failure to submit ISFs, vessel stow plans, or CSMs to CBP. The specific amount of an LDC depends on the type of violation. For example, late submission of an ISF- 10 can result in a $5,000 LDC, while late filing of a vessel stow plan can result in a $50,000 LDC. From May 2014 through June 2016, before imposing a LDC on an importer, an ATU had to document three prior violations, give a warning to the importer for each violation, and obtain CBP headquarters’ approval. In June 2016, CBP authorized ATUs to issue LDCs to importers without documenting three prior violations or obtaining headquarters’ approval. LDCs for carriers still require headquarters’ approval but do not require an ATU to document three prior violations. From 2013 through 2015, ATUs issued 67 LDCs to 20 importers and 12 carriers (see figure 6).

While CBP generally enforces the ISF rule requirements to submit an ISF-10 and vessel stow plan, it has not enforced the requirement that carriers submit CSMs. None of the targeters at the ATUs we visited had initiated any enforcement action (i.e., issued an LDC) against carriers for not submitting CSMs, and we found no instances of an LDC issued for CSM noncompliance in our analysis of CBP’s enforcement data. According to CBP policy, CBP’s enforcement strategy is designed to maximize importers’ and carriers’ compliance with the ISF rule, which requires carriers to submit CSMs to CBP no later than 24 hours after the CSM is entered into the carrier’s equipment tracking system. Targeters at four of the five ATUs we visited said that CSMs are useful when assessing the risk of arriving shipments because they provide a detailed history of containers’ movements. For example, targeters can see if a container was routed in an unusual way or transited a high-risk location.

Officials at CBP headquarters told us that ATUs do not have enough resources to issue an LDC for each case of CSM noncompliance because of the very high volume of CSMs—as many as 30 million per month—that CBP receives. Officials at one ATU also told us they do not enforce the CSM requirement because CSMs are often out of date. Although it may not be feasible to determine every instance of CSM noncompliance, targeters may identify cases of noncompliance when reviewing CSMs for containers of interest as they target. For example, a targeter at one ATU reviewed a container that had arrived at the port from Guatemala in late April 2016, but the most recent CSM for the container was from early March 2016. Therefore, according to the targeter, CBP likely did not receive the most recent messages from the carrier. CBP could issue LDCs when targeters identify CSM noncompliance during the targeting process. By enforcing the CSM requirement when targeters identify noncompliance, carriers would have a greater incentive to submit all CSMs, thus providing CBP targeters with more comprehensive information that could help them better assess the risk of cargo shipments arriving at U.S. ports—the key goal of the ISF program.

Our Analysis Shows an Association between the Use of Holds and Increased ISF-10 Submission Rates, and Submission Rates Varied Across Ports Using Different Enforcement Methods

Using CBP data on ISF holds and ISF-10 submission rates, we analyzed how CBP’s use of holds as an enforcement method was associated with ISF-10 submission rates during calendar years 2012 through 2015. Our analysis found that, nationally, the ISF-10 submission rate increased after July 9, 2013, when CBP began its period of full enforcement of the ISF rule and ATUs increased their use of ISF holds (see appendixes I and II). Nationally, the ISF-10 submission rate was about 1.7 percentage points higher on the 30th day after CBP began full enforcement, compared to the day before the policy change. Further, our analysis of CBP data found that ISF-10 submission rates varied across individual ports overseen by ATUs that primarily used LDCs or did not use any enforcement method. Submission rates at the two ports overseen by the ATU that used the most LDCs and comparatively few ISF holds of the ATUs we visited, remained relatively consistent at about 95 percentage points before and after July 9, 2013. Additionally, the ISF-10 submission rates at these ports were lower at various times from July 2013 through 2015 in our analysis, when compared to the rates at the ports overseen by the other four ATUs we visited. Similarly, the ISF-10 submission rate at the port overseen by an ATU we visited that generally did not take any enforcement actions against noncompliant importers was consistently lower, by approximately 2 to 15 percentage points, than the rates at the ports overseen by the other four ATUs we visited. Nevertheless, submission rates at this ATU increased after July 9, 2013, when CBP began full enforcement, similar to the patterns at the ports with the highest submission rates overseen by three of the ATUs we visited. This increase in submission rates after full enforcement began, without the ATU’s explicit use of holds, suggests that CBP’s broader enforcement policy may have had an implicit deterrent effect.

CBP Has Not Assessed the Effects of Its Enforcement Actions to Maximize Compliance

CBP officials said CBP has not assessed the effects of its enforcement actions—ISF holds and LDCs—including how its enforcement strategy could be used to maximize importers’ and carriers’ compliance with the ISF rule. CBP officials told us that, nationally, the ISF submission rate is high—at around 99 percent—and that they credit the overall rise in submission rates since 2009 to CBP’s enforcement efforts. However, submission rates vary at individual ports overseen by ATUs that enforce the ISF requirement differently. Some ATUs use holds and others use a combination of holds and LDCs. ATUs also use different criteria for when they place a hold on a noncompliant shipment or issue an LDC. Some ATUs place holds on shipments without an ISF 24 hours before the vessel arrives at the U.S. port, while other ATUs place holds on shipments 48 or 72 hours before the vessel arrives at the port. Further, ATUs apply different consequences to holds, such as using the hold to take an image of a container’s contents or physically inspecting the contents of a shipment.

According to CBP policy, the objective of CBP’s enforcement strategy is to maximize importers’ and carriers’ compliance with the ISF rule. However, officials said that CBP has not assessed whether its enforcement actions are helping achieve the agency’s objective of maximizing compliance, particularly among those ports with relatively low compliance rates. For example, officials said CBP has not conducted an evaluation to determine whether a particular enforcement action or consequence of that action is more effective than another. CBP officials said that compliance is already high, with an average national ISF-10 submission rate of about 99 percent. While the national submission rate is high, some ATUs oversee ports with relatively low submission rates. It is possible that submission rates might have been higher at individual ports if CBP had used different enforcement approaches. In a previous report, we reviewed various methods of evaluating programs and found that program evaluations may be needed to examine the extent to which programs are achieving their objectives. Specifically, outcome evaluations can be used to assess program processes to understand how outcomes are produced. We discussed with CBP officials different types of evaluations, such as case studies of individual ports, that would be feasible for it to conduct to evaluate ATUs’ different enforcement methods. An evaluation of the effectiveness of its enforcement actions could help inform CBP’s enforcement strategy and increase compliance at ports with relatively low ISF-10 submission rates. Without such an evaluation at the port level, CBP cannot be assured that its enforcement strategy is meeting the objective of maximizing compliance with ISF rule requirements.

CBP Reports the ISF Program Has Improved Its Ability to Identify High-Risk Shipments, but CBP Could Collect Additional Information to Better Evaluate Program Effectiveness

CBP Reports ISF Rule Data Have Helped Assess Shipment Risk, but Collecting Additional Performance Information Could Enhance Evaluation of Program Effectiveness

CBP officials told us that ISF rule data have improved CBP’s ability to assess the risk of cargo shipments, but evaluating the direct effects of ISF rule data on identifying high-risk shipments is difficult. However, we identified examples of additional information CBP could collect to better evaluate the program’s effectiveness. When assessing the risk of U.S.- bound cargo shipments, CBP relies, in part, on the use of ATS, as described earlier. In January 2011, CBP incorporated ISF data into ATS’s maritime national security weight set and since 2011 CBP staff have assessed the performance of the updated weight set against a performance target on a quarterly basis. The results of these assessments show that in 11 of the 12 quarters during calendar years 2013 through 2015, the maritime national security weight set performed better than a random inspection of shipments in identifying contraband. However, determining the direct effect of ISF data on the identification of high-risk shipments is not always possible because a shipment’s risk score could be based on a variety of factors other than ISF data. As a result, it is difficult to know the full effect of ISF data alone in identifying shipments that ultimately contained contraband.

According to CBP targeters we spoke with, for shipments that ATS identified as high-risk, having the ISF data early in the targeting process, such as names and addresses, and more specific descriptions of cargo than what a manifest provides, helps them better research shipments. Also, some targeters we spoke with have used the ISF data to conduct discretionary targeting and identify shipments for examination that were not already identified by ATS as high risk. According to CBP, vessel stow plans also help CBP assess shipment risk by allowing CBP to identify unmanifested containers—containers and their associated contents not listed on a vessel’s manifest—that pose a security risk in that no information is known about their origin or contents. CBP prepares daily reports identifying unmanifested containers arriving in the United States. Further, according to CBP, CSMs help with shipment risk assessments by providing CBP with information about containers’ movements and their status (i.e., empty or full) that could indicate heightened security risks.

While CBP officials told us it is difficult to evaluate the direct impact of ISF rule data in identifying high-risk shipments, collecting additional performance information could help CBP assess and demonstrate whether ISF rule data are contributing to the program’s goals. In our 2012 report addressing different types of evaluations for answering varied questions about program performance, we found that a good evaluation design should identify data sources and collection procedures to obtain relevant, credible information to determine how well a program is working. CBP, according to ISF program officials, has not evaluated the effectiveness of the program because it believes that compliance is already quite high, including a 99 percent submission rate for ISF-10s.

Although submission rates can be helpful in determining the extent to which the required ISF data are being provided to CBP, it is important to also demonstrate how or whether the ISF rule data are actually achieving the broader program goal of improving CBP’s ability to assess cargo shipments’ risks. For example, tracking the number of unmanifested containers that ATUs discover as a result of reviewing vessel stow plans could better reflect one benefit of the program. Additionally, identifying instances in which ATUs discover or seize contraband as a result of targeters reviewing ISF rule data when conducting discretionary targeting would provide CBP with examples of how the data result in the identification of high-risk shipments. By identifying and collecting such additional information, CBP could better determine whether or how ISF rule data are improving its ability to assess cargo shipment risks and provide greater assurance that the ISF program, including the resources invested, is helping to achieve intended goals.

Identifying and collecting additional performance information could also provide CBP with useful information when evaluating the effectiveness of the ISF program when it conducts its upcoming, required retrospective review. In accordance with the Regulatory Flexibility Act, CBP is required to evaluate the ISF program in 2018, as part of a 10-year retrospective review. We previously reported practices identified by federal agencies and nonfederal parties that could aid in the facilitation of useful retrospective reviews, including preplanning to identify data and analysis needed to conduct effective reviews. CBP officials told us they expect to begin planning this year for the 2018 review.

CBP has Taken Steps to Ensure the Use of Valid Country of Origin Codes for Targeting

Our analysis of ISF data submitted to CBP from 2012 through 2015 showed that some ISFs had missing or invalid country of origin codes— one of the 10 data elements required in an ISF-10. The number of missing and invalid codes is very small relative to the total number of ISF- 10s accepted during this time period, but as one of the ISF data elements used to determine a shipment’s risk score, it is essential that valid country of origin codes are fed into ATS. We discussed the results of our analysis with CBP officials and, according to CBP, in December 2016, CBP updated the validation rules used by its Automated Commercial Environment system so that the system will no longer accept an ISF unless it includes a valid, allowable country of origin code. We believe the actions that CBP has taken should resolve the invalid country of origin code problem we identified.

Conclusions

By implementing the ISF rule, CBP sought to reduce vulnerabilities in supply chain security by requiring importers and carriers to submit advance data that would help CBP better assess the risk of cargo shipments prior to their arrival at U.S. ports. CBP has taken steps to monitor and enforce the submission of ISFs and vessels stow plans required by the ISF rule, and uses ISF rule data when assessing the risk of arriving cargo shipments. However, CBP could take actions to better enforce compliance and evaluate the effectiveness of the ISF program. For example, by enforcing the requirement that carriers provide CSMs when targeters identify noncompliance, CBP would have more accurate and timely information for its targeters to use in identifying high-risk shipments. The ISF program could also benefit from an evaluation of the effectiveness of ATU’s enforcement methods since determining and implementing the most effective enforcement strategy could increase compliance with the ISF rule at ports with relatively low submission rates. Further, collecting ISF program performance information would allow CBP to better evaluate whether and how effectively the ISF program is meeting its intended goal of improving the identification of high-risk cargo shipments.

Recommendations for Executive Action

To enhance CBP’s identification of high-risk cargo shipments and its enforcement of the ISF rule, we recommend that the Commissioner of CBP take the following two actions: enforce the ISF rule requirement that carriers provide CSMs to CBP when targeters identify CSM noncompliance; and evaluate the ISF enforcement strategies used by ATUs to assess whether particular enforcement methods could be applied to ports with relatively low submission rates.

Further, we recommend that the Commissioner of CBP identify and collect additional performance information on the impact of the ISF rule data, such as the identification of shipments containing contraband, to better evaluate the effectiveness of the ISF program.

Agency Comments and Our Evaluation

We provided a draft of the sensitive version of this report to DHS for its review and comment. DHS provided technical comments, which have been incorporated into this report, as appropriate. DHS also provided written comments, which are reprinted in appendix III. In its comments, DHS concurred with the report’s three recommendations and described actions it has planned to address the recommendations by February 28, 2018.

DHS concurred with the first recommendation and stated that CBP plans to develop a CSM enforcement policy and, once developed, plans to disseminate the updated enforcement guidance to ATUs. DHS concurred with the second recommendation and stated that CBP will discuss the ISF enforcement strategies used by ATUs during monthly conference calls and will work with ATUs overseeing ports with lower ISF submission rates to identify potential solutions to increase submission rates at those ports. DHS concurred with the third recommendation and stated that it will analyze ISF data from a targeting standpoint to evaluate program performance. Among other things, CBP plans to determine the number of times potential terrorism matches were made against ISF data that were not identified using manifest data. If implemented as planned, these actions should address the intent of the recommendations to improve CBP’s enforcement and assessment of the ISF program. We will continue to monitor CBP’s efforts in addressing these recommendations.

If you or your staff have any questions about this report, please contact me at (202) 512-7141 or groverj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this reported are listed in appendix IV.

Appendix I: Objectives, Scope, and Methodology

This report addresses U.S. Customs and Border Protection’s (CBP) implementation of the Importer Security Filing (ISF) and Additional Carrier Requirements (ISF rule). More specifically, our objectives were to address: (1) importers’ and carriers’ compliance rates for ISF rule requirements, and the extent to which CBP monitors their compliance; (2) CBP’s actions to enforce the ISF rule and whether its enforcement actions have contributed to increased compliance among importers and carriers; and (3) whether the ISF program has improved CBP’s ability to identify high-risk cargo shipments prior to their arrival in the United States, and the extent to which data submitted under the program are accurate.

To determine importers’ and carriers’ submission rates for ISF rule requirements—ISFs, vessel stow plans, and container status messages (CSM)—we obtained CBP data on importers’ and carriers’ compliance with the ISF rule. Specifically, we analyzed CBP’s ISF data to determine national submission rates for ISF-10s and ISF-5s, by month, from January 2012 through December 2015—the 4 most recent years for which data were available at the time of our review. To assess the reliability of CBP’s ISF data, we reviewed the data for obvious errors, such as duplicative or missing fields. We discussed with CBP officials how ISF data are processed and maintained. We also discussed with officials from the Advance Targeting Units (ATU) we visited the reliability of the ISF submission rates for their respective ports. We determined the data were sufficiently reliable to illustrate the national ISF submission rate and for the ports overseen by the ATUs we visited. However, we determined that the data were not sufficiently reliable for determining ISF submission rates at all individual ports because CBP’s data also included shipments associated with the wrong port or with a land port or airport as submitted by carriers to CBP. We also obtained CBP data on vessel stow plan submissions; however, we were not able to determine submission rates because CBP was not able to provide us data on vessel stow plans that were required, but ultimately not submitted to CBP. Further, we could not determine submission rates for CSMs because CBP could provide us data on the number of CSMs it received from carriers, but not those it did not receive because CBP does not have access to carriers’ private systems to know when CSMs have been created and should be provided to CBP.

To determine the extent to which CBP monitors importers’ and carriers’ compliance with ISF rule requirements, we reviewed daily ISF and stow plan reports used by CBP officials to monitor compliance. We also interviewed CBP officials from the Office of Field Operations (OFO), including the Office of Cargo and Conveyance Security, National Targeting Center-Cargo (NTC-C) and selected ATUs. We selected five ATUs responsible for shipments arriving at eight U.S. ports to reflect ports with a range of ISF submission rates.

We used ISF-10 submission rates rather than ISF-5 submission rates as our primary selection criterion because CBP was not enforcing ISF-5 compliance at the time of our review. We selected ATUs based on calendar year 2015 data because it represented the most recent year for which full year data were available at the time of our selection, and CBP officials located at the ATUs selected would likely be more able to provide insights on 2015 data than previous years’ data. Although the results from our visits to the five ATUs are not generalizable to all targeting units, the visits provided us insights regarding how and when ATU officials monitor compliance for the requirements of the ISF rule and the factors that may affect a port’s submission rates. We also interviewed a nongeneralizable sample of three importers, three vessel carriers, and three trade industry associations to understand their ability to comply with the ISF rule requirements. Specifically, we asked importers, carriers, and members of the trade industry about the steps they took to comply with the ISF requirements and the factors that may affect compliance with any of the requirements. We selected importers and carriers who had experienced varying levels of CBP enforcement. We selected trade industry associations based on recommendations from CBP and our prior work on cargo security (see below for more detail on our selection criteria).

To determine the extent to which CBP has taken actions to enforce the ISF rule and assessed whether its enforcement actions have contributed to increased compliance, we compared CBP’s actions to enforce the ISF rule and assessments of its actions against CBP’s enforcement goals and criteria on conducting outcome evaluations. We reviewed relevant statutes and CBP policies, including CBP guidance to ATUs on enforcing the ISF rule. We spoke with CBP OFO officials from the Office of Cargo and Conveyance Security; NTC-C; and Office of Fines, Penalties, and Forfeitures to understand the steps CBP has taken to enforce the ISF rule and assess the effect of its enforcement actions. The five ATUs we visited are responsible for ports with varying ISF submission rates and were also selected because they used varying enforcement methods. Although the results from our visits to these five ATUs are not generalizable to all ATUs across the United States, the visits allowed us to understand how individual ATUs enforce the ISF rule given the discretion provided by the ISF program.

We obtained CBP data on ISF holds and liquidated damages claims (LDCs), which are the two types of enforcement actions that ATUs primarily use to enforce compliance with the ISF rule. Specifically, we analyzed hold data to determine the number of holds used by ATUs from 2012 through 2015, the same time period we used to analyze submission rates. We analyzed CBP’s data on LDCs to determine the number of LDCs that ATUs issued for ISF rule noncompliance, as well as the monetary amounts that CBP assessed and collected. We analyzed LDC data from July 2013 through 2015 because CBP authorized ATUs to use LDCs beginning in July 2013, and 2015 was the last full calendar year for which data were available. To assess the reliability of CBP’s enforcement data, we reviewed the data for obvious errors, such as duplicative or missing fields; performed a physical case file review of several cases of LDCs at ATUs that we visited; and discussed with CBP officials the results of our reviews. We also discussed with CBP officials how the hold and LDC data are entered and maintained in the Cargo Enforcement Reporting and Tracking System and the Seized Assets and Case Tracking System, respectively. We found CBP’s data on ISF holds and LDCs to be sufficiently reliable for reporting the number of holds and LDCs used by ATU, and for selecting ATUs to visit.

We analyzed the effectiveness of ISF holds for enforcement by developing a statistical model estimating the relationship between ISF holds and the rate at which importers submitted required ISF-10s. To develop the statistical model, we matched data on all shipments that required ISF-10s from calendar years 2012 through 2015 to data on whether importers submitted ISF-10s and whether ATUs placed ISF holds on shipments. This 4-year time period spanned the date when CBP increased enforcement of ISF-10 submissions through holds in July 2013, which allowed us to assess how ISF holds were associated with changes in ISF-10 submission rates. We analyzed the effectiveness of ISF holds at each of the five ATUs we visited to determine whether there were any differences in effectiveness at the ports overseen by those ATUs. Although we found that ISF data are not reliable for each port, we determined the data to be sufficiently reliable for our analysis of enforcement at the ports we visited after ATU officials validated their particular data. We could not analyze LDCs because CBP has issued too few LDCs for us to reliably assess their association with ISF submission rates. Appendix II provides technical details on the statistical methods we used.

We also interviewed a nongeneralizable sample of three importers, three vessel carriers, and three industry associations to obtain insight on the trade community’s views of CBP’s enforcement of the ISF rule. We selected importers and carriers that had experienced ISF holds and LDCs during calendar years 2013 through 2015. Specifically, we selected two importers with a consistently high number of holds and one importer with a declining number of ISF holds. We selected three carriers, including (1) the carrier that received the highest number of LDCs among those carriers that received LDCs; (2) the carrier that paid the highest total monetary amount to CBP for LDCs; and (3) the carrier with the second- highest number of LDCs, which also paid the second-highest monetary amount to CBP. We selected trade industry associations that represent importers, exporters, non-vessel operating common carriers, and vessel carriers based on recommendations from CBP and our prior work on cargo security.

To determine the extent to which the ISF program has improved CBP’s ability to identify high-risk cargo shipments prior to their arrival in the United States, we reviewed available performance data. Specifically, we reviewed the results of CBP’s quarterly performance assessments of ATS’s maritime national security weight set with ISF data incorporated, for calendar years 2012 through 2015. We excluded the 2012 results because there were limited data to evaluate, resulting in greater uncertainty in the measurement of weight set performance for that year. We were not able to determine the direct effect of ISF data on the identification of high-risk shipments because there are a variety of factors in addition to ISF data that can affect a shipment’s risk score. We discussed with CBP officials their plans for review and assessment of the ISF program consistent with regulatory requirements that call for CBP to conduct a retrospective review of the rule. We also reviewed our prior work on the importance of pre-planning to identify data needed in advance of conducting a retrospective review. We interviewed CBP officials and targeters at the five ATUs we visited to obtain insight on how ISF rule data are used to help assess the risk of arriving cargo.

To examine the extent to which the data submitted under the ISF program may be accurate, we analyzed the ISF data for calendar years 2012 through 2015 to assess the accuracy of country of origin data submitted to CBP. Specifically we compared country of origin data contained in the ISF-10s to CBPs legend of legitimate country of origin codes. We discussed with CBP officials the ISF validation that occurs under its legacy Automated Commercial System and the validation changes incorporated in to the newer Automated Commercial Environment, designed to prevent acceptance of ISFs with missing or erroneous country of origin codes.

The performance audit upon which this report is based was conducted from November 2015 to May 2017 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. We subsequently worked with CBP from May 2017 to July 2017 to prepare this version of the original sensitive report for public release. This public version was also prepared in accordance with these standards.

Appendix II: Statistical Modeling of ISF-10 Submission and Hold Rates

Statistical Modeling of ISF- 10 Submission and Hold Rates

We evaluated two recent changes to U.S. Customs and Border Protection’s (CBP) enforcement policies. The first change occurred on July 9, 2013, when CBP gave Advance Targeting Units (ATU), responsible for screening arriving shipments, the option to issue Liquidated Damages Claims (LDC), a type of fine, against shipments that did not comply with Importer Security Filing (ISF) requirements. The new policy also expanded ATUs’ ability to hold cargo shipments without proper ISF-10 submissions at ports of entry. The second change occurred on May 13, 2014, when CBP began issuing three warnings to noncompliant importers before issuing LDCs.

In this appendix, we summarize our statistical analysis of CBP administrative data to estimate the association between CBP’s enforcement interventions and rates of ISF-10 submissions and cargo holds.

Data and Analysis Population

Our target population included 36,137,951 bills of lading and their importers that required ISF-10 submissions from calendar year 2012 through 2015. (A bill of lading is an instrument that allows a carrier to transport merchandise from a shipper to a consignee.) We assigned each bill-importer to the enforcement policy period that applied upon arrival at the United States port of unlading.

A disaggregated analysis at the bill-importer level was not feasible, because importer identification numbers were unavailable. The identification number was required to match bills to their importers’ ISF-10 submissions and cargo holds. As a substitute, we analyzed aggregate data by calculating aggregate ISF-10 submission and hold rates, among other statistics, by day for analysis of nationwide data (n = 1461) and by week for analysis of data from specific ATUs (n = 210).

Appendix I describes the specific CBP databases we analyzed in more detail.

Models

We developed two types of interrupted time-series models of these data, using the “single case” and “comparison group” designs.

In a single case design, time series data exist for one cross-sectional unit. In our analysis, the single case was the United States as a whole. This version of our analysis estimated how the submission and hold series would have changed with and without each enforcement intervention.

In a comparison group design, the analysis is stratified across several groups that received different levels of the intervention, such as units that did not receive the treatment or kept status quo policies. A true comparison group design was not possible here, because CBP changed enforcement policies for all ATUs at the same times.

However, ATUs have exercised discretion under the policy to apply different targeting methods. For example, one ATU primarily used LDCs instead of cargo holds, and another ATU used relatively few holds or LDCs. Accordingly, we conducted a version of the analysis that was stratified across ATUs with sufficient data. This allowed us to estimate how trends and associations varied across ATUs.

Single Case Model (Pooled Across ATUs)

Our single case analysis used binomial generalized linear models (GLMs) to reflect that the outcomes of interest are counts and proportions of ISF- 10 submissions and holds from a fixed population of bill-importers. Binomial models ensure that predictions and confidence intervals remain within the unit interval. In addition, binomial models naturally accommodate the heteroscedasticity likely to exist in our data, caused by the varying number of bill-importers we used to estimate aggregate statistics at each time.

Our models took the following general form, with results from the third and most complex version reported below: Y denotes the number of ISF-10s submitted in time period t, observed from a population of n bill-importers requiring ISF-10s and having submission rate π. T denotes time rescaled to elapsed units since the sample origin, 2012-01-01. P(.) indicates whether the observation falls into either policy intervention period, when 2013-07-09 ≤ T < 2014-05-03 or T≥ 2014-05-03, respectively. m is a vector of 11 month and 6 day of week indicators (when aggregated daily) to allow for cyclical variation (absorbing April and Wednesday into β).

We used the model to estimate several quantities of interest for a Wednesday in April (i.e., at β0): π: Estimated ISF-10 submission probability

Logit-1(β + (β + β)T) - Logit-1(β + β): Change in probability at time T from counterfactual mean outcome in the absence of intervention 1.

Logit-1(β + (β + β)T) - Logit-1(β + β): Change in mean outcome at time T from counterfactual mean outcome in the absence of intervention 1 and 2.

Logit-1(β)T) - Logit-1(β+ β): Change in mean outcome at time T from counterfactual mean outcome in the absence of intervention 2 but in the presence of intervention 1.

We used Monte Carlo simulation methods to estimate the 99 percent confidence intervals of these quantities. Specifically, letting g(β, x) denote the functions of the parameter and covariate vectors above, we estimated confidence intervals as F-1 is the quantile function (inverse CDF of the sampling distribution) evaluated at p = {.005, .995}, 𝐹𝐹�−1 is a standard empirical quantile estimator, and 𝛽𝛽� is a random vector of 10,000 draws from the estimated covariance matrix of β.

Comparison Group Model (Stratified by ATU)

We also estimated models stratified by ATU to allow for different enforcement processes at different locations: The models are defined as for the single case, except that the intervention effect parameters are stratified across ATUs. That is, Ytj denotes the number of compliant bills in time period t for ATU, j = {1, 2, … , J}. Aj indicates the jth ATU, except A = 1 for all t. We estimated the same quantities of interest as for the single case, but also estimated differences in these quantities between certain ATUs. We estimated confidence intervals using the same Monte Carlo simulation methods as above.

Due to the data reliability problems we discuss in Appendix I, we could reliably link bills, ports, and ATUs only for the ports overseen by the ATUs we visited. We put all other ports into a residual category.

Model Diagnostics

We performed several diagnostics to assess model fit and assumptions. We assessed model fit using the model deviance explained, which quantifies how well the covariates explain the variation in the outcome of interest. We assessed the independence of model residuals—a particular concern for time series data—using Breusch-Godfrey test for autocorrelation and the associated estimate of serial correlation at one lag.

Table 2 provides the results of these diagnostics for the most complex version of our models. All models fit the data well, explaining at least 90 percent of the deviance. Although we rejected the Breusch-Godfrey test null hypothesis of zero residual autocorrelation (p < 10-5), all models had residual serial correlations less than or equal to 0.23. To adjust for potentially biased variance estimates due to positive autocorrelation, we used a more conservative α = .01.

Results

Aggregate time-series analyses of CBP ISF-10 filing data show that policy interventions on July 9, 2013, and May 13, 2014 are associated with significant (α =0.01) increases in submission and hold rates.

Table 3 shows submission and hold rate differences from combined analyses at 8 time period contrasts:

30 and 180 days after Policy Intervention 1 compared with 1 day

30, 180, and 540 days after Policy Intervention 2 compared with 1 day

30, 180, and 540 days after Policy Intervention 2 compared with 180 days after Policy Intervention 1 We found significant increases in submission and hold rates after the two policy interventions, at α=0.01 (see table 3). The largest differences in submission rates were estimated when comparing post intervention dates to 1 day prior to policy intervention 1 (increases of 2 to 3 percentage points), with smaller differences between post Policy Interventions 1 and 2. Hold rates showed similar patterns, with larger changes after Policy Intervention 1 (0.6 to 1.2 percentage points).

We estimated differences in submission and hold rates for each ATU and at 8 time periods, along with 99 percent confidence intervals. Key results included:

We found both significant and nonsignificant decreases and increases in submission rates after Policy Intervention 2 (-1.3 to 3.4 percentage points).

Similar patterns generally existed for Hold Rates.

Sensitivity Analysis with Generalized Additive Models

As a sensitivity analysis, we fit generalized additive models (GAM) to avoid specifying a linear model for trend ex ante. The general form of the GAMs built upon the models described above such that – π = Logit-1(βX + s))

Where β is the vector of predictors described in the parametric model above, X is the vector of time-dependent covariates described in the parametric model, and s(T) is a smooth function of time to be estimated. The model fits and estimates for GLM and GAM analyses were comparable, suggesting that the GLM results above are robust to linear trends.

Appendix III: Comments from the Department of Homeland Security

Appendix IV: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, Christopher Conrad (Assistant Director), Carla Brown, Lisa Canini (Analyst-in-Charge), Ben Nelson, Ashley Rawson, and Natarajan Subramanian made key contributions to this report. Also contributing to this report were Michele Fejfar, Eric Hauswirth, Susan Hsu, Won Lee, Heidi Nielson, Jeff Tessin, and Wayne Turowski.

Related GAO Products

Maritime Security: Progress and Challenges in Implementing Maritime Cargo Security Programs. GAO-16-790T. Washington, D.C.: July 7, 2016.

Supply Chain Security: CBP Needs to Enhance Its Guidance and Oversight of High-Risk Maritime Cargo Shipments. GAO-15-294. Washington, D.C.: January 27, 2015.

Maritime Security: Progress and Challenges with Selected Port Security Programs. GAO-14-636T. Washington, D.C.: June 4, 2014.

Supply Chain Security: CBP Needs to Conduct Regular Assessments of Its Cargo Targeting System. GAO-13-9. Washington, D.C.: October 25, 2012.

Supply Chain Security: CBP Has Made Progress in Assisting the Trade Industry in Implementing the New Importer Security Filing Requirements, but Some Challenges Remain. GAO-10-841. Washington, D.C.: September 10, 2010.