Supply Chain Security: CBP Needs to Enforce Compliance and Assess the Effectiveness of the Importer Security Filing and Additional Carrier Requirements
Published: 2017-07-20T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Cargo shipments can present security concerns as terrorists could use cargo containers to transport a weapon of mass destruction or other contraband into the United States. In January 2009, CBP, within the Department of Homeland Security (DHS), implemented the ISF rule. The rule requires importers and vessel carriers to submit information, such as country of origin, to CBP before cargo is loaded onto U.S.-bound vessels. The information is intended to improve CBP's ability to identify high-risk shipments.
GAO was asked to review the ISF program. This report addresses: (1) importers' and carriers' submission rates for ISF rule requirements, (2) CBP's actions to enforce the ISF rule and assess whether enforcement actions have increased compliance, and (3) the extent to which the ISF rule has improved CBP's ability to identify high-risk shipments. GAO, among other things, analyzed CBP's compliance and enforcement data for 2012 through 2015—the most recent data available at the time of GAO's review—and interviewed CBP officials and trade industry members.
section_title: What GAO Found
paragraphs: Through the Importer Security Filing (ISF) and Additional Carrier Requirements (the ISF rule), U.S. Customs and Border Protection (CBP) requires importers to submit ISFs and vessel carriers to submit vessel stow plans and container status messages (CSM). Submission rates for ISF-10s—required for cargo destined for the United States—increased from about 95 percent in 2012 to 99 percent in 2015. Submission rates for ISF-5s—required for cargo transiting but not destined for the United States—ranged from about 68 to 80 percent. To increase ISF-5 submission rates, CBP published a Notice of Proposed Rulemaking in July 2016 to clarify the party responsible for submitting the ISF-5. GAO could not determine submission rates for vessel stow plans, which depict the position of each cargo container on a vessel, because CBP calculates stow plan submission rates on a daily basis, but not comprehensively over time. CBP officials noted, though, that compliance overall is likely nearly 100 percent because Advance Targeting Units (ATU), responsible for identifying high-risk shipments, contact carriers if they have not received stow plans. GAO also could not determine submission rates for CSMs, which report container movements and status changes, because CBP does not have access to carriers' private data systems to know the number of CSMs it should receive. CBP targeters noted that they may become aware that CSMs have not been sent based on other information sources they review.
CBP has taken actions to enforce ISF and stow plan submissions, but has not enforced CSM submissions or assessed the effects of its enforcement actions on compliance at the port level. ATUs enforce ISF and vessel stow plan compliance by using ISF holds, which prevent cargo from leaving ports, and issuing liquidated damages claims. CBP has not enforced CSM submissions because of the high volume it receives and lack of visibility into carriers' private data systems. However, when CBP targeters become aware that CSMs have not been received based on reviewing other information sources, taking enforcement actions could provide an incentive for carriers to submit all CSMs and help targeters better identify high-risk cargo. GAO's enforcement data analysis shows that ATUs used varying methods to enforce the ISF rule and that ports' ISF-10 submission rates varied. By assessing the effects of its enforcement strategies at the port level, CBP could better ensure it maximizes compliance with the rule.
CBP officials stated that ISF rule data have improved their ability to identify high-risk cargo shipments, but CBP could collect additional performance information to better evaluate program effectiveness. Evaluating the direct impact of ISF rule data in assessing shipment risk is difficult; however, GAO identified examples of how CBP could better assess the ISF program's effectiveness. For example, CBP could track the number of containers not listed on a manifest—which could pose a security risk—it identifies through reviewing vessel stow plans. Collecting this type of additional performance information could help CBP better assess whether the ISF program is improving its ability to identify high-risk shipments.
This is a public version of a sensitive report that GAO issued in May 2017. Information CBP deemed Law Enforcement Sensitive has been deleted.
section_title: What GAO Recommends
paragraphs: GAO recommends that CBP (1) enforce the CSM requirement when targeters identify carriers' noncompliance; (2) evaluate the effect of enforcement strategies on compliance at the port level; and (3) collect additional performance information to better evaluate the effectiveness of the ISF program. DHS concurred with the recommendations.

section_title: Letter
paragraphs: The U.S. economy is dependent on the expeditious flow of millions of  tons of cargo each day throughout the global supply chain—the flow of  goods from manufacturers to retailers or other end users. In 2015,  approximately 10.5 million cargo shipments, including 12.1 million cargo  containers, arrived at U.S. seaports. Criminal or terrorist attacks using  cargo shipments could cause disruptions to the supply chain and limit  global economic growth and productivity. Cargo shipments can present  significant security concerns, as individuals have exploited vulnerabilities  in the supply chain by using cargo containers to smuggle narcotics,  stowaways, and other contraband. Given these vulnerabilities, there is a  risk that terrorists could use a cargo container to transport a weapon of  mass destruction or other terrorist contraband into the United States. U.S.  Customs and Border Protection (CBP), a component within the  Department of Homeland Security (DHS), has responsibility for  administering cargo security and reducing the vulnerabilities associated  with the global supply chain.
In response to a requirement in the Security and Accountability for Every  Port Act of 2006 (SAFE Port Act) that DHS collect additional information  to identify high-risk cargo shipments for inspection, in January 2009, CBP  implemented the Importer Security Filing (ISF) and Additional Carrier  Requirements, generally referred to as the ISF rule. The rule requires  that importers (who order containerized and break bulk goods to be  shipped from foreign sources to the United States via oceangoing  vessels) and vessel carriers (who physically transport goods from foreign  ports to ports in the United States) submit additional cargo information,  such as country of origin, to CBP before the cargo is loaded onto U.S.- bound vessels. Specifically, importers are responsible for submitting the  ISF and vessel carriers are responsible for submitting vessel stow plans  and container status messages (CSM). According to the rule, collection of  the additional cargo information is intended to improve CBP’s ability to  identify high-risk shipments and prevent the transportation of terrorist  weapons and other contraband into the United States.
You requested that we review CBP’s implementation of its ISF program  related to compliance, enforcement, and performance. Specifically, this  report addresses the following questions:  (1)  What are importers’ and carriers’ submission rates for ISF rule  requirements, and to what extent does CBP monitor compliance?  (2)  To what extent has CBP taken actions to enforce compliance with  the ISF rule and assessed whether its enforcement actions have  contributed to increased compliance among importers and  carriers?  (3)  To what extent has the ISF program improved CBP’s ability to  identify high-risk cargo shipments prior to their arrival in the United  States, and to what extent are the data submitted under the  program accurate?
This report is a public version of a prior sensitive report that we provided  to you. CBP deemed some of the information in the prior report Law  Enforcement Sensitive, which must be protected from public disclosure.  Therefore, this report omits sensitive information regarding enforcement  actions used at individual ports, methods for using the ISF rule data to  identify high-risk shipments, and the results of an analysis that examined  the consistency between ISF and entry data, among other things. The  information provided in this report is more limited in scope, as it excludes  such sensitive information, but it addresses the same questions as the  sensitive report and the overall methodology used for both reports is the  same.
To determine importers’ and carriers’ submission rates for ISF rule  requirements, we analyzed CBP’s ISF data to determine national  submission rates for ISFs from January 2012 through December 2015— the 4 most recent years of data available at the time of our review. To  assess the reliability of CBP’s ISF data, we reviewed the data for obvious  errors, such as duplicative or missing fields. We discussed with CBP  officials how ISF data are processed and maintained. We determined the  data were sufficiently reliable to illustrate the national ISF submission  rate. We obtained CBP data on vessel stow plan submissions; however,  as described later in this report, we were not able to determine  submission rates because CBP was not able to provide us data on vessel  stow plans that were required, but ultimately not submitted to CBP. We  also could not determine submission rates for CSMs because CBP does  not have access to carriers’ private systems to know when CSMs have  been created and should be provided to CBP. To determine the extent to  which CBP monitors importers’ and carriers’ compliance with ISF rule  requirements, we reviewed documents used by CBP officials to monitor  ISF and stow plan compliance. We also interviewed CBP officials located  at headquarters and at select Advance Targeting Units (ATU)—units  responsible for identifying shipments that may be at risk for containing  terrorist weapons or other contraband. We selected five ATUs  responsible for shipments arriving at eight U.S. ports to reflect ports with  a range of ISF submission rates. We selected these ATUs based on  calendar year 2015 data because 2015 represented the most recent year  for which full year data were available at the time of our selection.  Although the results from our visits to the five ATUs are not generalizable  to all ATUs, the visits provided us insight into how and when ATU officials  monitor compliance of the required data.
To determine the extent to which CBP has taken actions to enforce the  ISF rule and assessed whether its enforcement actions have contributed  to increased compliance among importers and carriers, we gathered and  analyzed data on CBP’s actions to enforce the ISF rule and assessed its  actions against CBP’s enforcement goals and criteria for conducting  outcome evaluations. We reviewed relevant CBP policies, including  guidance to ATUs on enforcing the ISF rule and spoke with CBP officials  to understand the steps CBP has taken to enforce the ISF rule and  assess the effects of its enforcement actions. The five ATUs we selected  to visit were chosen because they are responsible for ports that had  varying ISF submission rates and also because the ATUs employed  varying enforcement methods. Although the results from our visits to  these five ATUs are not generalizable to all ATUs across the United  States, the visits allowed us to understand how individual ATUs enforce  the ISF rule. We obtained CBP data on ISF holds and liquidated damages  claims (LDC), which are the two types of enforcement actions that ATUs  primarily use to enforce compliance with the ISF rule. Specifically, we  analyzed the hold and LDC data to determine the number of these  actions used by ATUs to enforce compliance from 2012 through 2015— the same time period we used to analyze compliance rates. To assess  the reliability of CBP’s enforcement data, we reviewed the data for  obvious errors, such as duplicative or missing fields; performed a physical  case file review of several cases of LDCs at ATUs that we visited; and  discussed with CBP officials the results of our reviews. We also  discussed with CBP officials how the hold and LDC data are entered and  maintained in CBP’s systems. We found CBP’s data on ISF holds and  LDCs to be sufficiently reliable for reporting the number of holds and  LDCs used by ATUs and for selecting ATUs to visit. We also analyzed the  effectiveness of ISF holds as an enforcement action by developing a  statistical model measuring the association between ISF holds and the  rate at which importers complied with the requirement to submit an ISF to  CBP. We analyzed the effectiveness of ISF holds at each of the five  ATUs we visited by determining whether there were any differences in  effectiveness at the ports overseen by those ATUs. We were not able to  include LDCs in our statistical analysis because the small number of  LDCs was not sufficient to allow us to reliably assess their impact on  enforcement.
To determine the extent to which the ISF program has improved CBP’s  ability to identify high-risk cargo shipments prior to their arrival in the  United States—the goal of the ISF program—we reviewed quarterly  performance assessments on CBP’s identification of high-risk shipments  using ISF data. We reviewed quarterly performance for calendar years  2013 through 2015 because these 3 years contained sufficient data  against which to measure the performance of the algorithm CBP uses to  identify high-risk shipments. As described later, we were not able to  determine the direct effect of ISF data on the identification of high-risk  shipments because there are a variety of factors besides ISF data that  affect a shipment’s risk score. We discussed with CBP officials their plans  for assessing the ISF program consistent with requirements that call for  CBP to conduct a retrospective review of the rule. We interviewed CBP  officials and targeters at the five ATUs we visited to obtain insight on how  CBP targeters use ISF rule data to help assess the risk of arriving cargo  shipments. We also discussed targeting strategies using ISF rule data  with targeters at the five ATUs we visited. To examine the extent to which  ISF data used for targeting may be accurate, we analyzed the ISF data  for calendar years 2012 through 2015 to assess the accuracy of country  of origin data submitted to CBP. We discussed with CBP officials the  validation of ISFs used by its systems to prevent acceptance of ISFs with  missing or invalid country of origin codes.
We also interviewed a nongeneralizable sample of three importers, three  vessel carriers, and three industry associations to obtain insight on the  trade community’s compliance with the ISF rule, views of CBP’s  enforcement of the rule, and differences in submitting ISF versus entry  data. We selected importers and carriers who had experienced varying  levels of CBP enforcement. We selected trade industry associations that  represent importers, exporters, non-vessel operating common carriers,  and vessel carriers based on recommendations from CBP and our prior  work on cargo security. Additional details regarding our scope and  methodology are provided in appendixes I and II.
The performance audit upon which this report is based was conducted  from November 2015 to May 2017 in accordance with generally accepted  government auditing standards. Those standards require that we plan  and perform the audit to obtain sufficient, appropriate evidence to provide  a reasonable basis for our findings and conclusions based on our audit  objectives. We believe that the evidence obtained provides a reasonable  basis for our findings and conclusions based on our audit objectives. We  subsequently worked with CBP from May 2017 to July 2017 to prepare  this version of the original sensitive report for public release. This public  version was also prepared in accordance with these standards.
subsections: section_title: Background
subsections: section_title: Entities Involved in the Global Supply Chain
paragraphs: A number of entities are involved in the global supply chain, including the  following:  Importers: Bring cargo from a foreign source into a domestic market.  Importers are responsible for submitting ISF data, but an importer  may designate an authorized agent to file the ISF on its behalf.
Vessel carriers: Transport cargo from a foreign port to a U.S. port.
For foreign cargo remaining on board (FROB), the carrier is  considered the importer and is required to submit the ISF for the  shipment.
Licensed customs brokers: Assist in clearing cargo through  customs by preparing and filing proper entry forms, advising importers  on duties to be paid, and arranging for delivery of imported goods to  the destination. They also may act as the designated agent for  importers in submitting their ISFs.
Shippers: Supply or own the commodities that are being shipped.
Non-vessel operating common carriers: Buy shipping space on a  vessel, through a special arrangement with a vessel carrier, and resell  the space to individual shippers.
section_title: ISF Rule Requirements for Importers and Carriers
paragraphs: Importers are responsible for submitting the ISF, and the required ISF  data elements differ depending on the cargo’s destination. For cargo  bound for the United States as the final destination, the rule requires  importers to submit an ISF-10 to CBP 24 hours prior to vessel loading.  For cargo transiting the United States, but for which the United States is  not the final destination, the rule requires importers to submit an ISF-5 to  CBP prior to loading. See table 1 for further details on the ISF-10 and  ISF-5 required data elements.
Carriers transporting containers are to submit the Additional Carrier  Requirements, which include the following:
Vessel stow plan: No later than 48 hours after departure from the  last foreign port, carriers are to submit vessel stow plans to CBP, to  include the vessel operator, voyage number, the stow position of each  container, hazardous material code (if applicable), and the port of  discharge. For a voyage of less than 48 hours (short haul), CBP  requires that the stow plan be provided any time prior to arrival at the  first U.S. port. See figure 1 for an example of a vessel stow plan.
Container status messages: Carriers create CSMs to monitor  terminal container movements, such as loading and discharging of  vessels; as well as changes in the status of containers, such as if they  are empty or full. A carrier is to submit CSMs to CBP no later than 24  hours after the message is entered into the carrier’s equipment  tracking system.
section_title: Targeting High-Risk Shipments Using ISF Rule Data
paragraphs: According to the rule, ISF data are intended to improve CBP’s ability to  identify (target) high-risk shipments. The data elements are processed  and provided to CBP’s Automated Targeting System (ATS), which is a  decision support system that compares cargo and conveyance  information against intelligence and other law enforcement data. ATS  consolidates data from various sources to create a single, comprehensive  record for each U.S.-bound cargo shipment. Among other things, ATS  uses a set of rules that assess different factors in the data to determine  the risk of a shipment for particular threats, such as national security  threats or illegal drug trafficking. For example, one set of rules within  ATS, collectively referred to as the maritime national security weight set,  is programmed to check for information or patterns that could be  indicative of suspicious or terrorist activity.
As we have previously reported, the effectiveness of CBP’s security  strategy depends on CBP’s ability to use ATS and other tools to  effectively target those shipments that pose the greatest security risks.  CBP officials (targeters) use information in ATS to identify which  shipments to examine, which may include a non-intrusive inspection (NII)  scan or a physical inspection. The ATS risk score, however, is not the  sole factor that determines whether a CBP targeter reviews the data for a  shipment or whether the shipment is selected for a security examination.  CBP targeters we spoke with told us that they use the ATS risk score as a  starting point for the targeting process, but their decisions are ultimately  also based on additional research.
section_title: CBP Advance Targeting Units
paragraphs: CBP targeters are assigned to ATUs located at or near selected domestic  ports across the United States. Targeters at the ATUs are to review the  information associated with shipments destined for ports within their  respective regions to identify those shipments that may be at risk for  containing terrorist weapons or other contraband. An ATU may be  responsible for targeting shipments arriving at one port or multiple ports  within its region. For example, targeters at the Houston ATU are also  responsible for targeting shipments that are bound for ports in Freeport  and Galveston. CBP targeters at ATUs can review data as soon as  carriers and importers submit the required data (in accordance with the  24-hour rule and the ISF rule) and the data are available in ATS. Once a  shipment is loaded onto a U.S.-bound vessel, CBP targeters are to  continue to review shipment data in ATS because the data can be  updated or amended while the shipment is in transit to the U.S. port,  resulting in risk score changes.
According to CBP policy, targeters at ATUs are required to review data in  ATS for all medium-risk and high-risk shipments that are destined to  arrive at their respective ports. For example, a targeter may review  individual data elements, such as the name of the importer or other  supply chain parties, for these shipments. A targeter may also review the  weight set rules that detected potential threats and, therefore, contributed  to the calculation of the risk score. ATU targeters are also required to hold  high-risk shipments for examination unless they can mitigate the risk  through additional research or analysis of available information. Targeters  may conduct discretionary targeting by running queries of interest for  national security purposes or for other efforts, such as counternarcotics.  For example, targeters may independently create queries to identify items  of interest, such as all shipments of a particular commodity or those  coming from a particular country of origin. Targeters also have  responsibility for enforcing the ISF rule, and ATUs have discretion in  conducting enforcement activities based on the individual  characteristics—such as volume of shipments or length of voyage for  arriving cargo— of the ports that the ATUs oversee.
section_title: Submission Rates for ISFs Vary, While Carrier Requirement Rates Could Not Be Determined; CBP Generally Monitors Compliance
subsections: section_title: Submission Rates for ISF- 10s Have Generally Been High, and CBP is Taking Steps to Increase ISF-5 Submission Rates
paragraphs: Submission rates for ISF-10s have generally been high and CBP is taking  steps to increase the ISF-5 submission rate. In particular, submission  rates for shipments requiring an ISF-10 increased from approximately 95  percent in 2012 to 99 percent in 2015 (see figure 2). According to CBP  officials, from the ISF program’s beginning in January 2009, the  submission rates for ISF-10s generally rose as CBP gave importers time  to adjust to the new requirements. From January 2012 through June  2013, however, the submission rates remained at approximately 96  percent, on average. CBP officials told us that they suspect ISF  submission rates did not increase during that time period because some  importers had become complacent given that CBP had not yet increased  its enforcement actions. After CBP began taking greater enforcement  actions, beginning in July 2013, the submission rates increased to  approximately 98 percent by the end of 2013, and generally continued to  rise to 99 percent by the end of 2015.
Representatives from two of the three importers we interviewed, and an  association representing customs brokers, told us that the biggest  challenge in complying with the ISF rule is depending on third parties to  provide the information for the required ISF data elements. A  representative for one importer told us his company stationed 23  representatives abroad to educate vendors on the ISF requirements and  the penalties associated with filing late and added that he believes these  actions have helped increase his company’s ISF submission rate.
ISF-5 submission rates were lower than ISF-10 rates during the same  time period, ranging from approximately 68 percent in 2012 to 80 percent  in 2015 (see figure 3). According to CBP officials, ISF-5 submission rates  were lower because, as we previously reported in September 2010, the  ISF rule lacked clarity regarding the party responsible for submitting the  ISF-5. Specifically, CBP determined that in some cases the rule  designated a party as the ISF Importer even though that party had limited  access to the ISF data. As a result, CBP determined that it would not be  appropriate to enforce the ISF-5 requirement.
In July 2016, CBP published a Notice of Proposed Rulemaking, which  seeks to address the ISF-5 issue by expanding the definition of ISF  Importer to ensure that the party that has the best access to the required  information will be responsible for filing the ISF. According to the notice,  CBP also proposes expanding the definition of the ISF Importer to include  non-vessel operating common carriers for FROB shipments, because  when a party uses a non-vessel operating common carrier to book space  on a vessel, the vessel carrier frequently does not have access to the  required ISF data elements. CBP is also proposing to expand the  definition of ISF Importer for immediate exportation shipments,  transportation and exportation shipments, and for shipments to be  delivered to a foreign trade zone to include the goods’ owner, purchaser,  consignee, or agent, such as a licensed customs broker. According to the  Notice of Proposed Rulemaking, by broadening the definition to include  these parties, the responsibility to file the ISF will be with the party  causing the goods to arrive in the United States that will most likely have  access to the required ISF information. CBP estimates it will publish the  final rule in December 2017.
section_title: Submission Rates for Vessel Stow Plans and Container Status Messages Could Not Be Determined
paragraphs: We were not able to determine submission rates for the two additional  carrier requirements—vessel stow plans and CSMs—for 2012 through  2015. CBP provided us data on vessels that arrived in the United States  with vessel stow plans on file during this time period, but the data did not  include vessels that arrived in the United States and did not submit vessel  stow plans. As a result, we were not able to determine carriers’  compliance with the requirement to submit vessel stow plans. CBP  provided examples of daily reports it produced calculating the acceptance  rate of vessel stow plans submitted, but it has not comprehensively  calculated submission rates over time. According to CBP officials,  carriers’ overall compliance overall with stow plan submissions is likely  nearly 100 percent given that targeters at ATUs follow up with carriers  prior to vessel arrival if they have not yet submitted the vessel stow plan.  Similar to vessel stow plans, CBP provided us data on the number of  CSMs it receives, but is not able to produce data on the number of CSMs  it should have received. Carriers generate CSMs in their individual data  systems to capture movements and status changes and CBP officials told  us they do not have direct access to carriers’ private data systems to  know if a CSM has been created and is, therefore, required to be  submitted. As a result, we were not able to determine carriers’  compliance with the requirement to submit CSMs.
section_title: CBP Monitors Compliance with ISF and Vessel Stow Plan Submissions, but not CSMs
paragraphs: CBP has processes for monitoring daily whether importers and carriers  have submitted required ISFs and vessel stow plans, but not CSMs. In  particular, CBP headquarters officials told us they review daily reports on  ISF-10 and ISF-5 submission rates at each U.S. port to monitor the  overall level of compliance with the ISF requirement. For all shipments  scheduled to arrive at U.S. ports in approximately 2 days, CBP calculates  the percentage of shipments that have ISFs. For example, for shipments  scheduled to arrive in the United States on September 20, 2015, CBP  generated a report on September 18, 2015, that indicated that 21,114  shipments out of 21,593 shipments (about 98 percent) requiring ISF-10s  had an ISF. Additionally, four of the five ATUs we visited conduct queries  in ATS to identify shipments arriving in the near future without ISF-10s.  Similar to ISFs, CBP generates daily reports on vessels scheduled to  arrive in the United States without vessel stow plans on file. Also, all five  ATUs we visited have a process to identify arriving vessels with missing  stow plans and coordinate with the responsible carriers to obtain those  stow plans prior to the vessels arriving at their first U.S. port.
CBP officials stated that they are not able to comprehensively monitor  CSM submissions because, as previously discussed, CBP does not have  access to carriers’ private data systems to know if a CSM has been  created and if it was provided to CBP within 24 hours of being entered in  the carrier’s system. However, as we observed during our ATU visits,  targeters can identify if CSMs were not sent to CBP based on their  current knowledge of a container’s location when reviewing other sources  of information.
section_title: CBP Has Taken Actions to Enforce ISF and Stow Plan Submissions, but Could Do More to Enforce CSMs and Assess the Effectiveness of its ISF Enforcement Actions
subsections: section_title: CBP Has Taken Actions to Enforce Submission of ISF-10s and Stow Plans, but not CSMs
paragraphs: CBP primarily uses two types of enforcement actions—ISF holds and  liquidated damages claims—to enforce compliance with the ISF rule  among importers and carriers. An ISF hold can prevent a shipment from  leaving the U.S. port of arrival, and an LDC is similar to a monetary fine or  penalty. Upon implementation of the ISF rule in January 2009, CBP  delayed enforcement for 1 year to give the trade community time to adjust  to the rule’s requirements. In January 2010, CBP extended the period of  delayed enforcement while beginning to take some limited enforcement  actions against noncompliant importers by placing their shipments on  hold. In July 2013, CBP began full enforcement of the ISF rule by  authorizing ATUs to issue LDCs. The use of ISF holds also increased at  that time. Figure 4 shows key changes in CBP’s enforcement of the ISF  rule over time.
ISF Holds: In June 2010, CBP authorized ATUs to hold all shipments  with no ISF-10 on file. Depending on an ATU’s individual enforcement  policy, the shipment could remain on hold until an ISF is filed, be scanned  by NII equipment, or be physically inspected. For example, two of the  five ATUs that we visited do not remove an ISF hold from a noncompliant  shipment until the ISF-10 is submitted. Another ATU that we visited sends  shipments for physical inspection if an ISF-10 has not been submitted  within 96 hours of a shipment’s arrival. On the basis of our analysis of  CBP data, from 2012 through 2015, ATUs placed approximately 181,000  shipments on ISF hold, representing about 20 percent of shipments  arriving at U.S. ports without an accepted ISF-10. Figure 5 shows the  number of shipments ATUs placed on hold from 2012 through 2015.
Liquidated Damages Claims: In July 2013, CBP authorized ATUs to  issue LDCs to noncompliant importers and carriers for failure to submit  ISFs, vessel stow plans, or CSMs to CBP. The specific amount of an LDC  depends on the type of violation. For example, late submission of an ISF- 10 can result in a $5,000 LDC, while late filing of a vessel stow plan can  result in a $50,000 LDC. From May 2014 through June 2016, before  imposing a LDC on an importer, an ATU had to document three prior  violations, give a warning to the importer for each violation, and obtain  CBP headquarters’ approval. In June 2016, CBP authorized ATUs to  issue LDCs to importers without documenting three prior violations or  obtaining headquarters’ approval. LDCs for carriers still require  headquarters’ approval but do not require an ATU to document three prior  violations. From 2013 through 2015, ATUs issued 67 LDCs to 20  importers and 12 carriers (see figure 6).
While CBP generally enforces the ISF rule requirements to submit an  ISF-10 and vessel stow plan, it has not enforced the requirement that  carriers submit CSMs. None of the targeters at the ATUs we visited had  initiated any enforcement action (i.e., issued an LDC) against carriers for  not submitting CSMs, and we found no instances of an LDC issued for  CSM noncompliance in our analysis of CBP’s enforcement data.  According to CBP policy, CBP’s enforcement strategy is designed to  maximize importers’ and carriers’ compliance with the ISF rule, which  requires carriers to submit CSMs to CBP no later than 24 hours after the  CSM is entered into the carrier’s equipment tracking system. Targeters  at four of the five ATUs we visited said that CSMs are useful when  assessing the risk of arriving shipments because they provide a detailed  history of containers’ movements. For example, targeters can see if a  container was routed in an unusual way or transited a high-risk location.
Officials at CBP headquarters told us that ATUs do not have enough  resources to issue an LDC for each case of CSM noncompliance  because of the very high volume of CSMs—as many as 30 million per  month—that CBP receives. Officials at one ATU also told us they do not  enforce the CSM requirement because CSMs are often out of date.  Although it may not be feasible to determine every instance of CSM  noncompliance, targeters may identify cases of noncompliance when  reviewing CSMs for containers of interest as they target. For example, a  targeter at one ATU reviewed a container that had arrived at the port from  Guatemala in late April 2016, but the most recent CSM for the container  was from early March 2016. Therefore, according to the targeter, CBP  likely did not receive the most recent messages from the carrier. CBP  could issue LDCs when targeters identify CSM noncompliance during the  targeting process. By enforcing the CSM requirement when targeters  identify noncompliance, carriers would have a greater incentive to submit  all CSMs, thus providing CBP targeters with more comprehensive  information that could help them better assess the risk of cargo shipments  arriving at U.S. ports—the key goal of the ISF program.
section_title: Our Analysis Shows an Association between the Use of Holds and Increased ISF-10 Submission Rates, and Submission Rates Varied Across Ports Using Different Enforcement Methods
paragraphs: Using CBP data on ISF holds and ISF-10 submission rates, we analyzed  how CBP’s use of holds as an enforcement method was associated with  ISF-10 submission rates during calendar years 2012 through 2015. Our  analysis found that, nationally, the ISF-10 submission rate increased after  July 9, 2013, when CBP began its period of full enforcement of the ISF  rule and ATUs increased their use of ISF holds (see appendixes I and  II). Nationally, the ISF-10 submission rate was about 1.7 percentage  points higher on the 30th day after CBP began full enforcement,  compared to the day before the policy change. Further, our analysis of  CBP data found that ISF-10 submission rates varied across individual  ports overseen by ATUs that primarily used LDCs or did not use any  enforcement method. Submission rates at the two ports overseen by the  ATU that used the most LDCs and comparatively few ISF holds of the  ATUs we visited, remained relatively consistent at about 95 percentage  points before and after July 9, 2013. Additionally, the ISF-10 submission  rates at these ports were lower at various times from July 2013 through  2015 in our analysis, when compared to the rates at the ports overseen  by the other four ATUs we visited. Similarly, the ISF-10 submission rate at  the port overseen by an ATU we visited that generally did not take any  enforcement actions against noncompliant importers was consistently  lower, by approximately 2 to 15 percentage points, than the rates at the  ports overseen by the other four ATUs we visited. Nevertheless,  submission rates at this ATU increased after July 9, 2013, when CBP  began full enforcement, similar to the patterns at the ports with the  highest submission rates overseen by three of the ATUs we visited. This  increase in submission rates after full enforcement began, without the  ATU’s explicit use of holds, suggests that CBP’s broader enforcement  policy may have had an implicit deterrent effect.
section_title: CBP Has Not Assessed the Effects of Its Enforcement Actions to Maximize Compliance
paragraphs: CBP officials said CBP has not assessed the effects of its enforcement  actions—ISF holds and LDCs—including how its enforcement strategy  could be used to maximize importers’ and carriers’ compliance with the  ISF rule. CBP officials told us that, nationally, the ISF submission rate is  high—at around 99 percent—and that they credit the overall rise in  submission rates since 2009 to CBP’s enforcement efforts. However,  submission rates vary at individual ports overseen by ATUs that enforce  the ISF requirement differently. Some ATUs use holds and others use a  combination of holds and LDCs. ATUs also use different criteria for when  they place a hold on a noncompliant shipment or issue an LDC. Some  ATUs place holds on shipments without an ISF 24 hours before the  vessel arrives at the U.S. port, while other ATUs place holds on  shipments 48 or 72 hours before the vessel arrives at the port. Further,  ATUs apply different consequences to holds, such as using the hold to  take an image of a container’s contents or physically inspecting the  contents of a shipment.
According to CBP policy, the objective of CBP’s enforcement strategy is  to maximize importers’ and carriers’ compliance with the ISF rule.  However, officials said that CBP has not assessed whether its  enforcement actions are helping achieve the agency’s objective of  maximizing compliance, particularly among those ports with relatively low  compliance rates. For example, officials said CBP has not conducted an  evaluation to determine whether a particular enforcement action or  consequence of that action is more effective than another. CBP officials  said that compliance is already high, with an average national ISF-10  submission rate of about 99 percent. While the national submission rate is  high, some ATUs oversee ports with relatively low submission rates. It is  possible that submission rates might have been higher at individual ports  if CBP had used different enforcement approaches. In a previous report,  we reviewed various methods of evaluating programs and found that  program evaluations may be needed to examine the extent to which  programs are achieving their objectives. Specifically, outcome  evaluations can be used to assess program processes to understand how  outcomes are produced. We discussed with CBP officials different types  of evaluations, such as case studies of individual ports, that would be  feasible for it to conduct to evaluate ATUs’ different enforcement  methods. An evaluation of the effectiveness of its enforcement actions  could help inform CBP’s enforcement strategy and increase compliance  at ports with relatively low ISF-10 submission rates. Without such an  evaluation at the port level, CBP cannot be assured that its enforcement  strategy is meeting the objective of maximizing compliance with ISF rule  requirements.
section_title: CBP Reports the ISF Program Has Improved Its Ability to Identify High-Risk Shipments, but CBP Could Collect Additional Information to Better Evaluate Program Effectiveness
subsections: section_title: CBP Reports ISF Rule Data Have Helped Assess Shipment Risk, but Collecting Additional Performance Information Could Enhance Evaluation of Program Effectiveness
paragraphs: CBP officials told us that ISF rule data have improved CBP’s ability to  assess the risk of cargo shipments, but evaluating the direct effects of ISF  rule data on identifying high-risk shipments is difficult. However, we  identified examples of additional information CBP could collect to better  evaluate the program’s effectiveness. When assessing the risk of U.S.- bound cargo shipments, CBP relies, in part, on the use of ATS, as  described earlier. In January 2011, CBP incorporated ISF data into ATS’s  maritime national security weight set and since 2011 CBP staff have  assessed the performance of the updated weight set against a  performance target on a quarterly basis. The results of these  assessments show that in 11 of the 12 quarters during calendar years  2013 through 2015, the maritime national security weight set performed  better than a random inspection of shipments in identifying contraband.  However, determining the direct effect of ISF data on the identification of  high-risk shipments is not always possible because a shipment’s risk  score could be based on a variety of factors other than ISF data. As a  result, it is difficult to know the full effect of ISF data alone in identifying  shipments that ultimately contained contraband.
According to CBP targeters we spoke with, for shipments that ATS  identified as high-risk, having the ISF data early in the targeting process,  such as names and addresses, and more specific descriptions of cargo  than what a manifest provides, helps them better research shipments.  Also, some targeters we spoke with have used the ISF data to conduct  discretionary targeting and identify shipments for examination that were  not already identified by ATS as high risk. According to CBP, vessel stow  plans also help CBP assess shipment risk by allowing CBP to identify  unmanifested containers—containers and their associated contents not  listed on a vessel’s manifest—that pose a security risk in that no  information is known about their origin or contents. CBP prepares daily  reports identifying unmanifested containers arriving in the United States.  Further, according to CBP, CSMs help with shipment risk assessments by  providing CBP with information about containers’ movements and their  status (i.e., empty or full) that could indicate heightened security risks.
While CBP officials told us it is difficult to evaluate the direct impact of ISF  rule data in identifying high-risk shipments, collecting additional  performance information could help CBP assess and demonstrate  whether ISF rule data are contributing to the program’s goals. In our 2012  report addressing different types of evaluations for answering varied  questions about program performance, we found that a good evaluation  design should identify data sources and collection procedures to obtain  relevant, credible information to determine how well a program is  working. CBP, according to ISF program officials, has not evaluated the  effectiveness of the program because it believes that compliance is  already quite high, including a 99 percent submission rate for ISF-10s.
Although submission rates can be helpful in determining the extent to  which the required ISF data are being provided to CBP, it is important to  also demonstrate how or whether the ISF rule data are actually achieving  the broader program goal of improving CBP’s ability to assess cargo  shipments’ risks. For example, tracking the number of unmanifested  containers that ATUs discover as a result of reviewing vessel stow plans  could better reflect one benefit of the program. Additionally, identifying  instances in which ATUs discover or seize contraband as a result of  targeters reviewing ISF rule data when conducting discretionary targeting  would provide CBP with examples of how the data result in the  identification of high-risk shipments. By identifying and collecting such  additional information, CBP could better determine whether or how ISF  rule data are improving its ability to assess cargo shipment risks and  provide greater assurance that the ISF program, including the resources  invested, is helping to achieve intended goals.
Identifying and collecting additional performance information could also  provide CBP with useful information when evaluating the effectiveness of  the ISF program when it conducts its upcoming, required retrospective  review. In accordance with the Regulatory Flexibility Act, CBP is required  to evaluate the ISF program in 2018, as part of a 10-year retrospective  review. We previously reported practices identified by federal agencies  and nonfederal parties that could aid in the facilitation of useful  retrospective reviews, including preplanning to identify data and analysis  needed to conduct effective reviews. CBP officials told us they expect to  begin planning this year for the 2018 review.
section_title: CBP has Taken Steps to Ensure the Use of Valid Country of Origin Codes for Targeting
paragraphs: Our analysis of ISF data submitted to CBP from 2012 through 2015  showed that some ISFs had missing or invalid country of origin codes— one of the 10 data elements required in an ISF-10. The number of  missing and invalid codes is very small relative to the total number of ISF- 10s accepted during this time period, but as one of the ISF data elements  used to determine a shipment’s risk score, it is essential that valid country  of origin codes are fed into ATS. We discussed the results of our analysis  with CBP officials and, according to CBP, in December 2016, CBP  updated the validation rules used by its Automated Commercial  Environment system so that the system will no longer accept an ISF  unless it includes a valid, allowable country of origin code. We believe  the actions that CBP has taken should resolve the invalid country of origin  code problem we identified.
section_title: Conclusions
paragraphs: By implementing the ISF rule, CBP sought to reduce vulnerabilities in  supply chain security by requiring importers and carriers to submit  advance data that would help CBP better assess the risk of cargo  shipments prior to their arrival at U.S. ports. CBP has taken steps to  monitor and enforce the submission of ISFs and vessels stow plans  required by the ISF rule, and uses ISF rule data when assessing the risk  of arriving cargo shipments. However, CBP could take actions to better  enforce compliance and evaluate the effectiveness of the ISF program.  For example, by enforcing the requirement that carriers provide CSMs  when targeters identify noncompliance, CBP would have more accurate  and timely information for its targeters to use in identifying high-risk  shipments. The ISF program could also benefit from an evaluation of the  effectiveness of ATU’s enforcement methods since determining and  implementing the most effective enforcement strategy could increase  compliance with the ISF rule at ports with relatively low submission rates.  Further, collecting ISF program performance information would allow CBP  to better evaluate whether and how effectively the ISF program is meeting  its intended goal of improving the identification of high-risk cargo  shipments.
section_title: Recommendations for Executive Action
paragraphs: To enhance CBP’s identification of high-risk cargo shipments and its  enforcement of the ISF rule, we recommend that the Commissioner of  CBP take the following two actions: enforce the ISF rule requirement that carriers provide CSMs to CBP  when targeters identify CSM noncompliance; and evaluate the ISF enforcement strategies used by ATUs to assess  whether particular enforcement methods could be applied to ports  with relatively low submission rates.
Further, we recommend that the Commissioner of CBP identify and  collect additional performance information on the impact of the ISF rule  data, such as the identification of shipments containing contraband, to  better evaluate the effectiveness of the ISF program.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of the sensitive version of this report to DHS for its  review and comment. DHS provided technical comments, which have  been incorporated into this report, as appropriate. DHS also provided  written comments, which are reprinted in appendix III. In its comments,  DHS concurred with the report’s three recommendations and described  actions it has planned to address the recommendations by February 28,  2018.
DHS concurred with the first recommendation and stated that CBP plans  to develop a CSM enforcement policy and, once developed, plans to  disseminate the updated enforcement guidance to ATUs. DHS concurred  with the second recommendation and stated that CBP will discuss the ISF  enforcement strategies used by ATUs during monthly conference calls  and will work with ATUs overseeing ports with lower ISF submission rates  to identify potential solutions to increase submission rates at those ports.  DHS concurred with the third recommendation and stated that it will  analyze ISF data from a targeting standpoint to evaluate program  performance. Among other things, CBP plans to determine the number of  times potential terrorism matches were made against ISF data that were  not identified using manifest data. If implemented as planned, these  actions should address the intent of the recommendations to improve  CBP’s enforcement and assessment of the ISF program. We will continue  to monitor CBP’s efforts in addressing these recommendations.
If you or your staff have any questions about this report, please contact  me at (202) 512-7141 or groverj@gao.gov. Contact points for our Offices  of Congressional Relations and Public Affairs may be found on the last  page of this report. GAO staff who made key contributions to this reported  are listed in appendix IV.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: This report addresses U.S. Customs and Border Protection’s (CBP)  implementation of the Importer Security Filing (ISF) and Additional Carrier  Requirements (ISF rule). More specifically, our objectives were to  address: (1) importers’ and carriers’ compliance rates for ISF rule  requirements, and the extent to which CBP monitors their compliance; (2)  CBP’s actions to enforce the ISF rule and whether its enforcement  actions have contributed to increased compliance among importers and  carriers; and (3) whether the ISF program has improved CBP’s ability to  identify high-risk cargo shipments prior to their arrival in the United  States, and the extent to which data submitted under the program are  accurate.
To determine importers’ and carriers’ submission rates for ISF rule  requirements—ISFs, vessel stow plans, and container status messages  (CSM)—we obtained CBP data on importers’ and carriers’ compliance  with the ISF rule. Specifically, we analyzed CBP’s ISF data to determine  national submission rates for ISF-10s and ISF-5s, by month, from  January 2012 through December 2015—the 4 most recent years for  which data were available at the time of our review. To assess the  reliability of CBP’s ISF data, we reviewed the data for obvious errors,  such as duplicative or missing fields. We discussed with CBP officials  how ISF data are processed and maintained. We also discussed with  officials from the Advance Targeting Units (ATU) we visited the reliability  of the ISF submission rates for their respective ports. We determined the  data were sufficiently reliable to illustrate the national ISF submission rate  and for the ports overseen by the ATUs we visited. However, we  determined that the data were not sufficiently reliable for determining ISF  submission rates at all individual ports because CBP’s data also included  shipments associated with the wrong port or with a land port or airport as  submitted by carriers to CBP. We also obtained CBP data on vessel stow  plan submissions; however, we were not able to determine submission  rates because CBP was not able to provide us data on vessel stow plans  that were required, but ultimately not submitted to CBP. Further, we could  not determine submission rates for CSMs because CBP could provide us  data on the number of CSMs it received from carriers, but not those it did  not receive because CBP does not have access to carriers’ private  systems to know when CSMs have been created and should be provided  to CBP.
To determine the extent to which CBP monitors importers’ and carriers’  compliance with ISF rule requirements, we reviewed daily ISF and stow  plan reports used by CBP officials to monitor compliance. We also  interviewed CBP officials from the Office of Field Operations (OFO),  including the Office of Cargo and Conveyance Security, National  Targeting Center-Cargo (NTC-C) and selected ATUs. We selected five  ATUs responsible for shipments arriving at eight U.S. ports to reflect ports  with a range of ISF submission rates.
We used ISF-10 submission rates rather than ISF-5 submission rates as  our primary selection criterion because CBP was not enforcing ISF-5  compliance at the time of our review. We selected ATUs based on  calendar year 2015 data because it represented the most recent year for  which full year data were available at the time of our selection, and CBP  officials located at the ATUs selected would likely be more able to provide  insights on 2015 data than previous years’ data. Although the results from  our visits to the five ATUs are not generalizable to all targeting units, the  visits provided us insights regarding how and when ATU officials monitor  compliance for the requirements of the ISF rule and the factors that may  affect a port’s submission rates. We also interviewed a nongeneralizable  sample of three importers, three vessel carriers, and three trade industry  associations to understand their ability to comply with the ISF rule  requirements. Specifically, we asked importers, carriers, and members of  the trade industry about the steps they took to comply with the ISF  requirements and the factors that may affect compliance with any of the  requirements. We selected importers and carriers who had experienced  varying levels of CBP enforcement. We selected trade industry  associations based on recommendations from CBP and our prior work on  cargo security (see below for more detail on our selection criteria).
To determine the extent to which CBP has taken actions to enforce the  ISF rule and assessed whether its enforcement actions have contributed  to increased compliance, we compared CBP’s actions to enforce the ISF  rule and assessments of its actions against CBP’s enforcement goals and  criteria on conducting outcome evaluations. We reviewed relevant  statutes and CBP policies, including CBP guidance to ATUs on enforcing  the ISF rule. We spoke with CBP OFO officials from the Office of Cargo  and Conveyance Security; NTC-C; and Office of Fines, Penalties, and  Forfeitures to understand the steps CBP has taken to enforce the ISF rule  and assess the effect of its enforcement actions. The five ATUs we visited  are responsible for ports with varying ISF submission rates and were also  selected because they used varying enforcement methods. Although the  results from our visits to these five ATUs are not generalizable to all ATUs  across the United States, the visits allowed us to understand how  individual ATUs enforce the ISF rule given the discretion provided by the  ISF program.
We obtained CBP data on ISF holds and liquidated damages claims  (LDCs), which are the two types of enforcement actions that ATUs  primarily use to enforce compliance with the ISF rule. Specifically, we  analyzed hold data to determine the number of holds used by ATUs from  2012 through 2015, the same time period we used to analyze submission  rates. We analyzed CBP’s data on LDCs to determine the number of  LDCs that ATUs issued for ISF rule noncompliance, as well as the  monetary amounts that CBP assessed and collected. We analyzed LDC  data from July 2013 through 2015 because CBP authorized ATUs to use  LDCs beginning in July 2013, and 2015 was the last full calendar year for  which data were available. To assess the reliability of CBP’s enforcement  data, we reviewed the data for obvious errors, such as duplicative or  missing fields; performed a physical case file review of several cases of  LDCs at ATUs that we visited; and discussed with CBP officials the  results of our reviews. We also discussed with CBP officials how the hold  and LDC data are entered and maintained in the Cargo Enforcement  Reporting and Tracking System and the Seized Assets and Case  Tracking System, respectively. We found CBP’s data on ISF holds and  LDCs to be sufficiently reliable for reporting the number of holds and  LDCs used by ATU, and for selecting ATUs to visit.
We analyzed the effectiveness of ISF holds for enforcement by  developing a statistical model estimating the relationship between ISF  holds and the rate at which importers submitted required ISF-10s. To  develop the statistical model, we matched data on all shipments that  required ISF-10s from calendar years 2012 through 2015 to data on  whether importers submitted ISF-10s and whether ATUs placed ISF holds  on shipments. This 4-year time period spanned the date when CBP  increased enforcement of ISF-10 submissions through holds in July 2013,  which allowed us to assess how ISF holds were associated with changes  in ISF-10 submission rates. We analyzed the effectiveness of ISF holds at  each of the five ATUs we visited to determine whether there were any  differences in effectiveness at the ports overseen by those ATUs.  Although we found that ISF data are not reliable for each port, we  determined the data to be sufficiently reliable for our analysis of  enforcement at the ports we visited after ATU officials validated their  particular data. We could not analyze LDCs because CBP has issued too  few LDCs for us to reliably assess their association with ISF submission  rates. Appendix II provides technical details on the statistical methods we  used.
We also interviewed a nongeneralizable sample of three importers, three  vessel carriers, and three industry associations to obtain insight on the  trade community’s views of CBP’s enforcement of the ISF rule. We  selected importers and carriers that had experienced ISF holds and LDCs  during calendar years 2013 through 2015. Specifically, we selected two  importers with a consistently high number of holds and one importer with  a declining number of ISF holds. We selected three carriers, including (1)  the carrier that received the highest number of LDCs among those  carriers that received LDCs; (2) the carrier that paid the highest total  monetary amount to CBP for LDCs; and (3) the carrier with the second- highest number of LDCs, which also paid the second-highest monetary  amount to CBP. We selected trade industry associations that represent  importers, exporters, non-vessel operating common carriers, and vessel  carriers based on recommendations from CBP and our prior work on  cargo security.
To determine the extent to which the ISF program has improved CBP’s  ability to identify high-risk cargo shipments prior to their arrival in the  United States, we reviewed available performance data. Specifically, we  reviewed the results of CBP’s quarterly performance assessments of  ATS’s maritime national security weight set with ISF data incorporated,  for calendar years 2012 through 2015. We excluded the 2012 results  because there were limited data to evaluate, resulting in greater  uncertainty in the measurement of weight set performance for that year.  We were not able to determine the direct effect of ISF data on the  identification of high-risk shipments because there are a variety of factors  in addition to ISF data that can affect a shipment’s risk score. We  discussed with CBP officials their plans for review and assessment of the  ISF program consistent with regulatory requirements that call for CBP to  conduct a retrospective review of the rule. We also reviewed our prior  work on the importance of pre-planning to identify data needed in  advance of conducting a retrospective review. We interviewed CBP  officials and targeters at the five ATUs we visited to obtain insight on how  ISF rule data are used to help assess the risk of arriving cargo.
To examine the extent to which the data submitted under the ISF program  may be accurate, we analyzed the ISF data for calendar years 2012  through 2015 to assess the accuracy of country of origin data submitted  to CBP. Specifically we compared country of origin data contained in the  ISF-10s to CBPs legend of legitimate country of origin codes. We  discussed with CBP officials the ISF validation that occurs under its  legacy Automated Commercial System and the validation changes  incorporated in to the newer Automated Commercial Environment,  designed to prevent acceptance of ISFs with missing or erroneous  country of origin codes.
The performance audit upon which this report is based was conducted  from November 2015 to May 2017 in accordance with generally accepted  government auditing standards. Those standards require that we plan  and perform the audit to obtain sufficient, appropriate evidence to provide  a reasonable basis for our findings and conclusions based on our audit  objectives. We believe that the evidence obtained provides a reasonable  basis for our findings and conclusions based on our audit objectives. We  subsequently worked with CBP from May 2017 to July 2017 to prepare  this version of the original sensitive report for public release. This public  version was also prepared in accordance with these standards.
section_title: Appendix II: Statistical Modeling of ISF-10 Submission and Hold Rates
subsections: section_title: Statistical Modeling of ISF- 10 Submission and Hold Rates
paragraphs: We evaluated two recent changes to U.S. Customs and Border  Protection’s (CBP) enforcement policies. The first change occurred on  July 9, 2013, when CBP gave Advance Targeting Units (ATU),  responsible for screening arriving shipments, the option to issue  Liquidated Damages Claims (LDC), a type of fine, against shipments that  did not comply with Importer Security Filing (ISF) requirements. The new  policy also expanded ATUs’ ability to hold cargo shipments without proper  ISF-10 submissions at ports of entry. The second change occurred on  May 13, 2014, when CBP began issuing three warnings to noncompliant  importers before issuing LDCs.
In this appendix, we summarize our statistical analysis of CBP  administrative data to estimate the association between CBP’s  enforcement interventions and rates of ISF-10 submissions and cargo  holds.
section_title: Data and Analysis Population
paragraphs: Our target population included 36,137,951 bills of lading and their  importers that required ISF-10 submissions from calendar year 2012  through 2015. (A bill of lading is an instrument that allows a carrier to  transport merchandise from a shipper to a consignee.) We assigned each  bill-importer to the enforcement policy period that applied upon arrival at  the United States port of unlading.
A disaggregated analysis at the bill-importer level was not feasible,  because importer identification numbers were unavailable. The  identification number was required to match bills to their importers’ ISF-10  submissions and cargo holds. As a substitute, we analyzed aggregate  data by calculating aggregate ISF-10 submission and hold rates, among  other statistics, by day for analysis of nationwide data (n = 1461) and by  week for analysis of data from specific ATUs (n = 210).
Appendix I describes the specific CBP databases we analyzed in more  detail.
section_title: Models
paragraphs: We developed two types of interrupted time-series models of these data,  using the “single case” and “comparison group” designs.
In a single case design, time series data exist for one cross-sectional unit.  In our analysis, the single case was the United States as a whole. This  version of our analysis estimated how the submission and hold series  would have changed with and without each enforcement intervention.
In a comparison group design, the analysis is stratified across several  groups that received different levels of the intervention, such as units that  did not receive the treatment or kept status quo policies. A true  comparison group design was not possible here, because CBP changed  enforcement policies for all ATUs at the same times.
However, ATUs have exercised discretion under the policy to apply  different targeting methods. For example, one ATU primarily used LDCs  instead of cargo holds, and another ATU used relatively few holds or  LDCs. Accordingly, we conducted a version of the analysis that was  stratified across ATUs with sufficient data. This allowed us to estimate  how trends and associations varied across ATUs.
section_title: Single Case Model (Pooled Across ATUs)
paragraphs: Our single case analysis used binomial generalized linear models (GLMs)  to reflect that the outcomes of interest are counts and proportions of ISF- 10 submissions and holds from a fixed population of bill-importers.  Binomial models ensure that predictions and confidence intervals remain  within the unit interval. In addition, binomial models naturally  accommodate the heteroscedasticity likely to exist in our data, caused by  the varying number of bill-importers we used to estimate aggregate  statistics at each time.
Our models took the following general form, with results from the third  and most complex version reported below:  Y denotes the number of ISF-10s submitted in time period t, observed  from a population of n bill-importers requiring ISF-10s and having  submission rate π. T denotes time rescaled to elapsed units since the  sample origin, 2012-01-01. P(.) indicates whether the observation falls  into either policy intervention period, when 2013-07-09 ≤ T < 2014-05-03  or T≥ 2014-05-03, respectively. m is a vector of 11 month and 6 day of  week indicators (when aggregated daily) to allow for cyclical variation  (absorbing April and Wednesday into β).
We used the model to estimate several quantities of interest for a  Wednesday in April (i.e., at β0): π: Estimated ISF-10 submission probability
Logit-1(β + (β + β)T) - Logit-1(β + β): Change in probability  at time T from counterfactual mean outcome in the absence of  intervention 1.
Logit-1(β + (β + β)T) - Logit-1(β + β): Change in mean  outcome at time T from counterfactual mean outcome in the absence  of intervention 1 and 2.
Logit-1(β)T) - Logit-1(β+ β): Change in  mean outcome at time T from counterfactual mean outcome in the  absence of intervention 2 but in the presence of intervention 1.
We used Monte Carlo simulation methods to estimate the 99 percent  confidence intervals of these quantities. Specifically, letting g(β, x) denote  the functions of the parameter and covariate vectors above, we estimated  confidence intervals as  F-1 is the quantile function (inverse CDF of the sampling distribution)  evaluated at p = {.005, .995}, 𝐹𝐹�−1 is a standard empirical quantile  estimator, and 𝛽𝛽� is a random vector of 10,000 draws from the estimated  covariance matrix of β.
section_title: Comparison Group Model (Stratified by ATU)
paragraphs: We also estimated models stratified by ATU to allow for different  enforcement processes at different locations:  The models are defined as for the single case, except that the  intervention effect parameters are stratified across ATUs. That is, Ytj  denotes the number of compliant bills in time period t for ATU, j = {1, 2,  … , J}. Aj indicates the jth ATU, except A = 1 for all t. We estimated the  same quantities of interest as for the single case, but also estimated  differences in these quantities between certain ATUs. We estimated  confidence intervals using the same Monte Carlo simulation methods as  above.
Due to the data reliability problems we discuss in Appendix I, we could  reliably link bills, ports, and ATUs only for the ports overseen by the ATUs  we visited. We put all other ports into a residual category.
section_title: Model Diagnostics
paragraphs: We performed several diagnostics to assess model fit and assumptions.  We assessed model fit using the model deviance explained, which  quantifies how well the covariates explain the variation in the outcome of  interest. We assessed the independence of model residuals—a particular  concern for time series data—using Breusch-Godfrey test for  autocorrelation and the associated estimate of serial correlation at one  lag.
Table 2 provides the results of these diagnostics for the most complex  version of our models. All models fit the data well, explaining at least 90  percent of the deviance. Although we rejected the Breusch-Godfrey test  null hypothesis of zero residual autocorrelation (p < 10-5), all models had  residual serial correlations less than or equal to 0.23. To adjust for  potentially biased variance estimates due to positive autocorrelation, we  used a more conservative α = .01.
section_title: Results
paragraphs: Aggregate time-series analyses of CBP ISF-10 filing data show that policy  interventions on July 9, 2013, and May 13, 2014 are associated with  significant (α =0.01) increases in submission and hold rates.
Table 3 shows submission and hold rate differences from combined  analyses at 8 time period contrasts:
30 and 180 days after Policy Intervention 1 compared with 1 day
30, 180, and 540 days after Policy Intervention 2 compared with 1 day
30, 180, and 540 days after Policy Intervention 2 compared with 180  days after Policy Intervention 1  We found significant increases in submission and hold rates after the two  policy interventions, at α=0.01 (see table 3). The largest differences in  submission rates were estimated when comparing post intervention dates  to 1 day prior to policy intervention 1 (increases of 2 to 3 percentage  points), with smaller differences between post Policy Interventions 1 and  2. Hold rates showed similar patterns, with larger changes after Policy  Intervention 1 (0.6 to 1.2 percentage points).
We estimated differences in submission and hold rates for each ATU and  at 8 time periods, along with 99 percent confidence intervals. Key results  included:
We found both significant and nonsignificant decreases and increases  in submission rates after Policy Intervention 2 (-1.3 to 3.4 percentage  points).
Similar patterns generally existed for Hold Rates.
section_title: Sensitivity Analysis with Generalized Additive Models
paragraphs: As a sensitivity analysis, we fit generalized additive models (GAM) to  avoid specifying a linear model for trend ex ante. The general form of the  GAMs built upon the models described above such that –  π = Logit-1(βX + s))
Where β is the vector of predictors described in the parametric model  above, X is the vector of time-dependent covariates described in the  parametric model, and s(T) is a smooth function of time to be estimated.  The model fits and estimates for GLM and GAM analyses were  comparable, suggesting that the GLM results above are robust to linear  trends.
section_title: Appendix III: Comments from the Department of Homeland Security
section_title: Appendix IV: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Christopher Conrad (Assistant  Director), Carla Brown, Lisa Canini (Analyst-in-Charge), Ben Nelson,  Ashley Rawson, and Natarajan Subramanian made key contributions to  this report. Also contributing to this report were Michele Fejfar, Eric  Hauswirth, Susan Hsu, Won Lee, Heidi Nielson, Jeff Tessin, and Wayne  Turowski.
section_title: Related GAO Products
paragraphs: Maritime Security: Progress and Challenges in Implementing Maritime  Cargo Security Programs. GAO-16-790T. Washington, D.C.: July 7, 2016.
Supply Chain Security: CBP Needs to Enhance Its Guidance and  Oversight of High-Risk Maritime Cargo Shipments. GAO-15-294.  Washington, D.C.: January 27, 2015.
Maritime Security: Progress and Challenges with Selected Port Security  Programs. GAO-14-636T. Washington, D.C.: June 4, 2014.
Supply Chain Security: CBP Needs to Conduct Regular Assessments of  Its Cargo Targeting System. GAO-13-9. Washington, D.C.: October 25,  2012.
Supply Chain Security: CBP Has Made Progress in Assisting the Trade  Industry in Implementing the New Importer Security Filing Requirements,  but Some Challenges Remain. GAO-10-841. Washington, D.C.:  September 10, 2010.