Imported Food Safety: FDA's Targeting Tool Has Enhanced Screening, but Further Improvements Are Possible
Published: 2016-05-26T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Imported food makes up a substantial and growing portion of the U.S. food supply. FDA is responsible for oversight of more than 80 percent of the U.S. food supply. However, because the volume of imported food is so high, FDA physically examines only about 1 percent of imported food annually. In 2011, FDA implemented PREDICT, a computerized tool intended to improve FDA's targeting of imports for examination by estimating the risk of imported products.
GAO was asked to review how FDA is using PREDICT to protect the U.S. food supply. This report examines (1) the data used by PREDICT and how PREDICT analyzes these data to identify high-risk food shipments for examination, (2) how implementation of FSMA will affect PREDICT, and (3) the extent to which FDA has assessed the effectiveness of PREDICT and used the assessments to improve the tool. To address these issues, GAO analyzed FDA documents, interviewed FDA officials, and analyzed data from fiscal years 2012 through 2014.
section_title: What GAO Found
paragraphs: The Food and Drug Administration's (FDA) Predictive Risk-based Evaluation for Dynamic Import Compliance Targeting (PREDICT) tool uses a variety of data and analyzes data by applying rules—conditional statements that tell PREDICT how to react when encountering particular information—to generate risk scores for imported food. Many of the data used by PREDICT come from internal FDA sources, such as FDA databases. PREDICT also uses data from sources outside of FDA, such as other federal agencies, states, and foreign governments. Some of the data are open source data—information that is publicly available, such as information from newspapers and websites. FDA's Office of Regulatory Affairs (ORA) relies on other FDA offices and federal agencies to provide open source data for PREDICT, but ORA does not have a documented process for identifying, obtaining, and using the data, relying instead on an ad hoc process. Federal standards for internal control call for agencies to document internal controls. Without a documented process for identifying, obtaining, and using open source data, FDA does not have reasonable assurance that ORA will consistently identify, obtain, and use such data for PREDICT.
The implementation of the FDA Food Safety Modernization Act (FSMA), enacted in 2011, will provide PREDICT with additional data for estimating the risk of imported food. FDA identified FSMA provisions likely to generate data, including the Foreign Supplier Verification Program (FSVP), which requires importers to verify that their foreign suppliers use processes and procedures that provide the same public health protection as applicable U.S. requirements. Because FSVP and other FSMA-related programs are still new and not yet fully implemented, the details of how PREDICT will use the data have not been worked out. However, according to FDA officials, the data will be useful. For example, FSVP data will identify suppliers that are not in compliance with standards, and PREDICT will use those data in assigning risk scores to imports from those suppliers.
FDA has assessed the effectiveness of PREDICT by monitoring of key data and by conducting an internal evaluation of the system, and it has implemented many, but not all, of its recommendations from the evaluation. GAO's analysis of FDA data from fiscal year 2012 through 2014 shows that in general, PREDICT is working as intended: imported food with higher-risk scores is more likely to be physically examined and to be found in violation of food safety standards or labeling requirements. In May 2013, FDA completed an evaluation of PREDICT that examined five key processes. As a result of the evaluation, FDA developed 24 recommendations for improving PREDICT and prioritized these recommendations based on feasibility and impact. According to FDA, the agency has fully implemented 15, partially implemented 6, and not implemented 3 of the recommendations. FDA officials said that the agency has not fully implemented all recommendations because of a lack of resources. However, federal standards for internal control specify that agencies are to ensure that the findings of reviews are promptly resolved. To that end, agencies are to complete, within established time frames, all actions that correct or otherwise resolve the matters brought to management's attention. Establishing a timeline for implementing the remaining recommendations as resources become available would help ensure that PREDICT continues to remain an effective tool for screening imported food.
section_title: What GAO Recommends
paragraphs: GAO recommends that FDA take two actions to improve the effectiveness of PREDICT: (1) document the process by which FDA is to identify, obtain, and use open source data and (2) establish a timeline for implementing the remaining recommendations from FDA's 2013 evaluation of PREDICT. FDA generally agreed with GAO's recommendations.

section_title: Letter
paragraphs: Imported food makes up a substantial and growing portion of the U.S.  food supply. According to a 2013 report by the Food and Drug  Administration (FDA), 15 percent of the U.S. food supply was imported,  including nearly 50 percent of fresh fruits and 20 percent of fresh  vegetables. In addition, according to the National Marine Fisheries  Service (NMFS) about 94 percent of seafood consumed in the United  States was imported in 2014. FDA, under the Department of Health and  Human Services, is responsible for oversight of more than 80 percent of  the U.S. food supply. In fiscal year 2002, imported entry lines—an entry  line is each portion of an import shipment that is listed as a separate item  on an entry document—of FDA-regulated food totaled approximately 4.4  million. According to FDA, by 2015, this number had nearly tripled to  approximately 12 million. The total volume of U.S. food imports under  FDA’s jurisdiction has increased from approximately 39 million metric tons  in 1999 to 67 million metric tons in 2014, based on data compiled by the  U.S. Department of Agriculture’s (USDA) Economic Research Service  (see fig. 1). Given the volume of imports into the country, there is  considerable potential for violative items—products that do not meet U.S.  safety standards or labeling requirements—to enter the U.S. food supply.
FDA is responsible for helping to ensure that certain food products  marketed in the United States meet certain statutory and regulatory  requirements, whether the products are produced in the United States or  another country. FDA staff work with personnel from the Department of  Homeland Security’s U.S. Customs and Border Protection (CBP) to  examine violative products. In 2007, we added the federal oversight of  food safety to our list of high-risk areas needing broad-based  transformation, largely because of inconsistent oversight, ineffective  coordination, and inefficient use of resources. Since then, we have found  shortcomings with FDA’s oversight of imported food safety. We made  numerous recommendations, and the agency has implemented all of  them.
The FDA Food Safety Modernization Act (FSMA), enacted in January  2011, expanded and modified previously existing FDA authorities, giving  FDA several new authorities that enhance the agency’s oversight of  imported food. Also, in December 2011, FDA fully implemented its  Predictive Risk-based Evaluation for Dynamic Import Compliance  Targeting (PREDICT), a computerized tool intended to improve the  screening of FDA-regulated imports and the targeting of entry lines for  examination. PREDICT is designed to estimate the risk of imports using  information such as the history of the facility, inspection records, and  country of origin. FDA staff called entry reviewers use these risk  estimates to target for examination shipments with high levels of risk,  while the tool encourages entry reviewers to allow shipments with low  levels of risk to enter the United States with minimal review.
In 2014, we found that FDA did not examine for pesticide residues the  majority of entry lines that had the highest PREDICT risk scores and that  the agency’s plan to test the tool’s reliability lacked statistically valid  measures. The Department of Health and Human Services indicated that  it would investigate the feasibility and potential costs of implementing this  recommendation and described its ongoing efforts. In 2015, we found  that FDA was not conducting the number of foreign food facility  inspections—the results of which are used in PREDICT—mandated  under FSMA. We recommended that the agency complete an analysis  to determine the annual number of foreign food inspections that is  sufficient to ensure comparable safety of imported and domestic food  and, if appropriate, recommend appropriate legislative changes. FDA  concurred with this recommendation, pending the acquisition of the  resources necessary to conduct the analysis.
With imported food making up a substantial and growing portion of the  U.S. food supply, you asked us to review how FDA is using PREDICT to  protect the safety of the U.S. food supply. This report examines (1) the  data used by PREDICT and how PREDICT analyzes these data to  identify high-risk food shipments for examination, (2) how implementation  of FSMA will affect PREDICT, and (3) the extent to which FDA has  assessed the effectiveness of PREDICT and used the results of those  assessments to improve the tool.
To determine what data were used by PREDICT and how PREDICT  analyzes these data to identify high-risk food shipments for examination,  we reviewed FDA data and interviewed FDA officials. We also assessed  the data used by PREDICT and determined that they were sufficiently  reliable to describe factors considered by PREDICT and system rules  used to generate PREDICT risk scores. We examined the data sources,  data quality controls, and methodology used by PREDICT and assessed  them to ensure that they were sufficiently reliable to support accurate  descriptions of the system inputs, processes, and resulting risk scores. To  determine how implementation of FSMA will affect PREDICT, we  reviewed FDA proposed and final rules implementing FSMA, reviewed  FDA documents, and interviewed FDA officials. To determine the extent  to which FDA has assessed the effectiveness of PREDICT and used the  results of those assessments to improve the tool, we interviewed FDA  officials about the ways in which FDA has assessed PREDICT, reviewed  the evaluation that FDA has conducted, and assessed the extent to which  FDA has implemented the recommendations from the evaluation.
As discussed above, we examined the data sources, data quality  controls, and methodology used by PREDICT and assessed them to  ensure that they were sufficiently reliable to support accurate descriptions  of the system inputs, processes, and resulting risk scores. To inform all  three objectives, we conducted site visits at FDA and CBP facilities  located at ports of entry located in Baltimore, Maryland; Long Beach,  California; Los Angeles, California; and San Diego, California. We  selected these sites to include air, ship, and truck ports of entry; variable  geographic locations; the variability of food products that enter through  the ports; and proximity to an FDA office. The information we obtained is  not generalizable to all facilities and ports of entry. Additional information  on our scope and methodology is included in appendix I.
We conducted this performance audit from January 2015 to May 2016 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: This section discusses the offices within FDA and other federal agencies  that help oversee imported food products. It also discusses FDA’s  approach for overseeing the safety of the imported food products for  which it is responsible, including milk, seafood, fruits, and vegetables.
subsections: section_title: Responsibilities for Oversight of Imported Food Products
paragraphs: FDA’s responsibilities for overseeing the safety of imported food products  are divided among its product centers and program offices. For example,  the Center for Food Safety and Applied Nutrition (CFSAN) is responsible  for regulating food and cosmetics products. FDA’s Office of Regulatory  Affairs (ORA) is the lead office for ensuring the safety of imported food  through various field activities, such as inspections of firms, review of  imported products, examination of products, and sample collection and  analysis. ORA works closely with the centers and with the Office of  International Programs (OIP), which coordinates some of the agency’s  international activities. FDA has staff in China, Chile, Costa Rica, India,  Mexico, and the European Union. These foreign offices help ORA obtain  foreign scientific and regulatory information, conduct investigations and  facility inspections in foreign countries, and facilitate collaboration with  foreign regulators in areas of common interest.
Primary responsibility for imported food safety is divided between FDA  and FSIS, but other federal agencies play a role. For example, CBP  supports FDA in its enforcement of its food safety regulations at the  border, among other things. CBP’s automated systems process all  imported shipments, including food. In addition, NMFS provides fee-for- service inspection services, upon request, to the seafood industry— including domestic and foreign processors, distributors, and other firms— for example, to certify that these seafood firms comply with FDA’s Hazard  Analysis and Critical Control Point regulations and other federal food  safety standards. Some retailers request this certification as a condition  for purchasing seafood products.
section_title: FDA’s Approach to Overseeing the Safety of Imported Food
paragraphs: FDA oversees the safety of imported food by promoting corporate  responsibility, examining food prior to it entering into U.S. commerce to  help ensure that unsafe food does not enter the country, and responding  when unsafe food does enter the country.
In 2011, FSMA provided FDA with new provisions that hold imported  foods to the same standards as domestic foods and prevent unsafe food  from entering the country. These new provisions included new  requirements and authorities for FDA, including a requirement to develop a comprehensive plan to expand the food  safety capacity of foreign governments; a requirement to increase inspections of foreign food facilities and the  authority to enter into cooperative arrangements with foreign  governments to facilitate the inspection of foreign food facilities; the authority, under certain circumstances, to require as a condition of  admission that imported food be accompanied by certifications or  assurances from accredited third-party certification bodies, agencies,  or representatives of foreign governments where the food originated;  and  the authority to require importers to verify that their foreign suppliers  meet applicable U.S. food safety standards.
Owing in part to the volume of imported food, FDA cannot physically  examine every shipment; the agency examines about 1 percent of entry  lines annually. FDA electronically screens all imported food shipments to  determine which imports to physically examine at the border and which  imports to allow into U.S. commerce.
The electronic screening process consists of two phases: (1) Prior Notice  screening, which is intended to protect against potential terrorist acts and  other public health emergencies, and (2) admissibility screening, which is  intended to ensure that the food is admissible under the Food, Drug, and  Cosmetic Act (FDCA). Imported food products are generally considered  admissible if they are in compliance with applicable FDCA regulations  that ensure food is not adulterated, misbranded, manufactured or packed  under insanitary conditions, or restricted in sale in the country in which it  was produced or from which it was exported, among other things.
The first phase, Prior Notice screening, requires that an importer, broker,  or other entity submit notification to FDA of food being imported or offered  for import into the United States before that food arrives at the port of  entry. Prior Notice information includes data such as the names and  addresses of the manufacturer or grower, importer, owner, and ultimate  consignee (the “deliver to” location); FDA product code for the food item;  and country of production. Information that FDA uses for Prior Notice  review can either be submitted electronically through CBP’s system,  which passes the information to FDA, or submitted electronically directly  to FDA. FDA targets, screens, and reviews the information to ensure that  the information meets the Prior Notice requirements and to determine  whether the food potentially poses a terrorism threat or other significant  health risk. If such risks are identified, FDA is to work with CBP to  examine the shipment upon arrival at the port. If adequate Prior Notice  information is not provided, the food is subject to refusal of admission and  may not be delivered to the importer, owner, or consignee. If FDA  subsequently verifies that adequate Prior Notice information is provided  and the food does not appear to pose a terrorism threat, the shipment is  allowed to proceed to FDA’s admissibility screening.
During the second phase, admissibility screening, FDA electronically  screens entry lines using PREDICT to determine their level of risk.  “Risk”—reflected by the PREDICT risk score—includes factors such as  the inherent health risk of the product, compliance risk associated with  firms, facility inspection results, and broker history, among others, and is  then compared to all other entry lines within a specified commodity over  the past 30 days. If PREDICT determines that an entry line poses a low  risk, PREDICT recommends the entry be allowed to enter into U.S.  commerce, and another FDA system, called the Operational and  Administrative System for Import Support (OASIS), issues a system- generated “May Proceed” message, allowing the product into U.S.  commerce without further review. If PREDICT determines that the entry  line poses a high risk, an entry reviewer decides whether to examine the  entry line for admissibility. The entry reviewer determines if the entry line  is under import alert. If the entry line is not under import alert and no  further examination is needed, the entry reviewer allows the entry line into  U.S. commerce by issuing a manual “May Proceed” message through  OASIS. The entry reviewer may determine that a field or laboratory  examination is needed for an entry line that is not under an import alert. In  this case, an FDA investigator examines the entry either at the port of  entry or at another location, such as the importer’s or consignee’s  warehouse or a cold storage facility. The investigator may examine a  product’s label to determine whether it meets labeling requirements.  Additionally, the investigator may examine the shipment for rodent or  insect activity or inadequate storage while in transit, among other things.  Figure 2 shows FDA investigators examining imported food.
If the product does not appear to be violative after the examination, the  owner or consignee receives a notice stating that the line is released, and  the product is allowed into U.S. commerce.
If the product appears to be violative, the investigator may decide to take  samples from the product for a laboratory examination to test for rodent or  insect activity and other such elements that could confirm that the entry is  violative. If the samples indicate that the food is not violative, the owner or  consignee receives a notice stating that the line is released, and the  product is allowed into U.S. commerce. If the samples are violative, the  owner or consignee receives a notice stating that the line is being  detained and is subject to refusal. Once the owner or consignee receives  a notice stating that the line is being detained and is subject to refusal,  the owner or consignee may request that FDA immediately refuse the  product, and the product must either be exported or destroyed. If the  owner or consignee does not request refusal, then the owner or  consignee decides whether to submit testimony or request to  “recondition” the product—for example, relabeling the product or  converting the product into a type of product not regulated by FDA. If  the owner or consignee submits testimony regarding the admissibility of  the food or FDA approves a request to recondition the product, then an  FDA hearing determines whether the product should be released. If FDA  determines that the owner or consignee has provided sufficient  information to overcome the appearance of a violation, the owner or  consignee receives a notice stating that the product is released. If FDA  determines that the owner or consignee’s actions failed to bring the  product into compliance, the food must be exported or destroyed.
If the product is subject to an import alert, FDA determines whether the  product should be detained without physical examination. If FDA  determines that the product should not be detained, the owner or  consignee receives a notice stating that the product is released. If FDA  determines that the product should be detained, FDA issues a notice  stating that the line is being detained and is subject to refusal, and the  owner or consignee decides whether to export or destroy the product.  Figure 3 illustrates the key elements of FDA’s screening process.
From 2012 to 2014, about 0.1 percent of all food entry lines were refused  and exported or destroyed each year. During this period the countries  with the highest number of entry lines that were refused were India,  Mexico, and China and the most commonly refused items included rice,  herbals and botanicals (not teas), tuna, shrimp and aquaculture- harvested seafood products, and vitamins and minerals.
When unsafe food enters the country, FDA may respond by issuing  advisories about the affected food. Depending on the circumstances, FDA  may also use other enforcement tools, such as seizure, injunction, and  administrative detention. FDA can also seek voluntary recalls of unsafe  food. If a responsible party does not voluntarily recall the food, FDA may  issue a mandatory recall order, provided that certain criteria are met.
In early 2010, FDA began an effort to assess foreign food safety systems  to determine whether certain other countries have a regulatory system— including food safety statutes, regulations, and an implementation  strategy—that is comparable to the U.S. food safety system. Under this  effort, FDA in 2010 developed a systems recognition tool to assess the  overall food safety systems of foreign countries for foods under FDA’s  jurisdiction. FDA completed its on-site review for a systems recognition  pilot with New Zealand food safety authorities in late 2010 and  established a Systems Recognition Arrangement in 2012. In May 2016,  FDA reported that it had established a similar arrangement with Canada.
section_title: PREDICT Uses Data and Analyzes the Data by Applying Rules That Contribute to Risk-Based Scores, but FDA Does Not Have a Documented Process for Obtaining Some Data
paragraphs: PREDICT uses a variety of data and FDA-created rules—conditional  statements that tell PREDICT how to react when encountering particular  information—to analyze these data and to identify high-risk imported food  shipments. These data include information from FDA sources; non-FDA  domestic sources, such as other federal and state agencies; and foreign  sources, such as foreign governments. Some of the domestic and foreign  sources are open sources—that is, publicly available. FDA does not  currently have a documented process for obtaining open source data.  PREDICT analyzes all of the data by applying rules that contribute to risk- based scores. PREDICT then recommends potential FDA actions, such  as holding an entry line for examination or allowing it to proceed into  commerce.
subsections: section_title: PREDICT Uses a Variety of Data, but FDA Does Not Have a Documented Process for Obtaining Open Source Data
paragraphs: PREDICT uses a variety of data to estimate the risk of imported food.  These data come from internal FDA sources, other domestic sources,  and foreign sources. Figure 4 provides an overview of the types of data  sources PREDICT uses.
Many of the data PREDICT uses come from sources within FDA. These  include electronic sources, such as databases, and human sources, such  as FDA officials.
Electronic sources. Most of the electronic FDA sources that  PREDICT uses are databases that contain historical data about  products, firms, and other elements of imported shipments, such as  the geographic location of facilities. PREDICT uses numerous kinds of  historical data stored in various FDA databases, such as data about  (1) the previous field examination and facility inspection history of  foreign firms, (2) the track record of importers and brokers, and (3) the  types of food products historically imported from other countries. One  of the other databases that PREDICT uses contains official FDA  safety violation data, such as FDA import alerts and import bulletins.  Another database that PREDICT uses contains registration data for  facilities that manufacture, process, or pack a specific type of food  product, acidified and low-acid canned foods, which are specifically  tracked because growth of the botulinum bacterium in canned food  may cause a deadly form of food poisoning.
Human sources. The human FDA sources that PREDICT draws  upon include FDA officials who compile and report data based on their  knowledge of products, firms, and other elements of imported  shipments, such as country of origin. For example, certain PREDICT  rules use FDA’s knowledge and experience to determine the inherent  risk of certain food products, and certain other PREDICT rules draw  from data reported by FDA entry reviewers about the past record of  products or firms.
subsections: section_title: Data from Non-FDA Domestic Sources
paragraphs: According to FDA officials, PREDICT also uses data from sources outside  of FDA, including domestic sources, such as databases from states and  other federal agencies. For example, PREDICT uses data from CBP.  Specifically, CBP collects entry data from importers for products being  imported into the United States and electronically transfers the data to  FDA. PREDICT also uses data from other federal agencies, such as  from NMFS. NMFS maintains a list of approved seafood establishments,  based on its fee-for-service inspections, on its website and may provide  this list electronically to FDA upon request. However, FDA only requests  information from NMFS on an occasional, as-needed basis. For example,  FDA requested information from NMFS in 2010 during the Gulf of Mexico  oil spill. More recently, FDA requested NMFS audit reports and facility  approvals for certain overseas locations.
section_title: Data from Foreign Sources
paragraphs: FDA officials told us that PREDICT also uses data from foreign sources,  such as foreign governments. Specifically, FDA’s domestic and overseas  offices obtain information from foreign regulatory counterparts on an  occasional, as-needed basis about food safety in their respective  countries and provide summary information to ORA that may be used in  PREDICT. FDA officials told us that the agency may receive nonpublic  information from foreign governments only in cases where formal  arrangements called Confidentiality Commitments are in place. Of the  more than 230 entities from which the United States imports food, FDA  has 39 foods-related Cooperative Arrangements with 24 foreign  governments—including 1 arrangement with the European Union, which  has 28 member countries. FDA officials told us that they are in the  process of developing additional cooperative arrangements.
An FDA office that is involved in obtaining information from foreign  regulatory counterparts is OIP, which has offices in the United States and  abroad that help obtain foreign scientific and regulatory information by  coordinating with foreign regulators or accessing foreign media sources.  One country covered by OIP’s Office of Regional and Country Affairs is  Japan. In March 2011, after a 9.0 magnitude earthquake and subsequent  tsunami in Japan, the Fukushima Daiichi nuclear power plant suffered  extensive damage. The Office of Regional and Country Affairs  coordinated with the Japanese government to identify Japanese exports  that have a high risk of contamination. As a result, FDA issued Import  Alert 99-33: “Detention Without Physical Examination of Products from  Japan Due to Radionuclide Contamination.” This alert informs FDA field  personnel that they may detain without physical examination certain food  shipments from Japan. FDA officials told us that years after the disaster,  the agency continues to obtain information from the Japanese  government, and the import alert is still in place and was last revised in  April 2016. As of March 10, 2014, FDA had tested 1,345 import and  domestic samples specifically to monitor for Fukushima contamination. Of  the 1,345 samples, 2 were found to contain detectable levels of a  contaminant, but the levels posed no public health concern.
section_title: Open Source Data
paragraphs: Both the non-FDA domestic sources and the foreign sources may include  open sources—that is, sources that are publicly available, such as  newspapers and Internet blogs. Data from open sources may provide  information about events—such as food recalls and natural disasters— that could affect the risk of imported foods. Although ORA collects some  open source data on its own, it also relies on other FDA offices and  federal agencies that gather open source data for their own purposes and  that may communicate to ORA information that seems relevant to the  import screening mission. For example, OIP’s Europe office monitors  news related to FDA-regulated products in Europe. If OIP obtains recall  information that it deems significant to FDA-regulated commodities, it  communicates the information to ORA and other relevant FDA offices.  ORA may also obtain open source data from other federal agencies, such  as from the agencies represented at CBP’s Commercial Targeting and  Analysis Center, which targets and screens commercial shipments that  may pose a threat to the health and safety of U.S. consumers. Open  source data have proven useful in the past. For example, in August 2015,  an explosion occurred at a warehouse in the port of Tianjin, China,  involving hazardous chemicals, including cyanide, posing a potential  threat to public health. In the aftermath, OIP, through its China Office,  collected information about the explosion from local media sources. As a  result, FDA increased surveillance of FDA-regulated imports from  manufacturing, processing, and/or packing facilities in the Tianjin area. In  this particular instance, FDA also notified importers of this increased level  of surveillance.
Prior to 2015, FDA had a formal contract dedicated to the collection of  open source data for PREDICT. FDA’s 2013 PREDICT operating manual,  PREDICT Guide: Rules and Scoring, documented how the agency was to  obtain open source data, the type of data to be provided by the  contracted company—such as data about natural disasters—and how  PREDICT used the data. According to the manual, the PREDICT Open- Source Intelligence Team used open source data to locate news stories  pertaining to food safety around the world, looking specifically for  geophysical events (e.g., tsunamis, typhoons, and earthquakes) or  ecological events (e.g., algal blooms, overuse of antibiotics, and  contamination from various spills that could affect FDA-regulated  commodities). Analysts then researched each event or action to  understand the content and context of the issue, develop an analytic  conclusion, and recommend a risk score and expiration date.
FDA terminated that contract because the agency determined that it was  not cost-effective, given the availability of information from other public  sources, such as the Internet and other offices within the agency. The  2015 version of FDA’s PREDICT operating manual does not document  the process for identifying the type of open source data to collect,  obtaining such data, and determining how PREDICT is to use the data.  Instead, FDA relies on ORA officials to informally communicate and  obtain information from officials in other FDA offices and from other  federal agencies on an ad hoc basis. FDA told us that because the  agency did not renew the contract, the 2015 version omitted references to  obtaining and using open source information. ORA officials told us they  now use a variety of informal and situation-dependent methods to obtain  open source intelligence, but these methods are not formally  documented. We have previously found that by using informal  coordination mechanisms, agencies may rely on relationships with  individual officials to ensure effective collaboration and that these informal  relationships could end once personnel move to their next assignments.  Under federal standards for internal control, agencies are to clearly  document internal control, and the documentation is to appear in  management directives, administrative policies, or operating manuals.  Without a documented process, FDA does not have reasonable  assurance that it is consistently identifying the type of open source data to  collect, obtaining such data, and determining how PREDICT is to use the  data in a regular and systematic manner. A senior ORA official in its  Division of Import Operations told us that formalizing and documenting  the process by which FDA receives information from other entities would  improve the current process for using open source information.
section_title: PREDICT Analyzes Data by Applying Rules That Contribute to Risk-Based Scores and Recommended Actions
paragraphs: FDA creates rules, conditional statements that tell PREDICT how to react  when encountering particular situations. For example, if a certain  PREDICT rule encounters the term “tamarind” in the description of a  product, it should recommend detaining the item without physical  examination. This rule was created as the result of an incident in which  filth was found in tamarind products. PREDICT’s application of rules  generates a cumulative risk score for each entry line; the cumulative risk  score includes factors such as the product’s inherent risk and the  manufacturing firm’s facility inspection history. The risk score can indicate  a negative risk value (signifying low risk) or a positive risk value  (signifying higher risk). The individual risk values assigned to the various  elements of the entry line help produce the cumulative risk score for the  entire entry line and possibly also one or more recommended actions.
The application of a rule may prompt a flag (for example, an import alert  flag), which may provide information—such as that a manufacturer’s  product was recently examined or that the product was refused admission  by one or more foreign countries—and recommended an action. For  example, it may point out that some of the entry line data are missing or  invalid—based on a comparison with other FDA databases—and  recommend performing a manual review of databases.
After PREDICT has applied the rules to the entry data, entry reviewers  analyze the results. The first thing an entry reviewer sees is the main  entry review screen, which lists all the current entry lines to be reviewed  and their PREDICT risk scores and flags. The entry reviewer can click on  a PREDICT risk score to see a table, referred to as the PREDICT  Mashup. The PREDICT Mashup displays the individual risk values that  contributed to the overall PREDICT risk score for that entry line. For  example, the PREDICT Mashup may show that PREDICT assigned the  product’s manufacturer a risk-based score of 10 for its facility inspection  history, which would signify a high risk. The same Mashup may show  that PREDICT assigned the product’s country of origin a risk-based score  of 3 for its history, which would signify a lower risk. The entry reviewer  can also “roll over,” or move the cursor over, flags to obtain more  information. For example, an entry reviewer may roll over a “recent  targeted activity” flag—which indicates that this product from this  manufacturer was recently examined—to obtain information (sometimes  through another database) about when the product was examined and  the results of the examination.
The entry reviewer uses all of this information to decide how to proceed.  For example, the entry reviewer may decide to allow the entry line to  proceed into commerce or to hold it for examination or laboratory  analysis. Any action taken by the entry reviewer, including the results of  any examination, will become part of the history of the facility, which will  then contribute to future PREDICT scores.
section_title: The Implementation of FSMA Will Provide PREDICT with Additional Data on Imported Food
paragraphs: FSMA, which was enacted in 2011, contains provisions that will provide  PREDICT with additional data to analyze (i.e., from additional sources)  when estimating the risk of imported food. FDA officials told us that even  with the changes brought about by FSMA, PREDICT will continue to be  used to target imports for examination but will now draw data from even  more sources. FDA officials identified five FSMA provisions and  authorities as likely to generate the most data for use in PREDICT. Of  these five, the first three are as follows:
Foreign Supplier Verification Program (FSVP): The FSVP rule  requires importers to verify that their foreign suppliers use processes  and procedures that provide the same level of public health protection  as the hazard analysis and risk-based preventive controls and other  applicable requirements of FDCA. For example, importers must  develop, maintain, and follow an FSVP that provides adequate  assurances that the foreign supplier is producing food that is, among  other things, not adulterated (e.g., that it does not contain Salmonella)  and is not misbranded with respect to labeling for the presence of  major food allergens. Verification activities could include monitoring  records, annual on-site inspections or review of such inspections  performed by a qualified auditor, and testing and sampling of  shipments. The date by which importers must comply with the FSVP  regulations depends on a number of factors.
Inspection of foreign food facilities: FSMA includes a provision that  authorizes FDA to direct resources according to the known safety  risks of facilities, especially those that present a high risk. The law  directs FDA to inspect at least 600 foreign facilities within 1 year of  enactment of FSMA and, in each of the 5 years following that period,  to inspect at least twice the number it inspected during the previous  year.
Laboratory accreditation: FSMA also includes a provision that  requires FDA to establish a program for the testing of food by  accredited laboratories and to recognize accreditation bodies for  accrediting the laboratories, including independent private  laboratories. The provision requires owners or consignees to have  food products tested by an accredited laboratory in certain  circumstances, including when a food product is under an import alert  that requires successful consecutive tests. The results of these tests  may be submitted to FDA electronically and used to determine  compliance and admissibility of the food product. FDA has not yet  issued a proposed rule for establishing the program, but an agency  official stated that the agency is working on a rule now.
The other two FSMA provisions likely to generate data for use in  PREDICT are related to third-party certification, for which FDA issued the  final rule in November 2015. FSMA directs the establishment of a system  for the recognition of accreditation bodies to accredit third-party  certification bodies (also known as auditors) to certify that a foreign  facility, or any product produced by the facility, meets the applicable FDA  food safety requirements. If an accredited third-party certification body  or its audit agent discovers a condition that could cause or contribute to a  serious risk to public health, the certification body must immediately notify  FDA. The following two FSMA provisions rely on third-party  certifications and will likely generate data for use in PREDICT:
Voluntary Qualified Importer Program (VQIP): FSMA requires FDA  to establish a program that offers expedited review and entry to  certain participating importers that import food from foreign facilities  certified by accredited third-party certification bodies. FDA published  draft guidance for VQIP in the Federal Register in June 2015.  According to an FDA Fact Sheet on VQIP, the agency expects to  begin receiving applications for the program in January 2018.
Import certifications: Under FSMA, FDA has the authority, under  certain circumstances, to require certifications or other assurances  from agencies, or representatives of foreign governments where the  food originated, or an accredited third-party certification body. FDA  can determine that an article of food requires certification based on,  among other factors, the known safety risks associated with the food;  the country, territory, or region of origin of the food; or evidence that  the food safety systems for that product are inadequate. FDA  published the final rule on accreditation of third-party certification  bodies in the November 27, 2015, Federal Register.
Because some of these FSMA-related programs are still new and not yet  fully implemented, the details of how certain data generated by these  programs will be integrated into PREDICT are not finalized, but according  to FDA officials, PREDICT will use data produced by these programs and  authorities in a variety of ways. Implementation of FSVP, for example, will  identify importers that are not in compliance, and PREDICT would use  that information in assigning risk scores to products from those importers.  The food products from suppliers and importers that have a history of  noncompliance with food safety regulations will receive higher PREDICT  risk scores than those that are consistently in compliance. In addition, if  the results from an inspection of a foreign food facility show that the  facility is noncompliant, that information would be provided to PREDICT,  and PREDICT would use that information in assigning risk scores to  imports from that facility. Figure 5 shows how the implementation of  FSMA will add to the data sources already used by PREDICT.
section_title: FDA Assessed the Effectiveness of PREDICT in 2013, but Some Recommended Improvements Have Yet to Be Implemented
paragraphs: FDA has assessed the effectiveness of PREDICT by ongoing monitoring  of key program data and by conducting an evaluation of the tool in 2013,  and it has implemented many, but not all, of the 2013 evaluation’s  recommendations to improve PREDICT. Data maintained by agency  officials show that PREDICT is working properly to focus reviewers’  attention on food items that have a high probability of being violative.  Moreover, when FDA officials conducted an evaluation of the system in  2013, results showed that PREDICT was working well to provide staff  with the data they need to make informed entry review decisions.  However, the agency has not yet implemented all of the 24  recommendations resulting from that evaluation.
subsections: section_title: FDA’s Ongoing Monitoring Shows That PREDICT Is Effectively Targeting Higher-Risk Items for Examination
paragraphs: FDA officials told us that they conduct monitoring of PREDICT on an  ongoing basis and that the data they collect as part of this monitoring  show that PREDICT is working as intended—that is, PREDICT is  focusing entry reviewers’ attention on items determined to be of higher  risk. Data provided by FDA from fiscal years 2012 to 2014 confirmed that  in general, PREDICT is fulfilling this role. Our analysis of these data  showed that in general, the higher the PREDICT risk score, the more  often entry lines were examined and the more often they were found  violative. As tables 1, 2, and 3 show, for fiscal years 2012 through 2014,  entry lines that received higher PREDICT scores generally were more  often selected for a field examination, for a label examination, or for  sampling, and entry lines with higher PREDICT scores were more often  found violative.
In all 3 years we reviewed, items with the highest PREDICT scores— those from 91 to 100—were selected for examination more often than  items with lower PREDICT scores. For example, in fiscal year 2012, FDA  examined or sampled more than 39,000 entry lines with a PREDICT  score from 91 to 100, or about 6.5 percent of all entry lines in this range,  nearly double the percentage of entry lines examined in the next-highest  10 point risk score range and at least 6 times more than entry lines with a  score of 50 or less.
The tables also show that in each of the 3 years we reviewed, the higher  the PREDICT score, the higher the percentage of entry lines that were  found to be violative. For example, in fiscal year 2014, FDA selected for  examination about 45,000 of the nearly 1.3 million entry lines with  PREDICT scores of 91 to 100. Of these, nearly 12 percent, or 5,363 line  items, were found to be violative, almost double the percentage of  violation rates found in the next-highest range of PREDICT scores.
section_title: FDA Conducted an Internal Evaluation of PREDICT and Developed 24 Recommendations to Improve the Tool
paragraphs: In May 2013, FDA completed an internal evaluation of PREDICT’s  effectiveness that examined five processes: work planning, examinations  and sampling, entry review, rules management, and communication. To  conduct this evaluation, FDA officials gathered information through a  variety of methods, including analysis of entry line data and the entry  review process, interviews with stakeholders from the various FDA  centers, and surveys of PREDICT users in the field. Overall, the  evaluation showed that PREDICT is helping to expedite the release of  low-risk items and to identify violative imports.
As a result of the internal evaluation, FDA developed a total of 24  recommendations: 23 for improving the five processes noted above and  another recommendation concerning the development of performance  metrics. The list below shows the categories of recommendations and the  number of recommendations in each category: work planning (4), examination and sampling (4), entry review (6), performance metrics (1).  rules management (5),  communication (4), and  FDA prioritized each of its 24 recommendations according to the  feasibility of near-term implementation and the impact on operations or  public health, among other things. A recommendation was determined to  have high feasibility if resources were available to implement it and the  implementation plan was clear. A recommendation was determined to  have high impact if, for example, many stakeholders cited the problem  that the recommendation was intended to solve. As figure 6 shows, of the  24 recommendations, 6 were determined to be highly feasible, and 16  were determined to be of high impact or medium/high impact; 2  recommendations were determined to be both highly feasible and of high  impact.
Examples of FDA’s recommendations and how they were prioritized  include analyze and set specific “May Proceed” thresholds by commodity  (designated as both highly feasible and of high impact), develop a system to test PREDICT rules (designated as highly  feasible and of medium impact), and identify sources of delays associated with the sampling process  (designated as of medium feasibility and medium impact).
section_title: FDA Has Implemented Many, but Not All, of Its Recommendations
paragraphs: FDA has evaluated the 24 recommendations and has implemented many,  but not all, of them. According to agency officials, the agency has  implemented 15, partially implemented another 6, and not implemented 3  (see fig. 7).
Of the 15 recommendations that were implemented, 13 were designated  either as highly feasible or of high or medium/high impact. Both of the  recommendations determined to be both highly feasible and of high  impact were implemented, including the recommendation to analyze and  set specific “May Proceed” thresholds by commodity. Of the 6 that were  partially implemented—which includes the recommendation to develop a  system to test PREDICT rules—5 were designated as either highly  feasible or of high or medium/high impact. Finally, of the 3 that were not  implemented—which includes the recommendation to identify sources of  delays associated with the sampling process—1 was of low feasibility and  high impact, 1 was of medium feasibility and high impact, and 1 was of  medium feasibility and medium impact.
Federal standards for internal control state that agencies are to ensure  that the findings of audits and other reviews are promptly resolved. To  that end, agencies are to complete, within established time frames, all  actions that correct or otherwise resolve the matters brought to  management’s attention. According to FDA officials, the reason for  partially implementing or not implementing recommendations was often a  lack of resources. For example, FDA officials said that a lack of resources  was why they only partially implemented the recommendation to develop  a system to test PREDICT rules and did not implement the  recommendation to identify sources of delays associated with the  sampling process. Agency officials agree that implementing the remaining  recommendations would improve the effectiveness of PREDICT, but the  agency has not yet established time frames for completing the  implementation of the remaining recommendations. Establishing a  practical timeline for implementing the remaining recommendations as  resources become available would provide FDA with reasonable  assurance that the improvements are made and that PREDICT remains  an effective tool for screening imports.
section_title: Conclusions
paragraphs: The volume of FDA-regulated imported food continues to grow, as does  the need to ensure that such imports are safe. FDA physically examines  about 1 percent of food shipment entry lines annually. The agency  developed PREDICT to help target food shipments deemed higher risk  and subject them to additional scrutiny. FDA’s assessment of PREDICT  shows that the tool is generally working to focus FDA’s resources on the  examination of food items determined to be of highest risk and expediting  the release of lower-risk food items.
PREDICT analyzes imported food entry lines using data from various  sources, including FDA sources, other domestic sources, and foreign  sources. Some of the domestic and foreign sources are open sources,  and FDA uses such sources to obtain information about imported foods  on an ad hoc basis. However, FDA does not have a documented process  for identifying the type of open source data to collect, obtaining such data,  and determining how PREDICT is to use the data. Without such a  documented process, FDA does not have reasonable assurance that it  will consistently obtain open source data for PREDICT in a regular and  systematic manner.
In addition, FDA’s May 2013 evaluation of PREDICT identified 24  recommendations to improve the tool, and FDA has fully implemented 15  of these recommendations. FDA officials explained that resource  constraints have often limited their ability to fully implement all the  remaining recommendations. Agency officials agree that implementing  these recommendations would improve the effectiveness of PREDICT,  but the agency has not yet established time frames for doing so.  Establishing a practical timeline for implementing the remaining  recommendations would help FDA ensure that the improvements are  made and that PREDICT remains an effective tool for helping to prevent  high-risk foods from entering the United States.
section_title: Recommendations for Executive Action
paragraphs: To further enhance FDA’s PREDICT tool and its ability to ensure the  safety of imported food, we recommend that the Secretary of Health and  Human Services direct the Commissioner of FDA to take the following  two actions:    document the process for identifying the type of open source data  to collect, obtaining such data, and determining how PREDICT is  to use the data and    establish a timeline for implementing, as resources become  available, the remaining recommendations from FDA’s 2013  evaluation of PREDICT.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided the departments of Commerce, Health and Human Services,  and Homeland Security a draft of this report for their review and  comment. In an e-mail, the Department of Commerce stated that it had no  comments. The Department of Health and Human Services generally  concurred with the recommendations and provided written comments on  the draft, which are summarized below and presented in their entirety in  appendix II of this report. The Department of Homeland Security provided  technical comments, which we incorporated as appropriate.
In its written comments, the Department of Health and Human Services  concurred with our first recommendation and concurred in part with the  second. For our first recommendation, HHS stated that ORA plans to  work with appropriate units across the agency to develop and document a  formal process to identify the type of information to collect, how to obtain  the information, and how PREDICT may use it.  For our second  recommendation, HHS stated that it has recently fully implemented one of  the internal recommendations from the 2013 internal evaluation that was  previously designated as “partially implemented.” However, because HHS  did not provide evidence of fully implementing the recommendation, we  were unable to verify such implementation and made no change to the  report. The Department also stated that it is supportive of the remaining  recommendations from its 2013 internal evaluation but noted that  establishing an implementation timeline is based on consideration of the  feasibility and impact relative to available FDA resources and other FDA  priorities, such as FSMA. The department indicated that the goals of the  unimplemented recommendations may ultimately be achieved via these  other FDA initiatives.
We are sending copies of this report to the appropriate congressional  committees, the Secretary of Commerce, the Secretary of Health and  Human Services, the Secretary of Homeland Security, and other  interested parties.  In addition, the report is available at no charge on the  GAO website at http://www.gao.gov.
If you or your staff have any questions about this report, please contact  me at (202) 512-3841 or morriss@gao.gov. Contact points for our Offices  of Congressional Relations and Public Affairs may be found on the last  page of this report. GAO staff members who made key contributions to  this report are listed in appendix III.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: You asked us to examine how the Food and Drug Administration (FDA) is  using Predictive Risk-based Evaluation for Dynamic Import Compliance  Targeting (PREDICT) to protect the safety of the U.S. food supply. Our  objectives in this report were to examine (1) the data used by PREDICT  and how PREDICT analyzes these data to identify high-risk food  shipments for examination; (2) how implementation of the FDA Food  Safety Modernization Act (FSMA) will affect PREDICT; and (3) the extent  to which FDA has assessed the effectiveness of PREDICT and used the  results of those assessments to improve the tool.
To determine the data used by PREDICT and how PREDICT analyzes  these data to identify high-risk food shipments for examination, we  reviewed documents that described the data used by PREDICT and  interviewed FDA officials. We assessed the data sources, data quality  controls, and the methodology used by PREDICT and determined that  they were sufficiently reliable to support accurate descriptions of the  system inputs, processes, and resulting risk scores. We also assessed  the data used by PREDICT to ensure that they were sufficiently reliable to  describe factors considered by PREDICT and system rules used to  generate risk scores. We interviewed FDA officials in the Office of  International Programs to understand how FDA coordinates with foreign  governments to obtain data for PREDICT. We also reviewed how the  agency identifies, obtains, and uses open source information and  compared this process to criteria for documentation found in the federal  standards for internal control. To determine the extent to which the  implementation of FSMA will affect PREDICT, we reviewed FSMA, FDA’s  rules implementing FSMA, and FDA planning documents and interviewed  FDA officials. To determine the extent to which FDA has assessed the  effectiveness of PREDICT and used the results of those assessments to  improve the tool, we interviewed FDA officials about ongoing monitoring  of PREDICT, reviewed the 2013 evaluation that FDA has conducted on  PREDICT, and assessed the extent to which FDA has implemented the  recommendations from the evaluation. We then compared FDA’s actions  to the criteria found in the federal standards for internal control regarding  the findings of audits and other reviews. In addition, we examined data  provided by FDA on PREDICT risk scores and violation rates from fiscal  years 2012 through 2014, the most recent years for which data were  available, and determined that the data were sufficiently reliable for our  purposes, as discussed above.
To inform all three objectives, we conducted site visits at four FDA and  U.S. Customs and Border Protection (CBP) facilities located at ports of  entry located in Baltimore, Maryland; Otay Mesa in San Diego, California;  Long Beach California; and Los Angeles, California. We selected these  sites to include air, ship, and truck ports of entry; variable geographic  locations; the variability of food products that enter through the ports; and  proximity to an FDA office. The information we obtained at these sites is  not generalizable to all facilities and ports of entry. In addition, we  interviewed officials from CBP and the National Marine Fisheries Service  to understand how FDA coordinates with other federal agencies to protect  the safety of the U.S. food supply. We also interviewed officials from a  number of stakeholder organizations that are affected by PREDICT or  contributed to PREDICT’s development.
We conducted this performance audit from January 2015 to May 2016 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Appendix II: Comments from the Department of Health and Human Services
section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Anne K. Johnson (Assistant  Director), Kevin Bray, Ellen Fried, Steven Putansu, Gloria Ross, Stuart  Ryba, Sara Sullivan, Vasiliki Theodoropoulos, and Karen Villafana made  key contributions to this report.