Food Safety: More Disclosure and Data Needed to Clarify Impact of Changes to Poultry and Hog Inspections
Published: 2013-08-22T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: USDA inspectors provide continuous inspection of each meat and poultry carcass and its parts that enter interstate commerce. In 1998, USDA began three pilot projects at slaughter plants for healthy young chickens, young turkeys, and young hogs, with a purpose to deploy inspection resources more effectively in accordance with food safety and other consumer protection requirements. Under the pilot projects, plant personnel sort carcasses before USDA's inspection. The pilot projects are to end when a final rule for each species is published. In January 2012, USDA published a proposed rule to modernize poultry slaughter inspections based, in part, on its pilot projects. GAO was asked to review these pilot projects. This report determines (1) the extent to which USDA has evaluated the three pilot projects, (2) strengths and weaknesses of the pilot projects based on the views of key stakeholder groups, and (3) the extent to which USDA disclosed limitations, if any, in sources of information it relied on to develop the proposed rule. GAO reviewed relevant laws and documents and interviewed USDA officials and 11 key industry, labor, consumer advocacy, and animal welfare groups familiar with the pilot projects.
section_title: What GAO Found
paragraphs: The U.S. Department of Agriculture (USDA) has not thoroughly evaluated the performance of each of the pilot projects over time even though the agency stated it would do so when it announced the pilot projects. For example, in 2011, USDA completed a report evaluating the pilot project at 20 young chicken plants concluding that an inspection system based on the pilot project would ensure equivalent, if not better, levels of food safety and quality than currently provided at plants not in the pilot project. However, among the limitations of its evaluation was the use of snapshots of data for two 2-year periods instead of data for the duration of the pilot project, which has been ongoing for more than a decade. In addition, USDA did not complete an evaluation on or prepare a report evaluating the pilot project at 5 young turkey plants and has no plans to do so because of the small sample size. Nevertheless, in publishing a proposed rule that includes an optional new poultry (chicken and turkey) inspection system, USDA stated that the new system was based on its experience with the pilot projects at young chicken and young turkey plants. As a result, USDA may not have assurance that its evaluation of the pilot project at young chicken plants provides the information necessary to support the proposed rule for both chickens and turkeys. However, the agency will not complete another evaluation before it issues a final rule. USDA has begun drafting a preliminary report evaluating the pilot project at young hog plants using analyses similar to those presented in the report evaluating young chicken plants, suggesting that similar limitations may apply. Agency officials stated that when USDA develops a proposed rulemaking to modify its slaughter inspection system for hogs, the agency will need to decide whether to collect additional data. Without collecting and analyzing additional data, it will be difficult for USDA to draw conclusions about whether the pilot project at young hog plants is meeting its purpose. While the pilot project is ongoing, USDA has the opportunity to collect and analyze additional information.
GAO identified strengths and weaknesses of the three pilot projects based on the views cited most frequently by 11 key stakeholder groups representing industry, labor, consumer advocacy, and animal welfare. On the basis of these views, GAO identified strengths including giving plants responsibility and flexibility for ensuring food safety and quality and allowing USDA inspectors to focus more on food safety activities. GAO identified weaknesses including that training of plant personnel assuming sorting responsibilities on the slaughter line is not required or standardized and that faster line speeds allowed under the pilot projects raise concerns about food safety and worker safety.
USDA did not disclose certain limitations in sources of information it relied on to develop the cost-benefit analysis supporting the proposed rule on modernizing poultry slaughter inspections. GAO identified three sources of information with certain limitations that were not disclosed. For example, USDA did not disclose that it gathered no cost information from young turkey plants in the pilot project. Furthermore, USDA generalized the results from 12 young chicken plants in the pilot project that responded to a 2001 cost survey to the universe of 335 young chicken and young turkey plants in the United States in 2012. As a result, stakeholders did not have complete and accurate information to inform their comments on the proposed rule and its potential impacts.
section_title: What GAO Recommends
paragraphs: GAO recommends that USDA (1) collect and analyze information to determine if the young hog pilot project is meeting its purpose and (2) clearly disclose to the public limitations in the information it relied on for the proposed rule to modernize poultry slaughter inspections. USDA concurred with GAO's recommendations.

section_title: Letter
paragraphs: According to the Centers for Disease Control and Prevention (CDC), the  U.S. food supply remains one of the safest in the world; nevertheless,  CDC estimates that each year about 48 million Americans become sick  from foodborne diseases. Meat and poultry products contaminated with  pathogens such as Salmonella, Toxoplasma gondii, Campylobacter, and  Lysteria monocytogenes cause many foodborne illnesses and deaths. To  control the spread of such foodborne illnesses, over 3,700 U.S.  Department of Agriculture (USDA) veterinarians and inspectors work at  more than 900 slaughter plants. USDA inspectors provide continuous  inspection of each meat and poultry carcass and its parts that enter  interstate commerce. USDA inspectors positioned on slaughter lines in  these plants are responsible for sorting carcasses to identify defects and  directing plant personnel to remove unacceptable carcasses, among  other things. There may be multiple USDA inspectors positioned on each  slaughter line, depending on the speed at which meat and poultry  carcasses move down the line. For example, at plants that slaughter  young chickens, there may be up to four inspectors on each line, and  each inspector is responsible for inspecting up to 35 carcasses per  minute. In addition, USDA inspectors positioned off the slaughter line  move freely about the plant and collect samples of carcasses to test for  microbial pathogens (e.g., Salmonella); perform food safety checks, such  as verifying that carcasses are free of fecal material; and ensure that  carcasses comply with the agency’s food quality standards for things  such as bruises on chickens, which does not affect food safety.
In 1997, USDA announced the need to modify its meat and poultry  slaughter inspection program to, among other things, make industry more  responsible for identifying carcass defects. In 1998, USDA began three  pilot projects—one for each species—at slaughter plants to test changes  to its inspection systems from having its inspectors provide continuous  inspection of each and every carcass for healthy young chickens, young  turkeys, and young hogs to a system that uses plant personnel to  examine each carcass for safety and quality and USDA inspectors to  verify that safety and quality standards are met. According to a Federal  Register notice announcing the pilot projects, the purpose was to deploy  inspection resources more effectively in accordance with food safety and  other consumer protection requirements. In announcing the pilot  projects, USDA stated that the agency would thoroughly evaluate the pilot  projects and, at the time of its evaluation, determine whether further  testing should be conducted or whether to initiate rulemakings to adopt  and implement new inspection systems based on the pilot projects. As of  July 2013, 29 plants were participating in the pilot projects—19 plants for  slaughtering young chickens, 5 for young turkeys, and 5 for young hogs.  At participating plants under the pilot projects, plant personnel have taken  on the sorting responsibilities of USDA inspectors positioned on the  slaughter line for identifying carcass defects and removing unacceptable  carcasses. However, a USDA inspector remains at the end of each  slaughter line at each young chicken and young turkey plant and at three  fixed locations on the slaughter line at each young hog plant to conduct a  carcass-by-carcass inspection after plant personnel have completed  sorting activities so that only those carcasses deemed likely to pass  inspection remain on the line. According to a Federal Register notice,  removing USDA inspectors from slaughter lines is intended to free the  inspectors to perform additional food safety and quality checks in the  areas of greatest risk, as needed, throughout the plant.
In December 2001, we reported on the design and methodology of  USDA’s pilot projects and noted several limitations that do not allow  results from plants participating in USDA’s pilot projects to be generalized  to the universe of plants. participating in the young chicken pilot project were not randomly selected  and that they did not include plants from all chicken producing areas or  plants of all sizes. Thus, the results cannot be generalized to the entire  population of chicken slaughter plants in the United States.
For example, we reported that plants  In January 2012, USDA published in the Federal Register a proposed rule  to modernize poultry slaughter inspections that affects all plants that  slaughter poultry (chickens and turkeys) and that includes an optional  new inspection system for all young chicken and turkey plants, which  resembles some aspects of the inspection systems used in the pilot  projects at young chicken and young turkey plants. During the public comment phase of the proposed rule for modernizing poultry slaughter  inspections, USDA received more than 175,000 comments from  stakeholders, including industry and consumer advocacy groups, as well  as individual members of the public. As of July 2013, USDA was working  to complete its final rule for modernizing poultry slaughter inspections,  according to agency officials.
GAO, Food Safety: Weaknesses in Meat and Poultry Inspection Pilot Should Be  Addressed before Implementation, GAO-02-59 (Washington, D.C.: Dec. 17, 2001).
This report responds to your request that we review USDA’s pilot projects  at slaughter plants for young chickens, young turkeys, and young hogs.  Our objectives were to determine (1) the extent to which USDA has  evaluated the three pilot projects, (2) strengths and weaknesses of the  three pilot projects based on the views of key stakeholder groups, and (3)  the extent to which USDA disclosed limitations, if any, in sources of  information it relied on to develop the proposed rule to modernize poultry  slaughter inspections.
To determine the extent to which USDA has evaluated the three pilot  projects, we reviewed relevant USDA documents, Federal Register  notices, and laws. We also compared USDA’s efforts to evaluate the pilot  projects with criteria based on social science and evaluation literature and  published GAO guidance identified in our previous work on pilot program  development and evaluation. In addition, we interviewed several officials  in various offices within USDA’s Food Safety and Inspection Service  (FSIS)—the agency responsible for USDA’s meat and poultry inspection  program. To determine the strengths and weaknesses of the three pilot  projects based on the views of key stakeholder groups, we identified 11  key stakeholder groups representing industry, labor (including plant  personnel and USDA inspectors and veterinarians), consumer advocacy,  and animal welfare with sufficient knowledge about USDA’s three pilot  projects that submitted comments on USDA’s proposed rule on  modernizing poultry slaughter inspections.to determine the extent to which the comments may apply to the pilot   We reviewed their comments  projects; interviewed representatives of these key stakeholder groups;  and followed up with e-mailed questions to gauge their level of familiarity  with each pilot project and then clarified responses, as needed. We  developed categories of strengths and weaknesses identified most  frequently by stakeholder groups and determined whether each  stakeholder group had identified strengths or weaknesses that fit into  these categories. To determine the extent to which USDA disclosed  limitations, if any, in sources of information it relied on to develop the  proposed rule, we reviewed the proposed rule and related Federal  Register notices, as well as selected documents the agency relied on to  develop the proposed rule. We also interviewed several officials in  various offices within USDA’s FSIS to clarify information, as needed.  Appendix I provides more information on our objectives, scope, and  methodology.
We conducted this performance audit from September 2012 to August  2013 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence presented provides a reasonable basis for our findings  and conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: The Federal Meat Inspection Act and the Poultry Products Inspection Act  give USDA overall responsibility for ensuring the safety and  wholesomeness of meat and poultry products that enter interstate  commerce. Acting under these legislative authorities, USDA inspectors  provide continuous government inspection of each and every meat and  poultry carcass and its parts at slaughter plants throughout the United  States. Within USDA, FSIS is responsible for inspections at all meat and  poultry slaughter and processing plants and has inspectors positioned  both on and off the slaughter line. FSIS inspectors positioned on the line  are to inspect every carcass and its parts, including the viscera (e.g.,  hearts and liver) organoleptically—by sight, touch, and smell—for defects,  and direct plant personnel to take appropriate corrective action when  defects are found. According to a Federal Register notice, plants rely on  FSIS inspectors to control and direct their production processes. For  example, FSIS regulates the speed of the slaughter line based on its  inspectors’ ability to perform proper inspection procedures. FSIS  inspectors positioned off the line move freely about the plant to focus on  areas of greatest risk and to perform collection of carcass samples for  testing of microbial pathogens. Among other activities, these inspectors  review plant records and live animals presented for slaughter. These  inspectors are also responsible for ensuring plants’ compliance with  regulatory requirements on a daily basis and for taking regulatory  enforcement action when deficiencies are found.
In 1997, FSIS announced the need to modify its meat and poultry  slaughter inspection program to, among other things, make industry more  responsible for identifying carcass defects. This approach is consistent  with the agency’s previous adoption of the Pathogen Reduction: Hazard  Analysis and Critical Control Point (HACCP) regulations. Under the risk- based HACCP approach, industry—rather than federal inspectors—is  responsible for identifying steps in food production where food safety  hazards, such as fecal material, are reasonably likely to occur and for  establishing controls that prevent or reduce these hazards. As part of this  approach, plants must identify the point (known as the critical control  point) where they will establish control for a food safety hazard and take  steps to prevent, eliminate, or control the hazard. FSIS had not extended  the HACCP principles to slaughter inspections because the agency  provides continuous inspection of each and every carcass. However,  FSIS believed that changing its existing inspection systems would also  reduce inspectors’ reliance on organoleptic inspections, allow for a shift to  prevention-oriented inspection systems based on risk, and permit  redeployment of its resources to better protect the public from foodborne  diseases. Before making a permanent change to its slaughter inspection  systems, FSIS developed the pilot projects in 1998 at young chicken,  young turkey, and young hog plants to test whether such a change would  achieve its intended purpose of deploying inspection resources more  effectively in accordance with food safety and other consumer protection  requirements. FSIS’ pilot projects at these plants are known as the  HACCP-based Inspection Models Project (HIMP).
FSIS developed food safety and quality performance standards for plants  in the pilot projects to meet. FSIS set the performance standard for food  safety defects at zero. An example of a food safety defect is a carcass  contaminated with fecal material because ingestion of meat and poultry  contaminated with fecal material poses potential harm to humans. FSIS’  performance standards for food quality defects vary, depending on the  animal species and the type of defects. An example of a food quality  defect is bruises on the carcass that, while not harmful if consumed,  affects the wholesomeness of meat and poultry products. According to  FSIS officials, the food quality performance standards are intended to be  more stringent than the performance standards in place at plants that are  not participating in the pilot projects.
In May 1999, FSIS negotiated an agreement with the union representing  its inspectors that limited the number of plants participating in the pilot  projects to 20 young chicken plants, 5 young turkey plants, and 5 young  hog plants. According to the agreement, the pilot projects for each  species are to end when a final rule is published for that species. Before  converting to the inspection systems under the pilot projects, participating  plants modified their operations to meet certain requirements, such as  installing a new workstation at the end of the slaughter line and before the  chiller for the FSIS inspector positioned on the slaughter line at poultry   Over the course of the pilot projects, some plants have dropped  plants. out, and others have joined. As of July 2013, 29 plants were participating  in the pilot projects in 18 states. Appendix II provides additional  information on the location of plants in the pilot projects and the volume of  poultry and hog slaughter in the United States.
The chiller is the point when eviscerated carcasses—carcasses that have had internal  organs and any processing defects removed—are chilled in order to inhibit microbial  growth and meet the regulatory requirements of 9 C.F.R. § 381.66(b)(1). young chicken and turkey plants. Plants participating in FSIS’ pilot  projects are required to participate in this program. The Salmonella  Initiative Program permits plants to operate with an exemption (known as  a waiver) from complying with certain regulatory requirements. For  example, young chicken plants in the pilot project are exempt from  meeting FSIS’ regulatory requirement that limits the slaughter line speed  at young chicken plants and can operate the slaughter line at a faster line  speed. Plants in the Salmonella Initiative Program must demonstrate to  FSIS that results from their Salmonella testing consistently demonstrate  that they maintain control over production processes. The goal of the  Salmonella Initiative Program is to reduce and eliminate Salmonella  before products reach consumers.
In January 2012, FSIS published in the Federal Register a proposed rule  to modernize poultry slaughter inspections based, in part, on the agency’s  experience with the pilot projects at young chicken and young turkey  plants. FSIS developed the proposed rule in response to an executive  order directing agencies to review existing regulations that may have  been outdated and modify them accordingly. According to the proposed  rule, the modernization is intended to improve food safety and the  effectiveness of poultry slaughter inspection systems, remove  unnecessary regulatory obstacles to innovation, and make better use of  the agency’s resources. The proposed rule further states that inspection  systems currently in place at chicken and turkey plants (not in the pilot  projects) are lacking in two important respects. First, the proper role of  industry and FSIS is obscured. Specifically, FSIS inspectors are currently  responsible for sorting acceptable carcasses from unacceptable  carcasses, finding defects, identifying corrective actions, and solving  problems in production control processes, but these are roles more  appropriately the responsibility of the slaughter plants. Second, a  significant amount of FSIS’ inspection program personnel resources are  allocated toward inspection activities to detect defects and conditions that  present minimal food safety risks. According to the proposed rule, this  allocation limits the agency resources available for food safety-related  inspection activities. Moreover, FSIS developed an economic cost and  benefit analysis to demonstrate the merits of the proposed rule, which,  according to USDA, is expected to have an annual impact on the  economy of more than $100 million.
FSIS’ proposed rule affects all poultry slaughter plants and includes  mandatory regulatory changes. For example, the proposed rule requires  all poultry plants to maintain written procedures to prevent contamination  of carcasses and parts by fecal material and pathogens (e.g., Salmonella  and Campylobacter) and to test for organisms (e.g., Salmonella) to  demonstrate control over their production processes at a point before the  carcass enters the chiller and at a point after the carcass exits the chiller.  In addition, the proposed rule includes the optional new poultry inspection  system for young chicken and turkey plants. According to the proposed  rule, FSIS expects that the majority of young chicken and turkey plants  will convert to the optional new poultry inspection system. The new  poultry inspection system resembles the inspection systems at young  chicken and young turkey plants in the pilot projects but also has some  differences. Similarities between the pilot projects and optional new  poultry inspection system include, among others:
For each slaughter line, there would be one FSIS inspector positioned  at the end of the line to perform a carcass-by-carcass inspection and  one FSIS inspector positioned off the line to perform, among other  things, food safety and quality checks on carcasses to verify that plant  personnel (known as sorters) have effectively performed their duties,  such as removing fecal material on carcasses.
The FSIS inspector positioned on the slaughter line would visually  inspect (observe) each carcass after the viscera are separated from it  and after plant personnel have sorted carcasses, at a point near the  end of the slaughter line.
A difference between the pilot projects and the optional new poultry  inspection system is that the new system would eliminate FSIS’  performance standards for food quality defects and replace them with a  requirement that plants maintain records documenting that their products  meet the regulatory definition of “ready-to-cook.” Ready-to-cook means  that the products are free of such defects as feathers, oil glands, and  diseases and are thus suitable for cooking without any further  preparation. According to FSIS officials, plants could use the existing food  quality standards—known as Finished Product Standards—to meet the  regulatory definition of ready-to-cook.
section_title: FSIS Has Not Thoroughly Evaluated the Three Pilot Projects
paragraphs: FSIS has not thoroughly evaluated the performance of each of the three  pilot projects over time even though the agency stated that it would do so  when it announced the pilot projects. Specifically, FSIS completed a  report evaluating the pilot project at young chicken plants, but its data  analyses have limitations. FSIS did not prepare a report evaluating the  pilot project at young turkey plants and has no plans to do so because  data from the five young turkey plants in the pilot project provide limited  information due to the small sample size. While FSIS has begun drafting  a preliminary report evaluating the pilot project at young hog plants, it  used analyses similar to those presented in the report evaluating the pilot  project at young chicken plants, suggesting similar limitations may apply.
subsections: section_title: FSIS Has Completed a Report Evaluating Some Data from the Chicken Pilot Project, but Its Data Analyses Have Limitations
paragraphs: In 2011, FSIS completed a report evaluating the pilot project at young  chicken plants; according to agency officials, the agency’s evaluation  efforts focused on this pilot project, in part, because it has the largest  number of plants participating. FSIS’ evaluation compared the  performance of the 20 young chicken plants in the pilot project (1) with a  similar group of 64 plants not participating in the pilot project, using  routinely collected data, and (2) against the performance standards  developed for the pilot project using data collected specifically for the pilot  project. FSIS’ evaluation concludes that an inspection system based on  the pilot project would ensure equivalent, if not better, levels of food  safety and quality than currently provided at plants not in the pilot project.  FSIS used this evaluation to support its January 2012 proposed rule  modernizing poultry slaughter inspections that includes the optional new  poultry inspection system.
We identified two limitations of FSIS’ evaluation that raise questions  about the validity of FSIS’ conclusion that an inspection system based on  the pilot project would ensure equivalent, if not better, levels of food  safety and quality than currently provided at plants not in the pilot project.  First, FSIS’ conclusion about the pilot project was based, in part, on  comparisons of data that were not designed to be comparable. For  example, FSIS concluded that the prevalence of Salmonella at the 20  plants participating in the pilot project was significantly lower than in the  similar group of 64 plants that were not participating in the pilot project  from 2006 to 2008. However, it based its conclusion on data that were  collected as a part of its microbial sampling program, rather than  collecting samples from the same plants for each year. The total number  of samples collected in each year declined from one year to the next— indicating that the total number of plants from which samples was  collected varied from year to year (see fig. 1). FSIS officials confirmed  that the total number of plants from which samples were collected varied  from year to year, and these officials were unable to tell us how many  plants were included in each year of the analysis. In addition, the  possibility that lower prevalence of Salmonella was caused by something  other than the pilot project cannot be ruled out.
Moreover, data from the last 2 years analyzed did not show a significantly  lower prevalence of Salmonella for plants participating in the pilot project.  According to FSIS officials, FSIS did not collect data to demonstrate the  relative effectiveness of plants participating and not participating in any of  the pilot projects. Instead, the agency analyzed data for a variety of  inspection activities performed in all plants (regardless of a plant’s  inspection system) to ensure their compliance with regulatory  requirements.
The second limitation that we identified in FSIS’ evaluation is that the  agency collected more than a decade’s worth of data on the extent to  which young chicken plants in the pilot project were meeting the food  safety and quality performance standards developed for the pilot project,  but it based its conclusion about the performance of the pilot project on  the use of snapshots of data from the pilot project for two 2-year periods.  Moreover, the time frames for the snapshots differed depending on  whether the categories related to food safety or food quality defects. More  specifically, the agency included snapshots of food safety data from 2000  to 2002—when the project first began—and food safety data from April  2009 to March 2011, and snapshots of food quality data from 2000 to  2002 with food quality data from 2009 to 2010—the most recent data  available at that time. Consequently, the results from the years that FSIS  selected for analysis may not be indicative of plants’ performance over  time. According to FSIS officials, the agency did not analyze the data for  the majority of the years because the data were recorded on paper forms  stored at individual plants and compiling the data for analysis was labor  intensive. According to the Standards for Internal Control in the Federal  Government, federal agencies are to employ internal control activities,  such as top-level review, to help ensure that management’s directives are  carried out and to determine if agencies are effectively and efficiently  using resources. Without analyzing data for the majority of the years of  the pilot project in its evaluation, FSIS could not determine whether an  inspection system based on the pilot project would ensure equivalent, if  not better, levels of food safety and quality than currently provided at  plants not in the pilot project over time. In addition, according to the  Federal Register notice announcing the pilot projects, FSIS stated that  the agency would thoroughly evaluate them; however, using snapshots of  data rather than data for the majority of the years of the pilot project, the  agency did not conduct a thorough evaluation. Instead, FSIS  veterinarians and inspectors evaluated the performance of their individual  plants against regulatory performance standards on a daily basis, as they  routinely do for all plants regardless of whether they are in the pilot  project or not, as well as the performance standards for the pilot project.  However, the agency has not aggregated and analyzed these daily  results to determine how the plants participating in the pilot project have  performed over time.
Notwithstanding these limitations, FSIS used its evaluation of the pilot  project at young chicken plants to support the proposed rule on  modernizing poultry slaughter inspections. Moreover, the design and  methodology limitations we identified in our 2001 report (e.g., young  chicken plants participating in the pilot project constituted a small,  nonrandom sample) continue to prevent the results obtained from the 20  participating young chicken plants from being generalized to the 239  young chicken plants and 96 young turkey plants in the United States in  2012. As a result, FSIS may not have assurance that its evaluation of the  pilot project at young chicken plants provides the information necessary  to support the proposed rule for poultry—both chickens and turkeys.  However, according to the Federal Register notice supporting the  proposed rule on modernizing poultry slaughter inspections, the agency  conducted a comprehensive evaluation; thus, the agency will not  complete another evaluation before it issues a final rule.
section_title: FSIS Did Not Prepare a Report Evaluating the Turkey Pilot Project
paragraphs: Unlike what FSIS did for the pilot project at young chicken plants, it did  not complete an evaluation on or prepare a report evaluating the pilot  project at young turkey plants. In publishing the proposed rule  modernizing poultry slaughter inspections that included an optional new  poultry inspection system, the agency stated that the new system was  based on its experience with the pilot projects at young chicken and  young turkey plants. According to FSIS officials, the agency did not  prepare a report evaluating the pilot project at young turkey plants and  has no plans to do so because data from five young turkey plants in the  pilot project provide limited information due to the small sample size.
Instead, as part of a quantitative microbial risk assessment to estimate  the public health impact of the proposed rule, the agency analyzed the  relationship between the presence or absence of pathogens and the  frequency with which FSIS inspectors positioned off the slaughter line  carried out specific inspection activities at young turkey plants.  According to FSIS’ analysis, there is a suggested relationship between  young turkey plants’ participation in the pilot project and a lower  prevalence of Salmonella and Campylobacter.
In addition, FSIS officials stated that, for the optional new poultry  inspection system, they generalized information and data from young  chicken plants in the pilot project to the young turkey plants in the pilot  project because the processes to slaughter chickens and turkeys are  similar. However, we identified differences in the food quality performance  standards and FSIS’ testing protocol for pathogens. FSIS acknowledged  that the food quality performance standards are not the same. We  determined that the five food quality performance standards developed  for the young chicken and for the young turkey pilot project can differ by  0.5 percent to almost 16 percent. For example, the performance standard  for the food quality defect category that includes animal diseases (such  as arthritis) is 1.7 percent for young chickens and 1.2 percent for young  turkeys—a difference of 0.5 percent. In another example, the  performance standard for the food quality defect that includes other  defects such as feathers is 80.0 percent for young chickens and 95.9  percent for young turkeys—a difference of 15.9 percent.
Furthermore, FSIS’ protocols differ for testing chicken and turkey  carcasses for Salmonella and Campylobacter. For example, for chickens,  FSIS inspectors rinse an entire chicken carcass in a bag filled with a  sterile water solution that is poured off and tested for these pathogens. In  contrast, for turkeys, FSIS inspectors use a sponge—containing the same  sterile water solution as that used to rinse chicken carcasses—to swab  certain areas of the turkey carcass and then test the sponge for  pathogens. According to a March 2011 Federal Register notice, FSIS  acknowledged that the method used to sample carcasses affects the  results and stated that they are not proposing to compare microbial data  from these two species. However, even with these differences in  sampling protocols, FSIS officials stated that the results for young  chickens could be generalized to young turkeys. These differences raise  questions about the extent to which FSIS can generalize results for food  quality defects and microbial testing results from one species to the other,  but, as we previously mentioned, the agency has no plans to do an  evaluation of the pilot project at young turkey plants before it issues a  final rule.
section_title: FSIS Has Not Completed Its Report Evaluating the Hog Pilot Project
paragraphs: In 2011, FSIS began drafting a preliminary report evaluating the pilot  project at young hog plants. The preliminary report uses analyses similar  to those presented in the report evaluating the pilot project at young  chicken plants, suggesting that similar limitations may apply. In particular,  FSIS did not collect comparable data from plants participating and not  participating in the pilot project. In addition, like the turkey pilot project,  information collected from the five young hog plants in the pilot project  would not provide reasonable assurance that any conclusions can apply  more broadly to the universe of 608 hog plants in the United States in  2012 because of the small sample size. FSIS officials agreed that there  would be concerns regarding the strength of any conclusions based on  five plants. These officials stated that, when the agency develops a  proposed rulemaking to modify its slaughter inspection system for hogs, it  will need to decide whether to collect additional data. Furthermore, a May  2013 USDA Office of Inspector General report identified areas of risk in  FSIS’ inspection of hog plants, including those participating in the pilot  project. The report found that FSIS did not critically assess whether the  pilot project had measurably improved food safety at each participating  plant because the agency did not adequately oversee the program. In  response, FSIS stated that it would complete an evaluation that would  include an analysis of plants participating and not participating in the pilot  project. However, as we previously stated, the analyses that the agency  plans to use in this evaluation are similar to those presented in the report  evaluating the pilot project at young chicken plants, suggesting that  similar limitations may apply. According to FSIS officials, the agency  intends to complete this evaluation by March 31, 2014. The officials said  that the agency could use its final report evaluating the pilot project at  young hog plants to support a rulemaking but currently has no time frame  for doing so. FSIS’ pilot project at young hog plants will end when a final  rule for hog slaughter is published.
As FSIS officials stated, when the agency develops a proposed  rulemaking to modify its slaughter inspection system for hogs, it will need  to decide whether to collect additional data. However, while the pilot  project is ongoing, FSIS has the opportunity to follow sound management  practices by planning for and collecting key information needed to  determine whether the pilot project is meeting its purpose. We have  previously reported that pilot programs can more effectively inform future  program rollout when sound management practices are followed.  Consistent with best practices in program management, our guide for  designing evaluations, and our prior work, we identified sound  management practices to design a pilot to guide consistent  implementation, including the type and source of data needed to evaluate  the pilot, and to conduct analysis of the results.
However, FSIS has not collected key information needed to determine  whether the pilot project is meeting its purpose of deploying inspection  resources more effectively in accordance with food safety and other  consumer protection requirements. For example, FSIS has not collected  information on the total costs of the pilot project to the agency or on any  changes in the number of FSIS inspectors at the plants participating in  the pilot project. Thus far, at young hog plants, FSIS veterinarians and  inspectors have evaluated the performance of their individual plants  against regulatory performance standards on a daily basis, as they  routinely do for all plants regardless of whether they are in the pilot  project or not, as well as the performance standards for the pilot project.  However, the agency has not aggregated and analyzed these daily  results to determine how the plants participating in the pilot project have  performed over time. Without collecting and analyzing additional data, it  will be difficult for FSIS to draw conclusions about whether the pilot  project at young hog plants is meeting its purpose of deploying inspection  resources more effectively in accordance with food safety and other  consumer protection requirements.
section_title: Views of Key Stakeholder Groups on Strengths and Weaknesses in the Pilot Projects
paragraphs: We identified strengths and weaknesses of the three pilot projects based  on the views of 11 key stakeholder groups representing industry, labor,  consumer advocacy, and animal welfare. Strengths we identified in the  pilot projects based on the views cited most frequently by stakeholder  groups included the following:
Responsibility and flexibility. Representatives of 7 stakeholder groups  stated that the pilot projects give plants responsibility and flexibility for  ensuring food safety and quality. For example, representatives of 1  stakeholder group stated that, under the pilot projects, plants had  greater flexibility to place plant personnel where they were skilled,  while a representative of another stakeholder group stated that the  pilot projects gave plants responsibility for producing safe food and  allowed plants the latitude to incorporate new methods to ensure food  safety.
More focus on food safety. Representatives of 5 stakeholder groups  stated that the pilot projects allow FSIS to focus more on food safety  activities. For example, a representative of 1 stakeholder group stated  that, in the pilot projects, FSIS inspectors focused on carcass and  verification activities designed to reduce the incidence of foodborne  pathogens.
Potential job creation and increased production. Representatives of 3  stakeholder groups stated that the pilot projects may result in potential  job creation and increased production at plants. For example, a  representative of 1 stakeholder group stated some plants that joined  the pilot projects hired new workers, purchased additional equipment,  and expanded their facilities. A representative of another stakeholder  group stated that the pilot projects allowed plants to increase line  speeds to process and sell a larger quantity of products.
In young chicken plants participating in the pilot project, inspectors view the backs of the  carcasses after the birds have been sorted and cleaned and generally do not touch the  carcasses. pilot projects may decide to address a food safety hazard—such as  fecal material—at a point on the slaughter line after the FSIS  inspector. For example, a representative of 1 stakeholder group  raised a concern that by allowing plants in the pilot project to move  the critical control point for preventing, eliminating, and controlling  fecal material to a location on the slaughter line after the FSIS  inspector, the FSIS inspector no longer had the ability to ensure that   More  the plant complied with the standard to control for that hazard.specifically, the FSIS inspector positioned on the line could no longer  cite the plant for noncompliance with FSIS’ zero-tolerance standard  for fecal material because the plant would not yet have had an  opportunity to control for this hazard. In response, FSIS officials  stated that there were more opportunities for identifying  noncompliance with fecal standards for young poultry plants in the  pilot projects because FSIS inspectors positioned off the slaughter  line perform more food safety activities than at plants not participating  in the pilot projects.
Conflict of interest of plant personnel sorting carcasses.
Representatives from 4 stakeholder groups stated that plants’  responsibility for sorting carcasses presents a conflict of interest. For  example, a representative of 1 stakeholder group expressed concern  that a plant’s financial incentive to process the maximum number of  birds conflicted with its responsibility to regulate itself and stated that if  plant personnel continually removed birds from the line, those  personnel might be taken off the line. FSIS officials stated that plants’  responsibility for sorting carcasses did not present a conflict of interest  because FSIS inspectors performed food safety activities, inspected  each carcass, and verified the effectiveness of plant personnel’s  sorting activities.
Insufficient evidence of success of the pilot projects. Representatives  of 3 stakeholder groups stated that FSIS does not have sufficient  evidence to demonstrate the success of the pilot projects. For  example, a representative of 1 stakeholder group stated that FSIS’  data from the pilot project at young chicken plants demonstrated that  FSIS inspectors on the slaughter lines missed food safety defects on  carcasses, such as fecal material. In particular, the stakeholder group  representative cited FSIS data showing that FSIS inspectors  positioned off the slaughter line who are responsible for, among other  things, verifying the food safety of a sample of carcasses, found food  safety defects at a much higher rate than FSIS inspectors positioned  on the slaughter line. According to FSIS officials, the rate at which  FSIS inspectors positioned on the slaughter line detected food safety  defects was not directly comparable to the rate at which FSIS  inspectors positioned off the slaughter line detected food safety  defects because the inspectors positioned on the line and off the line  had different but complementary roles. FSIS officials maintained, in its  2011 report evaluating the pilot project at young chicken plants, that  the pilot project improved the safety of chicken products at  participating plants. However, as previously noted in this report, we  identified limitations in FSIS’ evaluation.  Increased costs to industry. Representatives of 2 stakeholder groups  stated that the pilot projects can result in increased costs to industry  for additional capital investment, training, and staff. For example, a  representative of 1 stakeholder group stated that plants participating  in the pilot projects may need substantial capital for redesigning  equipment and adding personnel, and a representative of another  stakeholder group noted that plants participating in the pilot projects  needed to add plant personnel to replace FSIS inspectors. FSIS  officials stated that participation in the pilot projects was voluntary and  that plants could choose whether to invest capital when participating.
FSIS officials stated that they recognized there were similarities between  the pilot projects and the optional new poultry inspection system included  in the proposed rule, and they are working to address concerns identified  by stakeholder groups in the final rule on modernizing poultry slaughter  inspections.
section_title: FSIS Did Not Disclose Certain Limitations in the Cost-Benefit Analysis It Developed to Support the Proposed Rule
paragraphs: FSIS did not disclose certain limitations in sources of information it relied  on to develop the cost-benefit analysis supporting the proposed rule on  modernizing poultry slaughter inspections. As a result, the public,  including stakeholders, did not have complete and accurate information to  inform their comments on the proposed rule, including the uncertainty  behind selected estimates. According to the Office of Management and  Budget (OMB) Circular A-4, which provides guidance to federal  agencies on the development of regulatory analysis, and USDA’s  departmental regulation that details the process for developing  regulations, a good cost-benefit analysis is transparent and a qualified  third party reading the analysis should be able to understand the basic  elements.
We identified three sources of information FSIS used that contained  certain limitations that were not disclosed in the cost-benefit analysis.  First, in the 2012 proposed rule, FSIS did not disclose that it used a 2001  survey of plants’ costs of converting to the pilot projects to estimate  certain costs for a plant that slaughters young chickens or turkeys to  operate under the optional new poultry inspection system. According to  USDA’s departmental regulation, the quality of the data used in cost- benefit analyses should be discussed in the proposed rule. However, our  review of the proposed rule found the following limitations in the survey  data that FSIS did not disclose:
Estimated costs included in the January 2012 proposed rule were  based on cost information gathered in 2001. FSIS officials told us that  they did not attempt to obtain more recent information because it was  challenging to collect this type of information from plants, which  consider such information proprietary. FSIS officials stated that there  has been little or no change in the cost information but did not provide  any documentation to support this statement.
FSIS generalized the results from the 12 young chicken plants that  responded to the 2001 cost survey to the universe of 335 young  chicken and young turkey plants in the United States in 2012 to  estimate certain costs for poultry plants to operate under the optional  new poultry inspection system. However, results obtained from the 12  young chicken plants that responded to the survey are not  representative of the universe of young poultry plants.
For the young turkey plants participating in the pilot project, FSIS did  not obtain any cost data. According to FSIS officials, the agency did  not attempt to obtain cost information from participating young turkey  plants because, at the time of the 2001 survey, there were only two  such plants. These officials further stated that the process to slaughter  chickens is sufficiently similar to that for turkeys so costs can be  generalized. However, these officials acknowledged there are  differences between chickens and turkeys, such as carcass size, that  can affect costs, including chilling costs.
As a result of these limitations, the costs for a plant to convert to the  optional new poultry inspection system in today’s economy may be  unclear to the public, including stakeholders.
Second, to estimate selected benefits of the proposed rule, FSIS  assumed a single value for certain economic parameters, rather than  following USDA’s departmental regulation and OMB’s Circular A-4 and  using a sensitivity analysis to provide for a range of uncertainty in the  results. For example, to estimate the expected annual cost savings to  plants in the proposed rule, FSIS assumed that plants participating in the  optional new poultry inspection system would increase their slaughter line  speeds by an average of 6 percent. However, the agency noted in a  footnote in the proposed rule that line speeds could potentially increase  by up to 25 percent (from 140 carcasses per minute to 175 carcasses per  minute) at young chicken plants or by up to 22 percent (from 45  carcasses per minute to 55 carcasses per minute) at turkey plants. These  ranges in line speed are key parameters for estimating benefits to plants  and would affect the labor costs of processing each carcass, as well as  the number of carcasses processed and overall profits. Although FSIS  noted a range in line speed exists, it did not use a sensitivity analysis to  calculate a range of annual net benefits to plants resulting from  uncertainties in line speed. As a result, the public, including stakeholders,  did not have complete and accurate information to inform their comments  on the proposed rule and provide them with a clearer understanding of  the potential impacts of the final rule, including uncertainty behind  selected estimates.
According to USDA’s departmental regulation, uncertainty is inherent in a  cost-benefit analysis, and the uncertainties that are important to  regulatory decisions should be identified and presented as part of the  overall regulatory analysis. According to FSIS officials, the only  uncertainties that are important to this regulation are related to public  health, and the agency included a range in its estimation of the public  health benefits of the proposed rule. However, we believe that there are  other uncertainties in the proposed rule that are important to the cost- benefit analysis, such as the expected benefit to industry from lifting the  current restriction on line speed.
Third, FSIS used a variety of economic studies to assess the economic  conditions in the poultry industry under the proposed rule but did not  identify certain limitations of these studies, including that data in at least  one of the studies were more than 20 years old. Specifically, FSIS did not  disclose that these studies may not reflect current market conditions.  FSIS stated that it used the best available information when it drafted the  proposed rule and cited the studies in footnotes throughout the proposed  rule. However, FSIS did not identify how limitations in the studies could  potentially affect the overall cost-benefit analysis. For example, FSIS  stated in the proposed rule that it assumed the total labor-related cost to  process a bird was 15 percent on the basis of a study published in 2000,  but the agency did not disclose that the study was based on data from  1972 to 1992 for chickens and from 1967 to 1992 for turkeys. As another  example, FSIS estimated the mark-up price of poultry to be 10 percent  more than wholesale prices based on a study published in 2000, which  was based on information on the price of poultry from 1996. As a result,  the public, including stakeholders, may not know if the economic  conditions in the poultry industry presented in the proposed rule  accurately reflect current market conditions.
According to FSIS officials, the agency plans to address some of the  limitations in its revised cost-benefit analysis to support the final rule. For  example, the agency plans to include a range for annual production cost  savings in the revised cost-benefit analysis, which is part of the draft final  rule. However, according to FSIS officials, the draft final rule is  undergoing departmental review and was not made available to us, as it  is subject to additional changes. It is unclear whether all of the limitations  that we identified will be disclosed in the final rule.
Moreover, to estimate the public health impact of the proposed rule, FSIS  developed a risk assessment that contributes to its cost-benefit analysis.  FSIS’ risk assessment model examined how pathogen levels in poultry  products could be affected, depending on the frequency and scheduling  of activities performed by FSIS inspectors positioned off the slaughter  line. We were unable to determine if the results of FSIS’ risk assessment  accurately stated the public health benefits in the proposed rule because  the risk assessment did not include sufficient detail about its  methodology. According to USDA’s departmental regulation and OMB  Circular A-4, a good analysis is transparent, and a qualified third party  reading the regulatory analysis should be able to see what data, methods,  models, and assumptions were used to arrive at the agency’s estimates.  However, in its risk assessment, FSIS did not disclose its rationale for  numerous, complex key assumptions critical to the analyses it used to  calculate public health benefits for the proposed rule. In addition, in the  proposed rule, FSIS identified a range of 1,540 to 10,547 potential  illnesses averted by increasing the number of unscheduled inspection  activities off the slaughter line. However, this only covered 80 percent of  the estimated range of potential illnesses averted, and it is unclear why   Had a wider  FSIS did not disclose a wider range of potential benefits.range been used, FSIS would have increased the statistical confidence of  its estimate in the number of illnesses averted by increasing unscheduled  activities off the slaughter line. FSIS officials acknowledged the  importance of disclosing sufficient details about the methodology and  stated that the risk assessment represents a snapshot in time; they also  said that a revised risk assessment supporting the final rule—an  assessment that is not yet available to the public—includes more of this  critical information. The revised risk assessment is part of the draft final  rule that is undergoing departmental review and was not made available  to us.
section_title: Conclusions
paragraphs: In an effort to deploy inspection resources more effectively in accordance  with food safety and other consumer protection requirements, FSIS has  been conducting pilot projects at slaughter plants for young chickens,  young turkeys, and young hogs since 1998. These pilot projects are in  keeping with FSIS’ broader effort to move toward a risk-based inspection  system, which we believe is a positive step. However, we found that FSIS  has not thoroughly evaluated how the three pilot projects have performed  over time. Specifically, there are limitations in the agency’s data analyses  in its report evaluating the pilot project at young chicken plants, and there  is no report evaluating the pilot project at young turkey plants.  Nonetheless, the agency moved forward with a proposed rule in January  2012 on modernizing poultry slaughter inspections that included an  optional new poultry inspection system, based, in part, on its experience  with the pilot projects at young chicken and young turkey plants. As a  result, FSIS may not have assurance that its evaluation of the pilot project  at young chicken plants provides the information necessary to support the  proposed rule for both chickens and turkeys. However, the agency will not  complete another evaluation before it issues a final rule. To support the  proposed rule, the agency developed a cost-benefit analysis but did not  disclose certain limitations in sources of information—including that data  in at least one study was more than 20 years old—it relied on to develop  this analysis, which is not consistent with USDA regulation and OMB  Circular A-4 stating that a good analysis for rulemaking is transparent. As  of July 2013, FSIS officials stated that the agency plans to address some  of these limitations as it works to complete the final rule. Without  complete disclosure from FSIS, the public, including stakeholders, did not  have complete and accurate information to inform their comments on the  proposed rule and provide them with a clearer understanding of the  potential impacts of the final rule, including the uncertainty behind  selected estimates. By addressing these limitations moving forward in its  rulemaking for modernizing poultry slaughter inspections, FSIS can  prepare for more transparency in the development of a future proposed  rule to modify slaughter inspection for hogs based on the pilot project at  young hog plants.
In 2011, FSIS began drafting a preliminary report evaluating the pilot  project at young hog plants, which uses analyses similar to those  presented in the report evaluating the pilot project at young chicken  plants, suggesting that similar limitations may apply. FSIS officials stated  that the agency intends to complete its evaluation of the pilot project at  young hog plants by March 31, 2014. They also stated that the agency  would need to decide if additional data would be collected when it  proceeded forward with a rulemaking effort for hogs. Without collecting  and analyzing additional data, it will be difficult for FSIS to draw  conclusions about whether the pilot project at young hog plants is  meeting its purpose of deploying inspection resources more effectively in  accordance with food safety and other consumer protection requirements.  Because the pilot project at young hog plants is ongoing, FSIS has the  opportunity to follow sound management practices by planning for and  collecting key information needed to determine whether the pilot project is  meeting its purpose. As we have previously reported, pilot programs can  more effectively inform future program rollout when sound management  practices are followed.
section_title: Recommendations for Executive Action
paragraphs: We recommend that the Secretary of Agriculture direct the Administrator  of the Food Safety and Inspection Service to take the following two  actions:
Clearly disclose to the public limitations in the information—including  the cost-benefit analysis—the agency relied on for the rulemaking to  modernize poultry slaughter inspections.
As FSIS continues its evaluation of its pilot project for young hogs,  collect and analyze the information necessary to determine whether  the pilot project is meeting its purpose.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to USDA for its review and comment.  USDA’s written comments and our detailed response to them are  reproduced in appendix III. In its written comments, USDA concurred with  both of our recommendations. More specifically, USDA concurred with  our recommendation that it clearly disclose to the public limitations in the  information—including the cost-benefit analysis—it relied on for the  rulemaking to modernize poultry slaughter inspections. According to  USDA, when it issues the final rule, it will present the updated analyses in  a manner that will facilitate the public’s understanding of the information  used to support its rulemaking. USDA also concurred with our  recommendation to collect and analyze the information necessary to  determine whether the pilot project for young hogs is meeting its purpose,  while continuing its evaluation of this pilot project. According to USDA, it  plans to complete such an evaluation by March 31, 2014, at which time it  will determine whether a permanent program is warranted.
USDA also made several general comments. For example, USDA  commented that, throughout our report, we state that the purpose of the  pilot projects was to “…deploy inspection resources more effectively…”.
USDA further commented that, while this might have been true when the  pilot projects were initiated in 1997, the agency’s thinking has evolved  over the years to focus less on efficiencies and more on public health and  food safety. We recognize that USDA’s descriptions of the pilot projects,  as stated in the Federal Register notices, have evolved over the years.  However, we believe that the purpose stated in our report—deploying  inspection resources more effectively in accordance with food safety and  other consumer protection requirements—remains valid because the  Federal Register notices cited by USDA continue to mention effective use  of resources as a component of its pilot projects. Moreover, agency  officials directed us to the Federal Register notices because the agency  could not provide us with documents defining the pilot projects’ purpose.  As our report states, pilot programs can more effectively inform future  program rollout when sound management practices are followed,  including the development of an evaluation plan with well-defined, clear,  and measurable objectives.
Furthermore, USDA provided technical comments in its written response,  which we have incorporated, as appropriate, in the report.
As agreed with your office, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date. At that time, we will send copies of this report to the  Secretary of Agriculture, the appropriate congressional committees, and  other interested parties. In addition, the report will be available at no  charge on the GAO website at http://www.gao.gov.
If you or your staff members have any questions regarding this report,  please contact me at (202) 512-3841 or gomezj@gao.gov. Contact points  for our Offices of Congressional Relations and Public Affairs may be  found on the last page of this report. Key contributors to this report are  listed in appendix IV.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: This report responds to your request that we review U.S. Department of  Agriculture’s (USDA) pilot projects at slaughter plants for young chickens,  young turkeys, and young hogs. Our objectives were to determine (1) the  extent to which USDA has evaluated the three pilot projects, (2) strengths  and weaknesses of the three pilot projects based on the views of key  stakeholder groups, and (3) the extent to which USDA disclosed  limitations, if any, in sources of information it relied on to develop the  proposed rule to modernize poultry slaughter inspections.
To determine the extent to which USDA has evaluated the three pilot  projects, we reviewed relevant USDA documents, Federal Register  notices, and laws. We also compared USDA’s efforts to evaluate the pilot  projects with criteria based on social science and evaluation literature and  published GAO guidance that were identified in our previous work on pilot  program development and evaluation. Our previous work dealt with pilot  projects at the Federal Emergency Management Agency, the Internal  Revenue Service, and the Department of Defense where we identified  key features of a well-developed evaluation plan that increases the  likelihood that evaluations would yield methodologically sound results and  support effective program and policy decisions. We believe that the key  features identified in those reports are applicable to pilot projects, in  general, including USDA’s three pilot projects. In addition, we interviewed  several officials in various offices within USDA’s Food Safety and  Inspection Service—the agency responsible for USDA’s meat and poultry  inspection program.
To determine the strengths and weaknesses of the three pilot projects  based on the views of key stakeholder groups, we identified key  stakeholder groups representing industry, labor (including plant personnel  and USDA inspectors and veterinarians), consumer advocacy, and  animal welfare that submitted comments on USDA’s proposed rule on  modernizing poultry slaughter inspections. We identified 11 key  stakeholder groups with sufficient knowledge about USDA’s pilot projects  at young chicken, young turkey, and young hog plants to identify  strengths and weaknesses for these pilot projects. These stakeholder  groups were the American Federation of Government  Employees/National Joint Council of Food Inspection Locals, the  American Meat Institute, the Center for Foodborne Illness Research and  Prevention, the Consumer Federation of America, Food and Water  Watch, the Government Accountability Project, the Humane Society of  the United States, the National Association of Federal Veterinarians, the  National Chicken Council, the National Turkey Federation, and the North  American Meat Association. We reviewed their comments to USDA’s  proposed rule to determine the extent to which the comments may apply  to the pilot projects; interviewed representatives of these key stakeholder  groups; and followed up with e-mailed questions to gauge their level of  familiarity with each pilot project and then clarified responses, as needed.  Three of our analysts worked together to develop categories of strengths  and weaknesses identified most frequently by stakeholder groups, and  two analysts independently determined whether each stakeholder group  had identified strengths or weaknesses that fit into these categories. Any  discrepancies in coding were discussed, and agreement was reached  between the analysts, or resolved through a third analyst’s review. We  also visited 10 slaughter plants for young chickens, young turkeys, and  young hogs to learn about the variations in slaughter inspection systems  between plants participating and not participating in the pilot projects. In  selecting these plants, we chose plants in states that are the top  producers of young chickens, young turkeys, and young hogs and that  have at least one plant that is participating in a pilot project and at least  one plant that is not participating in a pilot project for one of those  species. On the basis of these selection criteria, we visited plants in  Georgia for young chickens, Indiana for young turkeys, and Minnesota for  young hogs. For each species, we visited at least one plant participating  and one plant not participating in the pilot projects. We interviewed plant  management and USDA veterinarians and inspectors working at these  plants, and we toured the plants.
To determine the extent to which USDA disclosed limitations, if any, in  sources of information it relied on to develop the proposed rule, we  reviewed the proposed rule and related Federal Register notices, as well  as selected documents the agency relied on to develop the proposed  rule. We also reviewed relevant guidance such as the Office of  Management and Budget Circular A-4, which provides guidance to  federal agencies on the development of regulatory analysis, and the  USDA departmental regulation that details the process for developing  regulations. We also interviewed several officials in various offices within  USDA’s Food Safety and Inspection Service to clarify information, as  needed.
We conducted this performance audit from September 2012 to August  2013 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence presented provides a reasonable basis for our findings  and conclusions based on our audit objectives.
section_title: Appendix II: Additional Information on the Young Chicken, Young Turkey, and Young Hog Plants in the Pilot Projects
paragraphs: As of July 2013, 29 plants were participating in the pilot projects in 18  states. More specifically, there were 19 participating young chicken plants  located in 10 states. These 10 states accounted for almost 66 percent of  the more than 49.3 billion pounds of young chickens slaughtered in  2012. (See fig. 2.)
In terms of the pilot project at young turkey plants, four of the five young  turkey plants in the pilot project were located in states that accounted for  about 33 percent of the more than 7.4 billion pounds of young turkeys  slaughtered in 2012; one young turkey plant was located in Michigan—a  state with a lower volume of young turkeys slaughtered—and data were  not available on the volume of young turkeys slaughtered in this state.  (See fig. 3.)
In terms of the pilot project at young hog plants, the five hog plants in the  pilot project were located in five states that accounted for about 30  percent of the more than 31 billion pounds of hogs slaughtered in 2012.(See fig. 4.)
section_title: Appendix III: Comments from the U.S. Department of Agriculture
subsections: section_title: GAO Comments
paragraphs: The following are GAO’s comments on the U.S. Department of  Agriculture’s letter dated August 15, 2013.  1. USDA commented that our report does not highlight what the agency  considers to be the main objectives of the proposed rule to modernize  poultry slaughter inspections—to reduce the risk of foodborne illnesses by  focusing FSIS inspection activities on those tasks that advance the  agency’s core mission of food safety. However, our report includes this  information. For example, our report states, “according to the proposed  rule, the modernization is intended to improve food safety and the  effectiveness of poultry slaughter inspection systems, remove  unnecessary regulatory obstacles to innovation, and make better use of  the agency’s resources,” among other things.   2. USDA commented that throughout our report we state that the purpose  of the pilot projects was to “…deploy inspection resources more  effectively…”. USDA further commented that, while this might have been  true when the pilot projects were initiated in 1997, the agency’s thinking  has evolved over the years to focus less on efficiencies and more on  public health and food safety. We recognize that USDA’s descriptions of  the pilot projects, as stated in the Federal Register notices, have evolved  over the years. However, we believe that the purpose stated in our  report—deploying inspection resources more effectively in accordance  with food safety and other consumer protection requirements—remains  valid because the Federal Register notices cited by USDA continue to  mention the effective use of resources as a component of its pilot  projects. Moreover, agency officials directed us to Federal Register  notices because USDA could not provide us with documents defining the  pilot projects’ purpose. As our report states, pilot programs can more  effectively inform future program rollout when sound management  practices are followed, including the development of an evaluation plan  with well-defined, clear, and measurable objectives.   3. USDA commented that we did not mention the validity of the agency’s  approach for comparing food safety data. We modified our report to  include this information. USDA also commented that we did not include a  number of conclusions from a 2002 external review, which focused on the  validity of using food safety, microbial, and other consumer protection  (food quality) data to assess the accomplishments of young chicken  plants participating in and those not participating in the pilot project. Our  report acknowledges this external review, but looking at the validity of the  performance measures the agency developed for the pilot project was  beyond the scope of our review.   4. We modified our report to clarify the footnote about the food safety  performance standard.  5. We modified our report to clarify the description of how the data from  young turkey plants were used in USDA’s risk assessment.   6. We modified our report to clarify USDA’s perspective on training and  that this training discussion is specific to those plants participating in the  pilot projects.  7. We modified our report to include a statement about USDA inspectors’  abilities to cite plants for noncompliance at plants not participating in the  pilot project.   8. USDA commented that our discussion of line speed is misleading  because, among other things, it implies that a higher line speed than the  average used in the analysis would lead to additional costs of processing.  We recognize that higher line speeds lead to lower per unit labor costs  and have not made statements to the contrary. However, we note that  USDA did not incorporate the impact of a range in line speeds in its  estimates of benefits to plants. These ranges in line speed are key  parameters for estimating benefits to plants and would affect the labor  costs of processing each carcass, as well as the number of carcasses  processed and overall profits. We modified our report to clarify that  although FSIS noted a range in line speed exists, it did not use a  sensitivity analysis to calculate a range of annual net benefits to plants  resulting from uncertainties in line speed.  9. USDA commented that our statement about how the agency plans to  address some of the limitations in the cost-benefit analysis is not  completely accurate. We modified our report to state that the agency  plans to include a range for annual production cost savings in the revised  cost-benefit analysis, which is part of the draft final rule. However, USDA  also stated that the final rule will not provide a range of benefits to plants  based on uncertainties in line speed. As explained above in comment 8,  we believe line speed is a key parameter in the estimation of benefits to  plants.
section_title: Appendix IV: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
paragraphs: J. Alfredo Gómez, (202) 512-3841 or gomezj@gao.gov.
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Mary Denigan-Macauley  (Assistant Director), Marie Bancroft, Kevin Bray, Stephen Carter, Barbara  El Osta, Michele Fejfar, Diana C. Goody, Armetha Liles, Cynthia Norris,  and Dae Park made key contributions to this report.