1

Use of FSIS Regulatory Verification Sampling to Generate 

Prevalence Estimates

DCC Prevalence Estimate Workgroup

April 2012
2

Table of Contents

Executive Summary ....................................................................................................................................4

Chapter 1: Background, FSIS Definition of Prevalence and Data and Statistical Requirements.......5

FSIS Prevalence Estimates ...................................................................................................................6

Relationship to FSIS Strategic Planning Efforts...................................................................................6

FSIS Public Health Information System (PHIS) ..................................................................................7

Evaluation of Pathogen Testing Programs............................................................................................7

FSIS Definition of Pathogen Prevalence ..............................................................................................9

FSIS Data and Statistical Requirements............................................................................................. 11

Chapter 2: E. coli O157:H7 Verification Sampling Program ............................................................... 13

Overview............................................................................................................................................. 14

Background: FSIS’ E. coli O157:H7 Verification Sampling Projects................................................ 15

Data Requirements.............................................................................................................................. 17

Analysis of Data Limitations .............................................................................................................. 23

Conclusions......................................................................................................................................... 24

Chapter 3: Salmonella Verification Sampling Program........................................................................ 25

Overview............................................................................................................................................. 26

Background......................................................................................................................................... 26

Data Requirements.............................................................................................................................. 28

Analysis of Data Limitations .............................................................................................................. 31

Conclusion .......................................................................................................................................... 33

Chapter 4: Ready-To-Eat (RTE) Sampling Program............................................................................ 34

Overview............................................................................................................................................. 35

Background......................................................................................................................................... 35

Data Requirements.............................................................................................................................. 37

Analysis of Data Limitations.............................................................................................................. 41
3

Conclusion .......................................................................................................................................... 41

Appendix A................................................................................................................................................ 42

Appendix B ................................................................................................................................................ 44

Appendix C................................................................................................................................................ 48

References.................................................................................................................................................. 49
4

Executive Summary

Background

In the past, the United States Department of Agriculture’s (USDA) Food Safety and Inspection 

Service (FSIS) has been criticized for asserting that some types of percent positive data were

representative of pathogen prevalence. Further, FSIS is aware that some stakeholders have used 

pathogen testing data to make their own assertions about the prevalence of pathogens on 

regulated products. However, at the present time, FSIS only estimates pathogen prevalence 

when conducting baseline studies. 

To address these issues, FSIS conducted an evaluation the Agency’s current pathogen 

verification testing programs to determine whether they provide sufficient data to calculate

prevalence estimates for pathogens in FSIS-regulated product at a given point in the production 

process. The three pathogens of interest were: Escherichia coli (E. coli) O157:H7, Salmonella,

and Listeria monocytogenes (Lm) and Salmonella in Ready-To-Eat (RTE) products, consistent 

with FSIS’ current major verification testing programs. The evaluation was performed by a new 

FSIS workgroup formed through the Agency’s Data Coordination Committee (DCC) and

included representatives from the Office of Data Integration and Food Protection (ODIFP), the 

Office of Public Health Science (OPHS) and the Office of Policy and Program Development 

(OPPD). 

The purpose of this evaluation was to 1) construct a working definition of pathogen prevalence

and identify data and statistical elements required to yield prevalence estimates, 2) review 

current pathogen verification testing programs in light of the data and statistical elements 

required to estimate prevalence and 3) identify what barriers or limitations exist to utilizing 

current programs to estimate prevalence. The following report provides a description of this 

evaluation and FSIS’ findings by pathogen. 

Overall Findings

At this time, given the current construction of the FSIS Pathogen Verification Sampling 

Programs, the Agency believes that it is only possible to utilize the existing E. coli O157:H7

pathogen verification testing project in raw ground beef (MT43) to estimate the national 

prevalence. Due to a variety of methodological and sampling related issues, FSIS does not 

believe it is possible to utilize existing pathogen verification testing projects to estimate 

prevalence for E. coli O157:H7 in beef trim and components, Salmonella in all raw products, or

Lm and/or Salmonella in RTE and post-lethality exposed products. 
5

Chapter 1: Background, FSIS Definition of Prevalence and Data and Statistical Requirements
6

FSIS Prevalence Estimates

The United States Department of Agriculture’s (USDA) Food Safety and Inspection Service 

(FSIS) recognizes the importance of timely and accurate estimates of pathogen prevalence in 

order to better understand how contamination rates change over time, set performance standards 

to reduce product contamination, develop targeted interventions and policies and measure the 

Agency’s performance towards meeting FSIS strategic planning goals, as well as the Healthy 

People 2020 goals. FSIS also uses prevalence estimates in economic analyses and risk 

assessments and routinely updated prevalence estimates would allow the Agency to more rapidly 

and effectively update existing analyses. Finally, prevalence estimates provide FSIS with a 

proxy measure of the Agency’s public health impact, in situations where direct illness outcome 

measures are lacking.

Historically, FSIS has used Agency’s traditional microbiological baseline studies to derive 

estimates of pathogen prevalence in FSIS-regulated products at a given point in the production 

process. This data is then utilized to establish performance standards for the regulated industry. 

However, baseline studies are usually targeted at a specific commodity-pathogen pair to answer 

specific questions and are not repeated annually. Similarly, although volume-weighted

calculations produced from some of FSIS’ pathogen verification testing programs provide the 

Agency with important public health measures, not all of these programs produce data that 

support the calculation of true pathogen prevalence estimates. Further, FSIS has been criticized 

for asserting that some types of percent positive data were representative of pathogen prevalence 

and the Agency is aware that some stakeholders have used pathogen testing data to make their 

own assertions about the prevalence of pathogens on regulated products. Therefore, the overall, 

primary purpose of this report is to evaluate the possibility of using current FSIS pathogen 

verification testing data to provide on-going estimates of the national prevalence of pathogens in 

FSIS-regulated product at the point of processing. 

Relationship to FSIS Strategic Planning Efforts

This report also builds on and supports FSIS’ strategic planning efforts in a number of ways. In 

September 2010, FSIS released the Strategic Data Analysis Plan for Domestic Inspection, which 

identified the need for ―on-going‖ baselines to measure prevalence and described changes to 

pathogen sampling programs required to achieve this goal.1 In September 2011, FSIS released 

the Agency’s Strategic Plan for 2011-2016, which included the goals that align with an 

evaluation of prevalence:

2

– Goal 1: Ensure that food safety inspection aligns with existing and emerging risks

– Goal 5: Effectively use science to understand foodborne illness and emerging 

trends

FSIS also released the Report on the Food Safety and Inspection Service‘s Microbiological and 

Residue Sampling Programs in December, 2011 which identifies all of FSIS' sampling programs 

 

1

Please see the following website for more information: 

http://www.fsis.usda.gov/OPPDE/NACMPI/Sep2010/2010_Strategic_Data_Analysis_Plan.pdf. 

2

Please see the following website for more information:

http://www.fsis.usda.gov/PDF/Strategic_Plan_2011-2016.pdf.
7

and discusses the statistical and policy basis for the programs.

3

 Finally, FSIS released the

FY2012 Sampling Program Plan in February 2012, which continues the Agency’s efforts to 

comprehensively identify FSIS activities and consider them in light of data-driven strategic 

planning efforts.

4

FSIS Public Health Information System (PHIS)

On April 11, 2011, FSIS launched its dynamic, comprehensive data analytics system called the 

Public Health Information System (PHIS). PHIS is a web-based application that integrates and 

automates FSIS’ paper-based business processes into one comprehensive and fully automated 

data-driven inspection system. As a result of implementing PHIS, many of FSIS' existing 

systems, such as the Performance Based Inspection System (PBIS), will be phased out and 

replaced by PHIS. Consequently, while the way in which sampling information is scheduled, 

shared and stored will change under PHIS, none of the fundamental elements of FSIS’ sampling 

activities, such as the sampling frame, methodology or collection methodology, will change. It is 

important to note, however, that at the time this report was developed, PHIS was not fully 

implemented.

Evaluation of Pathogen Testing Programs

To conduct this evaluation, FSIS formed a new Prevalence Workgroup through the Agency’s

Data Coordination Committee (DCC)5to study using verification testing data to produce national 

pathogen-specific prevalence estimates in FSIS-regulated products. Pathogen-specific subgroups 

were also formed. The workgroup/subgroups were charged with the following tasks:

Step 1: Construct a working definition of pathogen prevalence.

Step 2: Formulate a list of data and statistical requirements that would need to be met in 

order for FSIS’ pathogen verification testing programs to yield prevalence estimates, as 

defined in Step 1. 

Step 3: Evaluate the suitability of each pathogen verification program, based on the data 

and statistical requirements defined in Step 2, to yield estimates of pathogen prevalence.

In addressing task 3, the workgroup focused on three pathogens: Escherichia coli (E. coli)

O157:H7, Salmonella, and Listeria monocytogenes (Lm) and Salmonella in Ready-to-Eat (RTE) 

products, consistent with FSIS’ current major verification testing programs. 

The remaining subsections of Chapter 1 present the definition of pathogen prevalence developed 

by the workgroup and describe the data and statistical elements required to utilize pathogen 

verification testing programs for measurement of pathogen prevalence in FSIS-regulated 

product. Chapters 2 through 4 provide, by pathogen, an evaluation of the feasibility of using 

FSIS pathogen verification testing programs to estimate prevalence. Sample collection data 

provided in the tables within these chapters are those data that were available at the time of this 

 

3

Please see the following website for more information: 

http://www.fsis.usda.gov/PDF/FSIS_Sampling_Programs_Report.pdf.

4

Please see the following website for more information:

http://www.fsis.usda.gov/PDF/Sampling_Program_Plan_FY2012.pdf

5

For more information regarding the FSIS Data Coordination Committee, please see the following website: 

http://www.fsis.usda.gov/OPPDE/rdad/FSISDirectives/5800.1.pdf.
8

assessment and therefore are not official, FSIS end-of-year data. A glossary of relevant terms 

has also been developed and is included as Appendix A. A summary of the DCC Pathogen 

Subgroup’s evaluation of current sampling programs in light of the data and statistical 

requirements is provided in Appendix B. Appendix C provides more detailed information on 

FSIS’ Salmonella sampling projects.

The purpose of this report is to evaluate the suitability of using FSIS’ current pathogen 

verification testing programs to estimate prevalence. Moving forward, if FSIS determines that 

the Agency’s pathogen verification testing programs should be modified to estimate prevalence 

based on the findings of this report, a workgroup will be established and appropriate statistical 

analyses will be conducted to determine the most appropriate way to change pathogen testing 

programs to yield estimates of prevalence. The results of these analyses, once completed, will 

also be shared publicly.
9

FSIS Definition of Pathogen Prevalence 

FSIS defines prevalence as the proportion of population units that would test positive for a given 

pathogen if the entire population were sampled and analyzed during a specified time period. To 

further clarify this definition, the following additional definitions are necessary. 

A population may be defined as a species, an animal part, a product, or an 

environmental source (e.g. cows, chickens, turkeys, ground beef, sausage, deli meats, 

processed egg products and/or food contact surfaces) that is in an FSIS-regulated 

establishment.

A population unit is defined as the population element of interest, as described below:

o A population unit may be a carcass in a population of animals (e.g., a young 

chicken carcass in the population of federally-inspected, slaughtered young 

chickens).

o Alternatively, a population unit may be a specified quantity of product (e.g., a 

pound of raw ground turkey from the population of federally-inspected raw 

ground turkeys).

A sample is defined as the portion of the product that is collected for analysis.

In developing this definition of prevalence, FSIS has identified the following factors that are 

relevant when computing a prevalence estimate. Any FSIS estimate of prevalence:

Will not directly measure or describe risk of illness.

Will not directly measure or estimate the actual presence of contamination within an 

individual product.

Will vary by sampling program.

Will be dependent on the location at which the sample is collected (e.g., post-chill 

sampling compared to re-hang sampling).

Will require adjustments to account for production volume and/or other 

characteristics.

Will be affected by laboratory and collection methods.

Will be affected by information used to define the sampling frame (e.g. inaccurate 

production information, inaccurate Hazard Analysis and Critical Control Points 

(HACCP) categorization).

Will be affected by the number of samples in the analysis.

Will have a varying degree of error based upon these factors.

Further, although analysis of non-methodological factors was not part of the DCC workgroup’s

task, several of these issues are noted below to guide future considerations of prevalence 

estimation. 

A prevalence estimate is only one measure of a population. To be meaningful, a 

prevalence estimate typically warrants detailed discussion within a broader exposure 

assessment, as FSIS does when it presents prevalence estimates derived from baseline 

studies or utilized in risk assessments. 
10

Prevalence may most commonly be used to portray stable characteristics in closed 

populations. Prevalence estimates may be difficult to interpret when used to describe 

complex properties of dynamic populations.

Estimating the prevalence of rare characteristics, such as the presence of E. coli O157:H7 

adulteration, is challenging at best. Survey methodologies developed to assess hard-toreach populations may be applicable, but may require significant modifications to FSIS’ 

current sampling projects. 

Because some types of pathogen contamination, such as E coli O157:H7 adulteration, is 

very rare and because of the differences across slaughter and processing establishments, 

prevalence estimates are poor measures of both establishment process control and FSIS 

program performance. 

In the future, FSIS would consider estimating pathogen prevalence on a quarterly and annual 

basis using a year of rolling data. For example, for the calculation of the first quarter of Fiscal 

Year (FY) 2012, FSIS would use data from January 2011 through December 2011. Final Fiscal 

Year and Calendar Year (CY) estimates would also be computed using a full year of rolling data. 

Limitations of Prevalence Definition 

As the formulation and construction of FSIS pathogen verification programs vary significantly 

across pathogens, it is not possible to compare prevalence estimates across FSIS regulated 

products. 

Assumptions

Additionally, many testing or estimation procedures are subject to the validity of the assumptions 

made in producing the statistical results. Once specific estimation procedures are identified for 

producing prevalence estimates for individual products or projects, the associated assumptions 

will need to be examined to determine if any further modifications to the estimation procedures

are needed. Assumptions associated with all samples include:

The sample is representative of an identifiable subset of the total collection population.

Laboratory analyses provide consistent results across all samples tested.

Organisms are spread throughout the product being tested, such that the sampled area or 

product is representative of the population.

No additional contamination was added during sample collection or shipping.

Shipment procedures limit any recovery, outgrowth, or reproduction of the pathogen so 

that the estimated presence and levels are similar to the presence and levels at the sample 

collection point. However, further processing, partitioning, growth and cooking ensure 

that a proportional relationship might exist between tested product and actual servings. 

The assumptions above may not hold in all cases, which can introduce bias and therefore provide 

further limitations on the resulting estimates.
11

FSIS Data and Statistical Requirements 

FSIS has identified the following data and statistical requirements for the use of verification data 

to compute prevalence estimates:

Define the Population

o The population is the universe of units for which the characteristic of interest is being 

assessed.

o The population may be defined as a species, an animal part, a product, or an 

environmental source (e.g., cows, chickens, turkeys, ground beef, sausage, deli meats, 

processed egg products and/or food contact surfaces) that is sampled by FSIS.

Define the Population Unit

o A population unit is defined as the population element of interest. A population unit 

may be a carcass in a population of animals (e.g., a young chicken carcass in the 

population of federally-inspected, slaughtered young chickens), or a population unit 

may be a specified quantity of product (e.g., a pound of raw ground turkey from the 

population of federally-inspected raw ground turkey).

Define the Sampling Frame

o The sampling frame is a listing of all the units in the defined population.

o Alternatively, the sampling frame may be an aggregated listing of enumeration units, 

where each population unit (e.g., chicken carcass or pound of ground beef) is 

associated with one and only one enumeration unit (e.g., slaughter/processing 

establishment) in the sampling frame.

o For a national prevalence estimate of a given pathogen-product pair, the sampling 

frame would typically be a complete listing of FSIS-regulated establishments that 

produce a given product in that year and their associated production volume.

Define the Collection Frame

o This could be the entire sampling frame, or this could be a listing of establishments 

(or other enumeration units) selected for sampling from the sampling frame.

o The selection procedure used to establish the collection frame must also be defined.

Limitations on Unit Choice

o Random unit choice is preferred.

o Clustered samples are acceptable, but the effect of clustered choice must be 

addressed.

o Announced sampling is discouraged as they may make clustered choice not 

acceptable.

Limitations on Product Choice6

o One hundred percent judgment based selection is not acceptable.

o Product is stratified based on a risk characteristic (e.g. product type), and then chosen

randomly within each stratum.

Limitations on Establishment Choice

 

6 Occurs when a product class is comprised of several different types. For example, for E. coli O157:H7, trim and 

components can be comprised of beef heart, lymph and cheek meat, but for Salmonella, the only product choice

available is broiler carcasses.
12

o Establishments are stratified based on a characteristic (e.g. risk, production volume), 

and then chosen randomly within each stratum.

Additional required data and statistical considerations:

o These factors will be used to determine the probability of selection for each unit of 

product for each establishment:

 Day of week and time of day should be randomly chosen.

 The production volume for each establishment for each product is required.

Additional useful, but not required data points:

o Interventions being used

o HACCP size

o Product production volume

o Monthly total

o Daily total for sample collection day

Additional useful, but not required sample characteristics:

o Sample weight

o Shift sample collected

o Date

o Time elapsed since last cleanup
13

Chapter 2: E. coli O157:H7 Verification Sampling Program
14

Overview

Calculating an E. coli O157:H7 prevalence estimate using the current verification data is 

possible for MT43 (raw ground beef verification project). However, the E. coli O157:H7

Prevalence subgroup cautions that the power to detect anything less than a drastic change in 

prevalence is small. In other words, FSIS could only declare a statistically significant difference 

in prevalence when the change is substantial. In contrast, calculating a prevalence estimate using 

the current verification data is not possible for MT50, MT54 and/or MT55 (beef manufacturing 

trimmings, other components to raw ground beef, and bench trim) without substantial changes to 

the current sampling projects. Table 1 provides an overview of the data requirements evaluated 

and the findings of the E. coli O157:H7 subgroup in terms of current E. coli O157:H7 pathogen 

verification testing projects. 

The current E. coli O157:H7 verification projects (MT43, MT50, MT54 and MT55) test 

intermediate and final beef products before they are available to the consumer. Because of this, 

the relationship between the verification projects for trim, components and raw ground beef are 

complex and dynamic. It is also important to note at the beginning of this discussion that E. coli

O157:H7 adulteration is very rare, which is an important factor to consider in the calculation of 

prevalence estimates, their confidence intervals, and for determining when a significant change 

in prevalence has occurred. As such, data from the various E. coli O157:H7 projects should not 

be combined in an effort to calculate a comprehensive E. coli prevalence estimate.

Table 1: Overview of FSIS Findings Regarding Evaluation of Data Requirements of E. coli 

O157:H7 Pathogen Verification Testing Projects

Data and Statistical 

Requirements

E. coli O157:H7

MT43 MT50 MT54 MT55

Population Yes Yes Yes Yes

Sampling Frame Yes Yes Yes Yes

Collection Frame Yes Yes Yes Yes

Enumeration Unit 

Selection

Yes Yes Yes Yes

Population Unit Selection Yes Yes Yes Yes

Product Type Selection N/A, Yes N/A, Yes Yes N/A, Yes

Probability of Selection Yes Yes Yes Yes

Production Volume Yes Yes Yes Yes

Sampling representative of 

population Yes No No No

Sampling provides desired 

precision Yes No No No

CONCLUSION Yes NO NO NO
15

Background: FSIS’ E. coli O157:H7 Verification Sampling Projects

The E. coli O157:H7 verification sampling program formally began on October 17, 1994. The 

program was originally instituted to detect E. coli O157:H7 in raw ground beef as a means to 

verify process control under HACCP. The original objective of this program was to stimulate 

industry testing and other actions to reduce the presence of the pathogen in raw ground beef. The 

testing program has evolved over time and now there are verification programs for other 

commodities, such as beef trim and other raw ground beef components. Table 2 describes the 

various FSIS E. coli O157:H7 sampling verification projects. The projects that are in bold letters 

and highlighted in blue were evaluated in this project. Finally, Figure 1 displays a flow-chart 

which includes both verification projects and follow-up/trace-back projects.

Table 2: FSIS E. coli O157:H7 Verification Sampling Projects. 

*CY= Calendar year. HACCP: Hazard Analysis and Critical Control Point program. 

Product Type Project 

Code

Sample 

Size

Number of

Analyzed 

Samples 

(CY2010)

Regulatory 

Purpose of

Sampling Project

Type of Sampling 

Algorithm

Raw ground beef MT43 15,600 11,291

Verify nondetectable 

standard

Risk-Based

Follow up testing to a 

raw ground beef 

positive

MT44 NA 309 Verify corrective 

measures

Targeted 

Consecutive

Beef trim (slaughter 

establishments) MT50 2,600 1,274 Verify HACCP compliance Random

Raw ground beef 

components (other 

than trim)

MT54 780 169 Verify HACCP 

compliance Random

Bench trim MT55 1,800 547 Verify HACCP 

compliance Random

Follow up testing at 

supplier establishments 

following MT43, 

MT44, or MT55 

positive

MT52 NA 636 Verify corrective 

measures

Targeted 

Consecutive

Follow up testing to a 

MT50, MT54, MT55, 

MT53, or MT52 

positive

MT53 NA 125 Verify corrective 

measures

Targeted 

Consecutive
16

Figure 1. Flow chart of all FSIS’ E. coli O157:H7 sampling projects. 

MT 53

Follow-up trim 

and components

Slaughter Establishments Processing Establishments

MT 54

Routine

Domestic Beef

Components

MT 50

Routine

Beef Trim

MT 44

Follow-up RGB MT 52

Trace-back to

Supplier-Trim 

and Components

MT 43

Routine Raw Ground Beef

MT 55

Routine

Bench Trim

+

+

+

+

+

+

+

Additionally, an extensive overview of the four E. coli O157:H7 verification projects under

consideration—MT43, MT50, MT54 and MT55—for generating prevalence estimates, including 

a discussion of the purpose of each project, their sampling properties and methodologies, as well 

as information about the FSIS collection methodology, mean response rates and analyzed 

samples is provided in the FSIS Sampling Program Report released in December 2011.

7

 

 

7 The FSIS Sampling Program Report was publicly released in December 2011 and can be accessed at the FSIS 

website at: http://www.fsis.usda.gov/PDF/FSIS_Sampling_Programs_Report.pdf.
17

Data Requirements

Prevalence can be estimated using a number of analytical approaches with a range of 

assumptions and varying degrees of complexity in calculation and maintenance. The following 

text summarizes the specific data requirements developed by the E. coli O157:H7 subgroup for 

prevalence estimation, together with a short discussion of whether the requirement is met for 

each of the four projects under consideration for use in deriving prevalence estimates (MT43, 

MT50, MT54 and MT55).

Population

Requirement

The universe of units for which the characteristic of interest is being assessed is required.

Requirement Met?

MT43: Information is available to estimate the annual production volume of raw ground 

beef in pounds for FSIS-inspected establishments. This information is available by volume 

category and is sufficient for deriving prevalence estimates. Hence, this requirement is 

satisfied.

MT50: Information is available to estimate the annual production volume of beef 

manufacturing trimmings in pounds for FSIS-inspected establishments. This information is 

available by volume category, and is sufficient for deriving prevalence estimates. Hence, 

this requirement is satisfied.

MT54: Information is available to estimate the annual production volume of raw ground 

beef components other than trim in pounds for FSIS-inspected establishments. This is an 

estimation of the group of products as a whole; this data is not available at the specificcomponent level. This information is available by volume category, and is sufficient for 

deriving prevalence estimates. Hence, this requirement is satisfied.

MT55: Information is available to estimate the annual production volume of bench trim in 

pounds for FSIS-inspected establishments. This information is available by volume 

category, and is sufficient for deriving prevalence estimates. Hence, this requirement is 

satisfied.

Sampling Frame

For all these projects, the population units are pounds of product and the enumeration units 

are establishments.

Requirement

A listing of all the units in the defined population is required. Alternatively, an aggregated 

listing of enumeration units could be used, provided that each population unit is associated 

with only one enumeration unit in the frame.

Requirement Met?

MT43: The sampling frame includes all establishments that report production of raw ground 

beef, as well as all establishments from which FSIS has collected and analyzed an MT43 

sample in the past 12 months. There is an exclusion list that, at the time this evaluation was 

conducted, contained approximately 40 establishments. These establishments generally are 
18

those from whom FSIS has collected a sample in the previous 12 months, but that are no 

longer producing ground beef. This list may sometimes also include establishments that have 

temporarily or seasonally stopped producing raw ground beef. These are all valid exclusions 

from the frame and do not affect the Agency’s ability to meet the requirement. Hence, this 

requirement is satisfied.

MT50: The sampling frame includes all establishments that slaughter beef or their identified 

sister establishments. Sister establishments are those that are directly, physically next door to 

or across the street from a slaughter establishment and both establishments are under 

common ownership. FSIS does not store data on corporate affiliations, so sister 

establishments are difficult to identify. Those identified by field inspectors are hard-coded as 

inclusions to the frame. Because not all beef slaughter establishments make trim intended for 

raw ground beef, FSIS recognizes that the defined frame is imprecise. For this reason, when 

a sample collection form is returned with a discard code indicating that the product is not 

produced, the establishment is removed from the sampling frame for 12 months. Also, FSIS 

maintains an exclusion list populated from communication with field inspectors indicating 

that the establishment does not belong in the frame. Some justifiable reasons to put an 

establishment on the exclusion list are that it produces seasonally and is not currently 

producing, the establishment does not produce any trim, the establishment diverts all of its 

trim to cooking, etc. At the time of this evaluation, there were approximately 12 

establishments on the exclusion list. These are all valid exclusions from the frame, but the 

ability to estimate prevalence would be improved by a more accurately defined frame. 

Hence, this requirement is satisfied.

MT54: The sampling frame includes all establishments that slaughter beef. Also, 

ammoniated beef establishments are included in this frame. Because not all beef slaughter 

establishments make components intended for raw ground beef, FSIS recognizes that the 

defined frame is imprecise. For this reason, when a sample collection form is returned with a 

discard code indicating that the product is not produced, the establishment is removed from 

the sampling frame for 12 months. Also, FSIS maintains an exclusion list populated from 

communication with field inspectors indicating that the establishment does not belong in the 

frame. Some justifiable reasons to put an establishment on the exclusion list are that it 

produces seasonally and is not currently producing, the establishment does not produce any 

components, the establishment diverts all of its components to cooking, etc. At the time of 

this evaluation, there were approximately 12 establishments on the exclusion list. These are 

all valid exclusions from the frame, but the ability to estimate prevalence would be improved 

by a more accurately defined frame. Hence, this requirement is satisfied.

MT55: The sampling frame includes establishments with indicators that they might produce 

bench trim in either the PBIS extension profile survey or the 2007 FSIS E. coli Checklist.8 

Both of these are now relatively outdated data sources, but there are currently no other 

identifiers in the FSIS data that could improve this frame. FSIS recognizes that the defined

frame is imprecise. For this reason, when a sample collection form is returned with a discard 

code indicating that the product is not produced, the establishment is removed from the 

sampling frame for 12 months. Also, FSIS maintains an exclusion list populated from 

 

8 Alvares, C., Lim, C., & Green, K. (August 2008). Results of checklist and reassessment of control for Escherichia 

coli in O157:H7 in beef operations. Accessed at: 

http://www.fsis.usda.gov/PDF/Ecoli_Reassement_&_Checklist.pdf on March 26, 2012.
19

communication with field inspectors indicating that the establishment does not belong in the 

frame. Some justifiable reasons to put an establishment on the exclusion list are that it 

produces seasonally and is not currently producing, the establishment does not produce any 

bench trim, the establishment diverts all of its products to cooking, the establishment only 

grinds, etc. At the time of this evaluation, there were approximately 30 establishments on the 

exclusion list. Field inspectors can request that an establishment be added to the frame. 

Ammoniated beef establishments are excluded because they are included with certainty in the 

MT54 frame. These are all valid exclusions from the frame, but the ability to estimate 

prevalence would be improved by a more accurately defined frame. Hence, this 

requirement is satisfied.

Collection Frame

Requirement

A listing of units in the collection frame is required. The collection frame could be the entire 

sampling frame or the set of establishments (or other enumeration units) from which samples 

are selected. The procedure used to set the frame must be defined.

Requirement Met?

MT43: Establishments are selected based on their production volume and their hazard score. 

FSIS has determined how the sampling algorithm should be utilized in calculating the hazard 

score to define the collection frame. If establishments are chosen with a definable 

probability from all hazard categories, then this requirement is met. The sampling frame 

currently equals the collection frame, less valid exclusions, hence this requirement is 

satisfied.

MT50: Establishments are selected by simple random sampling. The sampling frame 

currently equals the collection frame, less valid exclusions, hence this requirement is 

satisfied.

MT54: Establishments are selected by simple random sampling. At the time of this 

evaluation, the four ammoniated beef establishments regulated by FSIS were selected with 

certainty. FSIS maintains is a list of products that are eligible for collection, and this is not 

assigned by the sampling algorithm. Rather, the inspector is instructed to select the product 

for collection at random from the components available for selection on the day of collection. 

The sampling frame currently equals the collection frame, less valid exclusions, hence this 

requirement is satisfied.

MT55: Establishments are selected by simple random sampling. The sampling frame 

currently equals the collection frame, less valid exclusions, hence this requirement is 

satisfied.

Enumeration Unit Selection (Establishments)

Required

The procedure used to select enumeration units (establishments) from the collection frame 

must be defined. Establishments may be stratified based on a characteristic (e.g. risk, 

production volume) and then chosen based on an identifiable probability of selection within 

each stratum. Random unit choice is preferred.
20

Requirement Met?

MT43: Under the MT43 project, product samples are selected with replacement from the 

sampling frame by an algorithm that employs scaling factors. Inputs to calculating the 

scaling factors are annual production volume group and history of positive pathogen testing. 

The current project has four volume groups.9The algorithm also includes sampling ceilings 

for each volume group and sampling floors. Establishments with a history of positive test 

results are sampled more frequently. Hence, this requirement is satisfied.

MT50: Under the MT50 project, establishments are selected without replacement from the 

sampling frame by an algorithm that employs simple random sampling. There is no 

stratification or weighting by establishment production volume; there are no ceilings or 

floors. Hence, this requirement is satisfied.

MT54: Under the MT54 project, there were four establishments that could be identified with 

certainty when this evaluation was conducted. The remaining sample is selected without 

replacement from the sampling frame by an algorithm that employs simple random sampling. 

There is no stratification or weighting by establishment production volume; there are no 

ceilings or floors. Hence, this requirement is satisfied.

MT55: Under the MT55 project, establishments are selected without replacement from the 

sampling frame by an algorithm that employs simple random sampling. There is no 

stratification or weighting by establishment production volume; there are no ceilings or 

floors. Hence, this requirement is satisfied.

Population Unit Selection

Requirement

The procedure used to select population unit(s) from a given enumeration unit must be 

defined. Clustered samples are acceptable, but effect of clustered choice must be addressed 

through statistical analysis undertaken as part of the prevalence estimation. Announced 

sampling is not preferred. Announced samples may make clustered choice not acceptable. 

Day of week and time of day when sample is taken should be randomly chosen.

Requirement Met?

MT43: Sample collection is done by collecting 325 grams of raw ground beef. Each 

establishment is sampled a maximum of four times per month and at least three times per 

year. Because FSIS recommends that establishments hold sampled lots until FSIS laboratory 

tests confirm, field inspectors have to announce sample collection to allow the establishment 

to prepare to hold product.10 Although there is an element of announcement required for 

practical implementation purposes, the subgroup does not feel that it precludes the possibility 

 

9 Volume groups were developed by a multi-disciplinary team of scientists and technical staff within the FSIS prior 

to 2003. There are currently more volume groupings for MT43 sampling within PHIS, but they map exactly to the 

Agency’s existing PBIS categories. 

10 While FSIS does not currently mandate that establishments hold product until negative test results are received, 

the Agency requested comments on a Federal Register Notice that would change the Agency’s procedures and 

withhold a determination as to whether meat and poultry products are not adulterated, and thus eligible to enter 

commerce, until all test results that bear on the determination have been received. Please see the following website 

for more details: http://www.fsis.usda.gov/OPPDE/rdad/FRPubs/2005-0044.pdf.
21

of producing a prevalence estimate. This area may need further investigation. Hence, this 

requirement is satisfied.

MT50: Sample collection is done by N60 method.11 An N60 sample can vary in weight, but 

generally is between 325 grams (the goal) and 700 grams. Because FSIS recommends that 

establishments hold sampled lots until FSIS laboratory tests confirm, field inspectors have to 

announce sample collection to allow the establishment to prepare to hold product.12 Hence,

this requirement is satisfied.

MT54: Sample collection is done by N60 method when appropriate; otherwise, 325 grams of 

components are collected.13 An N60 sample can vary in weight, but generally is between 

325 grams (the goal) and 700 grams. More than one type of product qualifies for MT54 

collection, and field inspectors are instructed to select randomly from those being produced 

on the day of collection. Because FSIS recommends that establishments hold sampled lots 

until FSIS laboratory tests confirm, field inspectors have to announce sample collection to 

allow the establishment to prepare to hold product.14 Hence, this requirement is satisfied.

MT55: Sample collection is done by N60 method.15 An N60 sample can vary in weight but 

generally is between 325 grams (the goal) and 700 grams. Because FSIS recommends that 

establishments hold sampled lots until FSIS laboratory tests confirm, field inspectors have to 

announce sample collection to allow the establishment to prepare to hold product. Hence,

this requirement is satisfied.

Product Type Selection

Requirement

The procedure used to select the product type for sampling for a given product class must be 

defined. This applies to instances where various components make up the product class (e.g. 

components for raw ground product), or the final product itself varies. One hundred percent 

judgment based selection is not acceptable.

Requirement Met?

MT43: MT43 involves sampling of a single product (raw ground beef). Therefore, this 

requirement is not applicable to the MT43 project. 

MT50: MT50 involves sampling of a single product (manufacturing trimmings). Therefore, 

this requirement is not applicable to the MT50 project. 

MT54: Field inspectors are directed to select at random from the products available for 

collection on the date of collection. If it is one of the four ammoniated beef establishments

producing at the time of this evaluation, then the product collected is ammoniated beef. 

Hence, this requirement is satisfied.

MT55: MT55 involves sampling of a single product (bench trim). Therefore, this 

requirement is not applicable to the MT55 project. 

 

11 FSIS is in the process of identifying changes to the N60 sampling program in light of Office of Inspector General 

(OIG) audit recommendations. Changes made to the trim and component sampling programs to satisfy OIG 

requirements will be published on the FSIS website for public comment. 

12 Supra footnote 10. 

13 Supra footnote 11.

14 Supra footnote 10.

15 Supra footnote 11.
22

Probability of Selection

Requirement

The probability of selection for each population unit (e.g., unit of product) in each 

enumeration unit (establishment) must be determined. To do this, every enumeration unit 

(establishment) in the collection frame must have some probability of being sampled during a 

defined time period. Selection probabilities do not have to be equal.

Requirement Met?

MT43: The probability of selection for each population unit can be estimated from available 

FSIS data. Hence, this requirement is satisfied.

MT50: The probability of selection for each population unit can be estimated from available 

data. Hence, this requirement is satisfied.

MT54: The probability of selection for each population unit can be estimated from available 

data. Hence, this requirement is satisfied.

MT55: The probability of selection for each population unit can be estimated from available 

data. Hence, this requirement is satisfied.

Production Volume

Requirement

Production volume for each establishment for the time period in which prevalence is being 

calculated is required for an accurate estimate.

Requirement Met?

MT43: Production volume is available only in categories. These categories are sufficient for 

deriving a prevalence estimate. However, the breadth of these categories, as they are 

currently defined, may limit the precision of the prevalence estimates that can be derived 

from current information. Additional product volume categories will be included in PHIS, 

and may support more precise prevalence estimates. Hence, this requirement is satisfied.

MT50: Production volume is available only in categories. These categories are sufficient for 

deriving a prevalence estimate. However, the breadth of these categories, as they are 

currently defined, may limit the precision of the prevalence estimates that can be derived 

from current information. Additional product volume categories will be included in PHIS, 

and may support more precise prevalence estimates. Hence, this requirement is satisfied.

MT54: Production volume is available only in categories. These categories are sufficient for 

deriving a prevalence estimate. However, the breadth of these categories, as they are 

currently defined, may limit the precision of the prevalence estimates that can be derived 

from current information. Additional product volume categories will be included in PHIS, 

and may support more precise prevalence estimates. Hence, this requirement is satisfied.

MT55: Production volume is available only in categories. These categories are sufficient for 

deriving a prevalence estimate. However, the breadth of these categories, as they are 

currently defined, may limit the precision of the prevalence estimates that can be derived 

from current information. Additional product volume categories will be included in PHIS, 

and may support more precise prevalence estimates. Hence, this requirement is satisfied.
23

Analysis of Data Limitations

Based on review of the information outlined above, the E. coli O157:H7 subgroup has concluded 

that prevalence estimates can be developed for E. coli O157:H7 in raw ground beef (MT43). 

However, because of the small sample sizes and other factors, E. coli O157:H7 prevalence 

estimates cannot be developed for beef trim and components under the current verification 

projects. Key data, resource, and operational limitations are described below.

1. Risk-Based Sampling

The current scheduling algorithm for MT43 raw ground beef is risk-based, which is critical 

in monitoring establishment performance. The primary effect of this approach on prevalence 

estimation is that the risk categories have varying degrees of precision. The separate 

category variances affect the overall variance estimate because it is estimated by 

mathematically combining the separate variances. The ability to calculate a prevalence 

estimate remains, as long as all categories in a given time period are represented, though the 

precision of the estimate may be lower than if another design was employed.

2. Prior Notification

It is possible that prior notification affects FSIS’ ability to collect representative samples. 

However, policy constraints require that notification be given to establishments so that they 

can plan for holding product until FSIS laboratory test results are confirmed in an effort to 

prevent recalling product.16 To create an appropriate prevalence estimate, every attempt at 

representative sampling should be made. 

3. Sample Sizes for MT50, MT54, MT55 Projects

The most important aspect of creating a prevalence estimate is for the sampling to be done in 

a manner representative of the characteristics to be estimated. In addition, the precision of

the estimate may provide an indicator of its reliability. 

Precision usually improves as sample sizes increase. That is, larger sample sizes typically 

lead to smaller variances. In particular, rare event sampling requires large sample sizes to 

obtain reasonable precision. The MT50 project detected four positive trim samples out of 

1,274 analyzed samples in 2010, which gives an indication that this is rare event testing. 

Likewise, there were no positives out of the 169 analyzed MT54 samples in 2010 and no 

positives out of the 574 analyzed in MT55 samples in 2010, which indicates that these 

sample sizes may be too small to yield an accurate prevalence estimate in these products. 

4. Representativeness of the Samples

For MT50, MT54 and MT55, the sampling results are representative of sampling in

establishments, but prevalence estimates require representative sampling of the product in 

question. Because the sample designs do not incorporate stratification or weighting by 

production volume, the samples may not be adequately representative of product from each 

production class.

 

16 Supra footnote 10. 
24

5. Industry Testing Affecting FSIS Estimates

Industry test and divert practices may result in a lower prevalence estimate obtained by FSIS 

verification testing than would be obtained through baseline testing, because a portion of 

positive product would already be removed.17 While this product is not a threat to consumers 

because it is not sent into commerce, test and divert could result in FSIS verification 

prevalence estimates that are lower than baseline prevalence estimates.

Conclusions

The E. coli O157:H7 subgroup has concluded the following with respect to the estimation of the 

prevalence of E. coli O157:H7 in raw ground beef, beef trim and beef components:

Raw ground beef: It is possible to develop national prevalence estimates for E. coli 

O157:H7 in raw ground beef using the currently available data. Improved prevalence 

estimates might be obtained through implementation of changes to the current 

verification testing project.

Beef trim and beef components: Development of national prevalence estimates for E. coli

O157:H7 in beef trim and components using current FSIS pathogen testing data is not 

possible at the current time. Prevalence estimates could be developed, but would require 

substantial increases in sample size during a period of fiscal restraint. In addition, any 

changes to these sampling projects to support the development of prevalence estimates 

will need to be coordinated with other possible changes to the current 

MT50/MT54/MT55 projects, such as FSIS’ activities in relation to the OIG N60 audit 

recommendations. 

 

17 Supra footnote 10. 
25

Chapter 3: Salmonella Verification Sampling Program
26

Overview 

Calculating an accurate prevalence estimate using the current Salmonella verification data is not 

possible because certain key elements in the data requirements are not being met. Table 4 

provides an overview of the data requirements evaluated and the findings of the Salmonella 

subgroup in terms of current Salmonella pathogen verification testing projects.

Table 4: Overview of FSIS Findings Regarding Evaluation of Data Requirements of Salmonella 

Pathogen Verification Testing Projects18

Data and Statistical 

Requirements

Salmonella 

Raw Intact 

Product

Raw Ground 

Product

Population Yes No

Sampling Frame Yes

Collection Frame Yes

Enumeration Unit Selection No

Population Unit Selection No

Product Type Selection Yes

Probability of Selection No

Production Volume Yes No

CONCLUSIONS NO NO

Background

The Salmonella verification sampling program formally began with FSIS Final Rule on Pathogen 

Reduction; Hazard Analysis and Critical Control Point (PR/HACCP) Systems that issued on July 

25, 1996 (61 FR 38805– 38989). Among other things, the PR/HACCP rule set Salmonella 

performance standards for establishments that slaughter selected classes of food animals or that 

produce selected classes of raw ground products. FSIS uses the Salmonella performance 

standards to ensure that each establishment is consistently achieving an acceptable level of 

performance with regard to controlling processes and reducing harmful bacteria on raw meat and 

poultry products.19 

This section presents a brief overview of FSIS Salmonella sampling projects, specific data 

requirements for possible prevalence estimation, and a summary of the Salmonella subgroup’s 

conclusions regarding the suitability of using verification data to estimate prevalence. 

Table 5 describes the various FSIS Salmonella sampling verification projects. Additionally, an 

extensive overview of the Salmonella sampling projects under consideration for generating 

prevalence estimates, including a discussion of the purpose of the project, their sampling 

properties and methodologies, as well as information about the FSIS collection methodology,

 

18 Sampling for Salmonella through the MT43S project was not included in this assessment. 

19 Federal Register, Vol. 73, No. 18, July 25, 1996.
27

mean response rates, and counts of analyzed samples is provided in the FSIS Sampling Program 

Report.20 

Table 5: FSIS Salmonella Verification Sampling Projects. 

Product class Salmonella

Sampling 

Projects

Number of 

Salmonella

Samples 

Analyzed 

FY2010

Regulatory Purpose of

Sampling Project

Type of 

Sampling 

Project

Steers/heifers21 HC01 6,550 Verify consistent process 

control Risk Based

Cows/bulls22 HC01 1,688 Verify consistent process 

control Risk Based

Raw Ground beef23 HC01 8,982 Verify consistent process 

control Risk Based

Market hogs24 HC01 305 Verify consistent process 

control Risk Based

Broilers25, 26 HC01 762 Verify consistent process 

control Risk Based

Ground chicken27 HC01 3,913 Verify consistent process 

control Risk Based

Ground turkey28 HC01 3,811 Verify consistent process 

control Risk Based

Turkeys29 HC01 1,303 Verify consistent process 

control Risk Based

Raw Ground beef MT43S 2,957 Verify consistent process 

control Random

RTE meat and poultry 

products ALLRTE 2,990

Verify adequacy of an 

establishment's ability to 

prevent microbiological 

contamination

Random

RTE meat and poultry 

products RTE001 8,700

Verify adequacy of an 

establishment's ability to 

prevent contamination 

from Salmonella and Lm

Risk Based

 

20 Supra footnote 7. 

21 Sample sets for Market Hogs, Cows/Bulls, or Steers/Heifers were not scheduled in the latter half of FY 2011.

22 Supra footnote 21

23 Sampling for Salmonella through the FSIS MT43S sampling program (Salmonella sampling in raw ground beef 

product) was not included in this assessment.

24 Supra footnote 21. 

25 No longer being scheduled for HC01 (but being scheduled under HC11) with the implementation of the new 

Salmonella and Campylobacter performance standards in July 2011.

26 FSIS initiated sampling with enumeration for Campylobacter in July 2011 in turkeys and broilers. FSIS will 

assess the ability to estimate prevalence using Campylobacter sampling results once data collection and analysis has 

been in place for at least a year period. 

27 In CY2012, FSIS anticipates issuing a Federal Register Notice informing stakeholders that the Agency intends to 

begin sampling comminuted and ground poultry products for Salmonella and Campylobacter during CY2012.

28 Supra footnote 27. 

29 Supra footnote 25
28

Product class Salmonella

Sampling 

Projects

Number of 

Salmonella

Samples 

Analyzed 

FY2010

Regulatory Purpose of

Sampling Project

Type of 

Sampling 

Project

Egg whites with or 

without added 

ingredients

EM-31 292

Verify adequacy of an 

establishment’s ability to 

prevent contamination 

from Salmonella

Random

Whole eggs/yolks with 

<2% added ingredients 

other than salt or sugar

EM-32 389

Verify adequacy of an 

establishment’s ability to 

prevent contamination 

from Salmonella

Random

Whole eggs/yolks with 

≥2% added ingredients 

other than salt or sugar

EM-33 141

Verify adequacy of an 

establishment’s ability to 

prevent contamination 

from Salmonella

Random

Whole eggs/yolks with 

≥2% salt or sugar added EM-34 287

Verify adequacy of an 

establishment’s ability to 

prevent contamination 

from Salmonella

Random

Dried yellow egg 

products EM-35 114

Verify adequacy of an 

establishment’s ability to 

prevent contamination 

from Salmonella

Random

Spray dried egg whites 

(with or without added 

ingredients)

EM-36 104

Verify adequacy of an 

establishment’s ability to 

prevent contamination 

from Salmonella

Random

Pan dried egg whites EM-37 10

Verify adequacy of an 

establishment’s ability to 

prevent contamination 

from Salmonella

Random

Domestic liquid, frozen

or dried egg products EGGDOM 61

Verify adequacy of an 

establishment’s ability to 

prevent contamination 

from Salmonella

Random

Data Requirements

The Salmonella subgroup established that certain key elements are necessary in order to 

accurately estimate prevalence. These requirements must be met for any calculations to be 

statistically valid.

Population

Required

The universe of units for which the characteristic of interest is being assessed is required.
29

Requirement Met?

The population of interest for raw intact products is the number of carcasses of a certain 

product class (broilers, turkeys, cows/bulls, steer/heifer, and market hog) at post-chill 

produced at non-exempt FSIS regulated establishments.30, 31 This volume data is readily 

available from the FSIS Electronic Animal Disposition Reporting System (eADRS) slaughter 

database, which sufficiently satisfies this requirement for raw intact products. 

The population of interest for raw ground products is the volume of a certain product class 

(ground beef, chicken, or turkey) produced at non-exempt FSIS regulated establishments.32 

Accurate volume data is not available for these products at this time. However, a rough 

estimate for raw ground beef can be calculated. Therefore, this requirement is not met for 

raw ground products.

Sampling Frame

Required

A listing of all the units in the defined population is required. Alternatively, an aggregated 

listing of enumeration units can be used, provided that each population unit is associated 

with only one enumeration unit in the frame. For example, if the population unit is an 

individual FSIS inspected steer or heifer carcass, then the list of non-exempt federally 

inspected establishments that slaughter steers and heifers is an appropriate listing of 

enumeration units.

Requirement Met?

The set of all non-exempt federally inspected establishments that produce a given product 

class during a given period of interest can be rapidly assembled from data currently existing 

in the FSIS data warehouse; hence this requirement is satisfied.

Collection Frame

Required

A listing of units in the collection frame is required. This could be the entire sampling frame, 

or a subset of enumeration units from which samples are selected. The methodology used to 

set the frame must be defined and statistically valid for a prevalence calculation.

Requirement Met?

The current collection frame does not include all the establishments from the sampling frame.

For example, low volume producers have traditionally been excluded for sampling rate 

reasons. Please see the FSIS Standard Operating Procedure (SOP) for Eligibility.33 Since 

low volume establishments comprise such a small percentage of industry production, the 

statistical impact of excluding them is minimal (this method was also used in the 2007-2008 

 

30Some FSIS inspected establishments have a religious exemption and are not sampled, so carcasses produced at 

these establishments should be excluded from the population.

31 Supra footnote 21

32 Supra footnote 30.

33 Please see the following website for more information: 

http://www.fsis.usda.gov/PDF/SOP_Salmonella_Eligibility_Testing_092211.pdf.
30

FSIS Nationwide Microbiological Baseline Data Collection Program: Young Chicken 

Survey).34 Hence, this requirement is satisfied.

Enumeration Unit Selection

Required

The procedure used to select enumeration units (establishments) from the collection frame 

must be defined. Establishments may be stratified based on a characteristic (e.g. risk, 

production volume) and then chosen based on an identifiable probability of selection within 

each stratum. Random unit choice is preferred.

Requirement Met?

Establishments are currently chosen for sampling based on risk and past performance. New 

establishments and those in the highest risk category are selected first. All product classes 

are part of the same scheduling algorithm so establishments producing certain product classes 

are given higher priority in scheduling over other products. Please see criteria for selection 

and product class priority.35 Thus, FSIS does not control the number of verification samples 

for specific product classes over the course of time. This method is not ideal for 

prevalence estimation, so the requirement is not fulfilled.

Population Unit Selection

Required

The procedure used to select population unit(s) from a given enumeration unit must be 

defined. Clustered samples are acceptable, but effect of clustered choice must be addressed. 

Announced sampling is not preferred. Announced samples may make clustered choice 

unacceptable. Day of week and time of day when a sample is collected should be randomly 

chosen.

Requirement Met?

Current policy specifies that HC01 samples are to be collected in sets. Once an 

establishment is chosen for sampling, it is sent 75 sample forms.

36 Field inspectors are 

instructed to collect one sample per day for each day that the establishment is producing the 

product. Sample collection ends when the number of samples successfully analyzed reaches 

the number required to complete a set—see Appendix C for a complete description. Thus, 

samples are both clustered and announced. The day of the week when the sample is 

collected is not random or varied because once a sample set is started, an establishment is 

aware samples will be collected consecutively every day it is producing that product for a 

given length of time. Nevertheless, FSIS field inspectors are instructed to collect samples at 

varied times of the day, though information regarding the distribution of collection times is 

not available. Given the information above, this element does not meet the requirement 

necessary for prevalence estimation.

 

34 Please see the following website for more information: 

http://www.fsis.usda.gov/PDF/Baseline_Data_Young_Chicken_2007-2008.pdf.

35Please see the following website for more information:

http://www.fsis.usda.gov/Science/Scheduling_Criteria_Salmonella_Sets_092211/index.asp.

36 Historically, steer/heifer establishments were sent 90 forms.
31

Product Type Selection

Required

The procedure used to select the product type for sampling for a given product class must be 

defined. This applies to instances where various components comprise the product class. 

For example, trim and components for raw ground beef (which are not currently sampled 

under Salmonella verification testing) might consist of cheek meat, weasand meat and/or

heart meat, etc. This requirement could also apply if the final product itself varies.

Requirement Met?

Each class of raw intact product is sampled separately and consists of only one type, not 

parts. Also, all ground products are sampled after grinding (not components), so no sample 

selection procedure is necessary. Thus, this data requirement is met.

Probability of Selection

Required

The probability of selection for each population unit in every enumeration unit must be 

determined. To do this, every enumeration unit in the collection frame must have some 

probability of being sampled each sampling period. Probabilities do not have to be equal 

across enumeration units or over time.

Requirement Met?

These probabilities cannot be calculated in the current risk-based sampling algorithm because 

not all establishments (enumeration units) in the sampling frame have a probability of being 

selected each sampling period (month). Therefore, this element is not satisfactory for 

calculating prevalence.

Production Volume

Required

Production volume for each enumeration unit for the time period in which prevalence is 

being calculated is required for an accurate estimate.

Requirement Met?

The eADRS slaughter database contains detailed volume data of carcasses slaughtered for 

each product class for every establishment in the sampling frame. Volume totals are 

available for any time period (daily, weekly, monthly, annually, etc.) so this requirement is 

met for raw intact products. Unfortunately, exact volume data is not available for ground 

products, so the requirement is not satisfied for those product classes.37

Analysis of Data Limitations

After reviewing all available information, the Salmonella subgroup determined that prevalence 

cannot be estimated using the current verification data. There were several significant reasons 

for this assessment.

 

37 For raw ground beef, the E. coli O157:H7 subgroup decided that the production volume groups were not ideal, but 

satisfactory for calculating prevalence.
32

1. Risk-based Sampling

The current scheduling algorithm is risk-based, which is critical in positively affecting public 

health, but disproportionately focuses sample collection. This means that there is a large 

differentiation between well-performing establishments (Category 1) and poor, or potentially 

poor ones (Category 3), in that the former might not be scheduled for sampling for a year or 

more, whereas the latter could be scheduled quite often.38 For this reason, not all 

establishments in the collection frame have a known probability of selection each month. 

2. Product Priority

Establishments producing certain products are scheduled ahead of others. This prevents 

those establishments/products with lower priority from being sampled regularly because only 

a given number of sample sets can be scheduled each month. This results in insufficient data 

that is not representative of certain product classes. Furthermore, some product classes that 

have been completely excluded during certain months would not have a probability of 

selection for sampling for that period.

3. Announced Sampling

Once a sample set begins, an establishment is aware that it will be sampled every day the 

product is being produced over the next few months (or longer for smaller establishments

that produce less frequently) until the set is completed. This knowledge might create a bias 

because establishments may, intentionally or not, be more conscientious in adhering to 

proper sanitary procedures during this time. This could result in an abnormally low number 

of positive Salmonella results than would occur otherwise, and any prevalence calculation 

would be underestimated.

4. Sample Sets

Salmonella samples are scheduled in sets, which results in a high degree of clustering. That 

is, establishments are sampled intensively and then not at all for a period of time. Moreover, 

with regard to the above analysis of the risk-based sampling algorithm, this is problematic 

from a process control perspective because there are no available data over a long period for 

so-called well-performing (Category 1) establishments. Thus, it is unknown whether these 

establishments are consistently maintaining good process control, or if their good 

performance was a temporary result of announced sampling.

 

38 In June 2006, FSIS began employing a ―category‖ system to measure establishments’ Salmonella performance 

due to a change in how the establishments were selected for testing. Category 1 represents establishments that have 

achieved 50 percent or less of the performance standard or baseline guidance, for two consecutive FSIS test sets. 

Category 2 represents establishments that have achieved greater than 50 percent on at least one of the two most 

recent FSIS test sets without exceeding the performance standard or baseline guidance. Category 3 represents 

establishments that have exceeded the performance standard or baseline guidance on the most recent FSIS test set. 

FSIS has developed new tightened performance guidance based on results from the year long Nationwide Young 

Chicken Microbiological Baseline completed in mid- 2008. A Federal Register Notice announcing the new guidance 

policies was published in May 2010 and the standards were implemented in July 2011 and the ―Category 1‖ 

performance standard was modified to reflect the new standards. 
33

5. Production Volume

The major difference between the sampling data for intact and ground products is that 

volume information is not available at the establishment level for ground chicken or ground 

turkey, and only a rough estimate can be determined for ground beef. This data must be 

obtained to calculate a prevalence estimate in those product classes.

Conclusion

The Salmonella subgroup has determined that calculating an accurate prevalence estimate using 

the current verification data is not possible because certain key elements in the data 

requirements are not met. 
34

Chapter 4: Ready-To-Eat (RTE) Sampling Program

Pathogen Verification for Lm and Salmonella
35

Overview

Calculating an accurate Lm or Salmonella prevalence estimate using the current RTE testing 

project verification data it is not possible because certain key elements in the data requirements 

are not being met. Table 6 provides an overview of the data requirements evaluated and the 

findings of the RTE subgroup in terms of current RTE pathogen verification testing program.

Table 6: Overview of FSIS Findings Regarding Evaluation of Data Requirements of Current 

RTE Pathogen Verification Testing Projects

Data and Statistical 

Requirements

RTE

RTE001 ALLRTE

Population Yes No

Sampling Frame Yes

Collection Frame No Yes

Enumeration Unit Selection No Yes

Population Unit Selection No Yes

Product Type Selection No Yes

Probability of Selection No Yes

Production Volume Yes No

CONCLUSION NO NO

Note: RTE001 and ALLRTE are being considered as separate sampling projects.

Background

FSIS has conducted a regulatory microbiological testing program in RTE meat and poultry 

products since 1983. From 1983 until 2004, establishments were randomly selected for 

regulatory samples from different sub-populations or from the total population of establishments 

producing RTE products.39 Lm has been implicated in illness outbreaks since the early 1980s. In 

1987, FSIS increased testing for Lm in regulated products, including domestic cooked meat and 

poultry and imported cooked products.40 In 1989, after a confirmed human listeriosis case linked 

to cooked poultry, FSIS identified Lm as an adulterant subject to recall if found in commerce.41 

After the implementation of FSIS PR/HACCP regulations in 1996,42 FSIS organized Lm testing 

around the four HACCP processes of 1) fully cooked, not shelf stable products, 2) heat-treated, 

shelf stable products, 3) not heat-treated, shelf stable products and 4) products with secondary 

inhibitors that are not shelf stable. Thus, FSIS began random testing of RTE product samples in 

the 1990s, while risk-based testing of RTE products for Lm began in 2005. Since the inception 

of the Lm verification testing program for RTE meat and poultry products, FSIS has also 

sampled packaged RTE products for the presence of Salmonella. 

 

39 Please see the following website for more information: 

http://www.fsis.usda.gov/Science/Micro_Testing_RTE/index.asp.

40 Federal Register, Volume 52, No. 47, March 11, 1987.

41 Federal Register Volume 54, No. 98, Tuesday May 23, 1989.

42 Pathogen Reduction/Hazard Analysis and Critical Control Point System final rule (61 FR 38806, July 25, 1996).
36

The ALLRTE sampling project for Lm, which began in 2004, was designed to obtain random 

samples across the full range of RTE products and across all establishments producing a RTE 

product, regardless of risk, with the intention of estimating the prevalence of Lm. The risk-based 

sampling project (RTE001) began in 2005 with the intention of identifying and sampling RTE

establishments according to risk as defined by the interim final Lm rule (9 CFR 430). Only RTE 

establishments with exposure of products to the environment subsequent to a lethality treatment 

(i.e., cooking, fermentation, curing or drying), otherwise known as post-lethality exposure, are 

sampled in the RTE001 sampling project. An Lm risk-ranking algorithm is used to select 

establishments with the highest risk rankings for risk-based RTE001 sampling each month. Field 

inspectors are instructed to collect the riskiest RTE product samples produced in the 

establishment at the time of collection. 

Beginning in FY2008, FSIS began using the volume weighted Lm percent positive from the 

RTE001 project as a performance measure RTE001 collects more samples than ALLRTE. The 

Lm percent positive estimate from the ALLRTE project continued to be estimated as before. 

However, because of the judgmental selection procedure of establishments and products, the 

estimates derived from the RTE001 data could be biased with respect to the percent positive over 

all RTE products. Consequently, the decision was made to continue both projects, unchanged 

from FY2008, for comparative purposes.

Also, beginning in FY2008, FSIS began sampling all establishments in the RTE001 program so 

that no establishment producing post-lethality exposed RTE products would miss a sample result 

over single year. This was accomplished by retaining the same sample allocation, but reducing 

the number of risk-based samples by approximately 25% and allocating the difference to random 

sampling of establishments not selected for risk-based sampling.

This document presents a brief overview of FSIS RTE sampling projects, specific data 

requirements for possible prevalence estimation, and a summary of the RTE subgroup’s 

conclusions regarding the suitability of using verification data to estimate prevalence. Sampling 

projects for Lm and Salmonella in domestic establishments that produce RTE meat and poultry 

products are summarized in Table 5. Additionally, an extensive overview of the RTE sampling 

projects under consideration for generating prevalence estimates, including a discussion of the

purpose of each project, sampling properties and methodologies, as well as information about the 

FSIS collection methodology, mean response rates and analyzed samples is provided in the FSIS 

Sampling Program Report.43 

 

43 Supra footnote 7. 
37

Table 5: FSIS RTE Domestic Sampling Projects

Product class RTE 

Sampling 

Projects

Pathogens 

Tested

Number 

of 

FY2010

Samples 

collected

Regulatory 

Purpose of

Sampling Project

Type of 

Sampling 

Project

Post-lethality exposed and 

non-post-lethality exposed 

RTE products

ALLRTE Lm, 

Salmonella

44 2,990 Monitor industry 

performance Random

Post-lethality exposed 

RTE products RTE001 Lm, 

Salmonella

8,700 Verify nondetectable standard Risk Based

RLm product samples RLMPRO Lm 1,960 Monitor industry 

performance Risk Based

RLm food contact surface

samples RLMCONT Lm 6,600 Monitor industry performance Risk Based

RLm non-food contact 

environ. samples 

(Composit. 5-sample 

Units; Lm)

RLMENVC Lm 690 Monitor industry 

performance Risk Based

Intensified Verification 

Testing (IVT) product 

samples

INTPROD Lm or

Salmonella 225

Response to 

positive ALLRTE, 

RTE001, 

RLMPROD and/or 

RLMCONT sample

For Cause

IVT food contact surface 

samples INTCONT Lm or

Salmonella 550

Response to 

positive ALLRTE, 

RTE001, 

RLMPROD and/or 

RLMCONT sample

For Cause

IVT non-food contact 

environmental samples INTENV Lm or

Salmonella 275

Response to 

positive ALLRTE, 

RTE001, 

RLMPROD and/or 

RLMCONT sample

For Cause

Data Requirements

The following provides a summary of the RTE subgroup’s evaluation of the current Lm and 

Salmonella pathogen testing projects in light of the data and statistical requirements identified by 

the DCC Prevalence Workgroup. 

Population 

Required

The universe of units for which the characteristic of interest is being assessed is required. 

 

44 In addition to Lm and Salmonella, testing for E. coli O157:H7 was performed for specific product types, notably, 

dry and semi-dry fermented sausages and fully cooked meat patties until April, 2011. This testing was discontinued 

after an analysis showed that testing over 10,000 such products for E. coli O157:H7 over a nine-year period yielded 

no positive samples.
38

Requirement Met?

The population of interest for RTE meat and poultry products is the volume of a certain 

product produced at non-exempt FSIS regulated establishments. Presently, the individual 

annual product distributions for such product groups as deli meat, hot dogs, and fermented 

sausage can be derived from the FSIS Form 10,240-1, which contains product volume data 

for establishments producing post-lethality exposed RTE products. RTE establishments 

without these data must use production volume estimates based on HACCP establishment 

size. However, when PHIS is fully implemented, comparable data needs to be readily 

available for all RTE establishments. Estimates of inspector generated and certified daily 

production volumes must be determined for each establishment. In order to estimate 

prevalence on a per pound product basis, it is necessary to know the number of pounds 

corresponding to positive results, as well as the total number of pounds produced. The lot 

size in pounds should provide this information (which also should be inspector-certified). 

With these data, the selection probabilities for these products can be determined for each 

establishment. Since it is required that field inspectors report production volumes as 

predetermined range values, there will likely be increased uncertainty in prevalence estimates 

performed using PHIS data, as compared to estimates made from FSIS Form 10,240-1 annual 

point estimates. Consequently, the requirement is met for the RTE001 project, but not 

for the ALLRTE project. This is because RTE001 establishments (all those with postlethality exposure) have production volume data from FSIS Form 10,240-1, whereas 

ALLRTE establishments with no post-lethality exposed products are not required to 

report production volume data. However, the requirement will be met for both 

projects once inspection in all RTE establishments will be able to report production 

volume data through PHIS.

Sampling Frame

Required

A listing of all the units in the defined population is required. Alternatively, an aggregated 

listing of enumeration units can be used, provided that each population unit is associated with 

only one enumeration unit in the frame. For example, if the population unit is an RTE meat 

or poultry product, then the list of non-exempt federally inspected establishments that 

produce RTE products is an appropriate listing of enumeration units. In a multiple stage 

sampling plan, these establishments will be the primary sampling units in the first sampling 

stage.

Requirement Met?

The current Lm risk ranking algorithm probabilities that encompass listeriosis cases at 

product consumption can be used as the basis for establishment selection over the entire RTE 

establishment sampling frame, provided appropriate data are available. Furthermore, within 

the establishment, second stage random selection of products sampling can be made based on 

the distribution of products produced.45 Monthly sampling frames may be combined to make 

 

45 FSIS defines first stage sampling as including those establishments that are selected for Agency sampling in any 

given month, while second stage sampling is defined as including those products that are selected within 

establishments that are selected for Agency sampling.
39

quarterly and annual prevalence estimates using appropriate statistical models.

Consequently, the requirement is met for both projects, with the understanding that the 

sampling frame for RTE001 is only for establishments with post-lethality exposure. 

Enumeration Unit Selection

Required

The procedure used to select enumeration units (establishments) from the collection frame 

must be defined. Establishments may be stratified based on a characteristic (e.g. risk, 

production volume) and then chosen based on an identifiable probability of selection within 

each stratum. Random unit choice is preferred, but risk-based sampling is acceptable if all 

selection probabilities can be determined. The enumeration units should come from the 

population of establishments from which all RTE products are produced.

Requirement Met?

Establishments are currently chosen for Lm sampling based on random selection for the 

ALLRTE and based on risk for the RTE001 project so the selection method is acceptable. 

However, since the ALLRTE collection frame includes the RTE001 collection frame the 

enumeration units cannot be the same because RTE001 lacks enumeration units present in 

ALLRTE which contains all the enumeration units. Consequently, the requirement is not 

met for the RTE001 project, but is met for the ALLRTE project.

Collection Frame

Required

A listing of units in the collection frame is required. This could be the entire sampling frame, 

or a subset of enumeration units from which samples are selected. The methodology used to 

set the frame must be defined and statistically valid for a prevalence calculation.

Requirement Met?

Presently, for the risk-based RTE001 sampling project, field inspectors are instructed to 

collect the riskiest RTE product samples produced in the establishment at the time of 

collection. These second stage collection procedures within a given establishment will need 

to be changed, eliminating the judgmental sample selection and implementing random 

sampling of different products over time, though there could still be some dependence on 

product risk. The ALLRTE sampling project is already established as a random sampling 

project. Consequently, the requirement is not met for the RTE001 project, but is met 

for the ALLRTE project.

Population Unit Selection

Required

The procedure used to select population unit(s) from a given enumeration unit must be 

defined. Day of week and time of day when a sample is collected should be randomly 

chosen. Clustered samples are acceptable, but effect of clustered choice must be addressed. 

Announced sampling is not preferred. Announced samples may make clustered choice 

unacceptable. 
40

Requirement Met?

Current FSIS policy specifies that ALLRTE and RTE001 samples are to be collected 

individually and independently. Consequently, this requirement is met for ALLRTE, but 

not for RTE001 because RTE001 sample selection is risk-based and not random.

Product Type Selection

Required

The procedure used to select the product type for sampling for a given product category must 

be defined. 

Requirement Met?

There have been seven RTE product categories being used in PBIS for both ALLRTE and 

RTE001 sampling (deli sliced, deli unsliced, hot dogs, cooked products, fermented products, 

dried products, salt-cured products, frozen products, and pate/meat spreads). Under PHIS, 

there are four basic product categories: 1) Acidified/fermented, 2) salt-cured, 3) dried and 4) 

fully cooked. The first three categories are divided into unsliced and deli sliced product 

types, for a total of six product types among these three categories. Within the category of 

fully cooked, there are eight product types: 

1. RTE Fully Cooked Hotdog Products

2. RTE Fully Cooked - Other/Unsliced at establishment

3. RTE Fully Cooked - Deli Sliced 

4. RTE Fully Cooked Salad/Spread/Pate

5. RTE Fully Cooked Meat and Non-meat Multi-component

6. RTE Fully Cooked Sausage Products

7. RTE Fully Cooked Diced/Shredded

8. RTE Fully Cooked Patties/Nuggets

Thus, there are a total of four PHIS product categories and 14 PHIS product types within 

those categories that require sampling for Lm. Each type of RTE meat and poultry product is 

sampled separately and consists of only intact packages. Consequently, the requirement is 

not met for the RTE001 project (because it is not presently random), but is met for the 

ALLRTE project.

Probability of Selection

Required

The probability of selection for each population unit in every enumeration unit must be 

determined. To do this, every enumeration unit in the collection frame must have some 

probability of being sampled each sampling period. Probabilities do not have to be equal 

across enumeration units or over time.

Requirement Met?

In the existing risk-based sampling algorithm, these probabilities cannot be calculated 

because not all establishments (enumeration units) in the sampling frame including all RTE 

establishments have a probability of being selected each sampling period (month). 
41

Consequently, the requirement is not met for the RTE001 project, but is met for the 

ALLRTE project.

Production Volume

Required

Production volume for each enumeration unit for the time period in which prevalence is 

being calculated is required to estimate prevalence for the population of interest.

Requirement Met?

When changing from FSIS’ PBIS data to PHIS, future volume data to be collected for all 

RTE establishments will need to be converted to annual or monthly total pounds production. 

Ideally, estimates of actual volume, not a range are needed, but this is not possible because of 

the proprietary nature of the actual production volumes within each establishment. Thus, 

future information may not be sufficient for making estimates of prevalence without a high 

degree of uncertainty. Consequently, the requirement is met for the RTE001 project, but 

is not met for the ALLRTE project.

Analysis of Data Limitations

FSIS believes that it is not possible to use the Agency’s RTE sampling program data to estimate

a statistically derived national Lm prevalence. To accomplish this, certain changes in the 

selection procedure, particularly for the RTE001 project, are needed. However, certain 

limitations must be acknowledged:

1) Sample collection instructions

Current sampling instructions to FSIS field personnel influence the types of RTE samples 

that are collected and therefore the ability to estimate prevalence from current sampling 

data. Further, the modifications needed to achieve random product sampling, such as 

instructing field inspectors regarding probability proportional to size sampling.

2) Announced sampling

Establishments are aware that they will be sampled for particular RTE products, which 

might create a bias because establishments may, intentionally or not, be more 

conscientious in adhering to proper sanitary procedures during this time. This could 

result in an abnormally low number of positive Lm and/or Salmonella results than would 

occur otherwise, and any prevalence calculation would be underestimated.

3) Point estimates of actual volume (not a range). When changing from PBIS data to PHIS 

the volume data that will be collected for all RTE establishments is in daily volume 

ranges and not readily convertible to annual, or monthly total pounds production. The 

information being planned for will not be sufficient for making estimates of prevalence 

without hard to quantify uncertainty. 

Conclusion

The Lm RTE subgroup has determined that calculating an accurate prevalence estimate using the 

current verification data is not possible because certain key elements in the data requirements 

are not met. 
42

Appendix A: Definition of Terms

Data Warehouse: FSIS collects numerous types of data from a variety of different sources. This 

data is stored in an electronic ―warehouse,‖ known as the FSIS Data Warehouse (DW).

Discard Code: When FSIS inspectors are not able to collect a specific sample for a pathogen 

verification testing project, particular discard codes are used on sampling forms returned to FSIS 

laboratories indicating when the sample was not collected.

Exclusion Criteria: Exclusion criteria are the standards FSIS uses to determine whether an 

establishment should be included in the sampling frame. For example, establishments that 

produce a very low volume of product may be excluded from the sampling frame. Therefore, 

producing low volume is the exclusion criterion. 

Random Sampling: A random sample is one chosen by a method involving an unpredictable 

component. Random sampling can also refer to taking a number of independent observations 

from the same probability distribution, without involving any real population. 

Replacement: When a sampling unit is drawn from a finite population and is returned to that 

population, after its characteristic(s) have been recorded, but before the next unit is drawn, the 

sampling is said to be ―with replacement.‖ In the contrary case, the sampling is ―without 

replacement.‖ A different usage occurs in sample surveys when samples are taken on successive 

occasions. If the same members are used for successive samples there is said to be no 

replacement; but if some members are retained and others are replaced by new individuals there 

is said to be ―partial replacement‖.46

Risk Based Sampling: A sampling plan in which establishments are sampled at a greater or lesser 

frequency based on the risk the establishment poses. For example, establishments that have 

fewer positive pathogen test results might be considered to be low risk and are therefore sampled 

less frequently than establishments that have more positive pathogen test results.

Percent Positive: The percentage of positive samples is expressed as a percentage, determined as 

the number of positive samples for the pathogen per the total number of samples tested, 

multiplied by 100. The expected value of this percentage in this document is called ―the percent 

positive.‖47

Performance Based Sampling: A sampling plan in which establishments are sampled at a greater 

or lesser frequency based on their performance. For example, establishments that have fewer 

positive pathogen test results might be considered to be high performers and are therefore 

sampled less frequently than establishments that have more positive pathogen test results.

Sample Ceiling: The maximum number of samples in a sampling frame. 

 

46 Please see the following website for more information: http://stats.oecd.org/glossary/detail.asp?ID=3835.

47Please see the following website for more information: 

http://www.fsis.usda.gov/PDF/Draft_Guidelines_Sampling_Beef_Trimmings_Ecoli.pdf.
43

Sampling Frame48: Sampling frame is the actual set of units from which a sample has been 

drawn. In the case of a simple random sample, all units from the sampling frame have an equal 

chance to be drawn and to occur in the sample. In the ideal case, the sampling frame should 

coincide with the population of interest.

Sample Floor: The minimum number of samples in a sampling frame. 

Sample Size: The sample size of a statistical sample is the number of observations that constitute 

it. It is typically denoted n, a positive integer. The sample size is an important feature of any 

empirical study in which the goal is to make inferences about a population from a sample. In 

practice, the sample size used in a study is determined based on the cost of data collection, and 

the need to have sufficient statistical power. In a census, data are collected on the entire 

population; hence the sample size is equal to the population size. Larger sample sizes lead to 

increased precision when estimating unknown parameters. For example, to know the proportion 

of cattle that is infected with a pathogen, a more accurate estimate of this proportion will result 

from a sample of 200, rather than 100 cattle.

Time Series: A time series is a set of regular, time-ordered observations of a quantitative 

characteristic of an individual or collective phenomenon taken at successive, in most cases 

equidistant, periods/points of time. Breaks in statistical time series occur when there is a change 

in the standards for defining and observing a variable over time. Such changes may the result of 

a single change or the combination of multiple changes at any one point in time of observation of 

the variable.49 For example, changes to the way in which the E. coli O157:H7 sampling frame is 

constructed over time disrupts the time series and makes it difficult to compare results from year 

to year.

 

48 Please see the following website for more information: www.statistics.com.

49 Please see the following website for more information: http://stats.oecd.org/glossary/search.asp.
44

Appendix B

E. coli O157:H7 Raw Ground Beef (MT43)

Required Available

Population The universe of units for which the characteristic of interest is being 

assessed

Estimated annual production volume of raw ground beef in pounds at 

FSIS inspected establishments.

Sampling Frame A list of all FSIS inspected establishments that produce raw ground 

beef

All establishments that produce raw ground beef product (CFR 9) or for 

whom FSIS has analyzed an MT43 sample in the last 12 months are 

included in the sampling frame. There is an exclusion list.

Collection 

Frame

Entire sample frame - perhaps excluding establishments that have a 

history component (HS=hazard score)

Establishments are chosen based on volume weight and HS. Whether 

HS is still being used in the algorithm needs to be determined by 

translating the visual basic code.

Enumeration 

Unit Selection

Define the procedure used to select enumeration units 

(establishments) from collection frame. Random selection is 

preferred. Establishments can be stratified on a characteristic (e.g. 

risk, production volume) and then chosen randomly within each 

strata

MT 43 samples selected with replacement from sampling frame by an 

algorithm employing scaling factors, which act like weights; inputs to 

calculating the scaling factors are annual production volume group and

history of positive tests. Current project has 4 volume groups and 

algorithm includes sampling ceilings for each volume group. 

Establishments with a history of + test results will be sampled more 

frequently.

Population Unit 

Selection

Define the procedure used to select population unit(s) from a given 

enumeration unit. Clustered samples are acceptable, but effect of 

clustered choice must be addressed. Announced sampling is not 

preferred and may make clustered choice not acceptable.

MT43 sample collection is done by collecting 325 g of raw ground beef. 

Each establishment is sampled a maximum of 4 times per month. 

Product Type 

Selection

Define the procedure used to select the product type for sampling for 

a given product class. 100% judgment based selection is not 

acceptable.

We are only looking at MT 43 raw ground beef sampling.

Collection 

Date/Time

Day of week and time of day when sample is taken should be 

randomly chosen

Other Data Production volume for each establishment is required. Production volume is obtained in categories. 

Other Factors Determine the probability of selection for each unit of product for 

each establishment

The probability of selection may already exist in a column in each Excel 

sheet containing sampling frames
45

Salmonella Raw Intact Products

Required Available

Population The universe of units for which the characteristic of interest is being assessed.

All carcasses of a certain product class (broilers, turkeys, 

cow/bull, steer/heifer, and market hog) produced at FSIS 

regulated establishments.

Sampling Frame

A listing of all the units in the defined population. Alternatively, an aggregated 

listing of enumeration units provided that each population unit is associated with 

only one enumeration unit in the frame.

The set of all establishments that produce a given product class 

during the period of interest.

Collection 

Frame

This could be the entire sampling frame, or this could be the set of establishments 

(or other enumeration units) selected for sampling. Define the procedure used to 

set the frame.

Not all establishments in the sampling frame are included in 

the collection frame (low volume producers are excluded). 

Enumeration 

Unit Selection

Define the procedure used to select enumeration units (establishments) from the 

collection frame.

Random unit choice is preferred.

Establishments can be stratified based on a characteristic (e.g. risk, production 

volume) and then chosen randomly within each strata.

Establishments are chosen for sampling based on risk. New 

establishments and those in the highest risk category are 

selected first. 

Population Unit 

Selection

Define the procedure used to select population unit(s) from a given enumeration 

unit.

Clustered samples are acceptable, but effect of clustered choice must be 

addressed.

Announced sampling is not preferred. Announced samples may make clustered 

choice not acceptable.

HC01 samples are collected in sets. Once an establishment is 

chosen for sampling, it is sent 75 sample forms (steer/heifer 

establishments are sent 90). Field inspectors are instructed to 

collect one sample per day for each day that the 

establishment is producing the product being sampled. Only 

the first 50-56 successfully analyzed samples count towards 

the overall pass/fail result of the set depending on the product 

class being sampled. Thus, samples are both clustered and 

announced.

Product Type 

Selection

Define the procedure used to select the product type for sampling for a given 

product class.

100% judgment based selection is not acceptable.

N/A

Each class of raw intact product consists of only one type 

(not parts).

Collection 

Date/Time Day of week and time of day when sample is taken should be randomly chosen

Day of the week is not random or varied because once a 

sample set is started, an establishment knows it will be 

sampled every day it is producing that product for the next 

51-75 days. Field inspectors are instructed to collect samples 

at varied times of the day.

Other Data Production volume for each establishment is required.

The eADRS slaughter database contains detailed volume data 

for each product class at every establishment in the sampling 

frame. Volume totals are available for any time period 

(daily, weekly, monthly, yearly, etc.).

Other Factors

Determine the probability of selection for each unit of product for each 

establishment TBD
46

Required Available

Useful 

Characteristics 

(not required)

Interventions being used.

HACCP size of establishments (frame/population?).

Monthly Production Volume.

Daily Production for sample day.

Sample Weight.

Shift sample collected.

Date.

Time since last cleanup.

Interventions being used. Yes, likely available in plant 

profile. 

HACCP size of establishments. Yes, available in plant 

profile. 

Monthly Production Volume. Yes, available in eADRS 

database.

Daily Production for sample day. Yes, available in eADRS 

database.

Sample Weight. No

Shift sample collected. No

Date. Yes, Available on sample collection form 

Time since last cleanup. No
47

Lm - ALLRTE and RTE001

Required Available

Population The universe of units for which the characteristic of 

interest is being assessed.

The population of interest for RTE meat and poultry products is the volume of a certain product 

produced at non-exempt FSIS regulated establishments. 

Sampling 

Frame

A listing of all the units in the defined population. 

Alternatively, an aggregated listing of enumeration units 

provided that each population unit is associated with only 

one enumeration unit in the frame.

The sampling frame(s) is (are) composed of the list of non-exempt federally inspected 

establishments that produce RTE products.

Collection 

Frame

This could be the entire sampling frame, or this could be 

the set of establishments (or other enumeration units) 

selected for sampling. Define the procedure used to set the 

frame.

Presently, for the risk-based RTE001 sampling project, field inspectors are instructed to collect 

the riskiest RTE product samples produced in the establishment at the time of collection. The 

ALLRTE sampling project is already established as a random sampling project.

Enumeration 

Unit 

Selection

Define the procedure used to select enumeration units 

(establishments) from the collection frame. Random unit 

choice is preferred. Establishments can be stratified based 

on a characteristic (e.g. risk, production volume) and then 

chosen randomly within each stratum.

Establishments are currently chosen for Lm sampling based both on random selection 

(ALLRTE) and risk (RTE001). 

Population 

Unit 

Selection

Define the procedure used to select population unit(s) from 

a given enumeration unit.

Clustered samples are acceptable, but effect of clustered 

choice must be addressed.

Announced sampling is not preferred. Announced samples 

may make clustered choice not acceptable.

Current policy specifies that ALLRTE and RTE001 samples are to be collected individually and 

independently. 

Product Type 

Selection

Define the procedure used to select the product type for 

sampling for a given product class.

100% judgment based selection is not acceptable.

Under PHIS, there are four basic categories: Acidified/fermented, salt-cured, dried and fully 

cooked. The first 3 categories can each be divided into unsliced and deli sliced product types, 

for a total of 6 product types among these 3 categories. Within the category of fully cooked, 

there are eight product types. Thus, there are a total of 4 product categories and 14 product types 

within those categories that require sampling for Lm.

Collection 

Date/Time

Day of week and time of day when sample is taken should 

be randomly chosen For ALLRTE and RTE001, one sample is collected monthly from a given establishment.

Other Data Production volume for each establishment is required.

Presently the individual product distributions for such product groups as deli meat, hot dogs, and 

fermented sausage can be derived from the FSIS form 10,240-1 product volume data for 

establishments producing post-lethality exposed RTE products. RTE establishments without 

these data must use model estimates based on HACCP establishment size. However, when PHIS 

is implemented these data need to be readily available for all RTE establishments. 

Other 

Factors

Determine the probability of selection for each unit of 

product for each establishment
48

Appendix C: 

Current Salmonella Set Sizes by Product

Product Completed 

Set Size

Forms Sent 

to Plant

Broilers 51 75

Cows/Bulls 58 75

Ground Beef 53 75

Ground Chick 53 75

Ground 

Turkey 53 75

Market Hog 55 75

Steer/Heifers 82 90

Turkeys 56 75
49

References

Food Safety and Inspection Service (FSIS). 2008. Risk-based Sampling for Escherichia coli

O157:H7 in Ground Beef and Beef Trim. Prepared by J. Withee and W. Schlosser. Available 

from: http://www.fsis.usda.gov/PDF/Ecoli_Sampling_RA_Report_Feb08.pdf. 

FSIS. Annual Sampling Program Report. In development. 

Levy, P.S., Lemeshow, S. 2009. Sampling of Populations: Methods and Applications, 4th

edition. Wiley

Thompson, S.K. 2002. Sampling, 2nd edition. Wiley.