TO: Matthew Michael, Food Safety and Inspection Service 

FROM: Shawn Karns 

Mary Muth 

Michaela Coglaiti 

CC: Janet Elm 

DATE: September 22, 2005 

SUBJECT: Relative Risks of Meat and Poultry Products: An Expert Elicitation (Contract 

no. 43-3A94-2-0260) 

This memorandum describes the process RTI International (RTI) followed in conducting an 

expert elicitation for the U.S. Department of Agriculture’s Food Safety and Inspection 

Service (FSIS). The purpose of the expert elicitation was to collect data on the relative risks 

posed to public health by various types of processed meat and poultry products. Aggregated 

results of the expert elicitation are presented below. 

THE EXPERT PANEL 

RTI recruited 23 experts to participate on the expert panel. The experts were recruited from 

a list of potential experts, identified by FSIS and RTI, who have an understanding of food 

science, meat and poultry processing, and foodborne illness. The experts are employed in 

industry, academia, and the federal government. Table 1 lists the experts who participated 

and their areas of specialization.1 

EXPERT ELICITATION PROCESS 

The process of conducting the expert elicitation included recruiting experts, developing 

materials, conducting the elicitation, and summarizing the data. Specifically, we 

• contacted experts to determine availability and willingness to participate; 

• set up a panel participation (consulting) agreement with each expert who agreed to 

participate;2 and 

1An additional nine experts were contacted but either declined to participate or could not be reached 

despite repeated attempts by e-mail and telephone. 

2Some panel participants (i.e., federal government employees) were not able to accept an 

honorarium; thus, the panel participation agreement was not necessary. 
Matthew Michael 

September 22, 2005 

Page 2 

Table 1. Participants in the Expert Elicitation 

Panelist Organization Area of Specialization 

Gary Acuff Texas A & M Food microbiologist with experience in the 

microbiological quality and safety of foods 

A recognized authority on Hazard Analysis and Critical 

Dane Bernard Keystone Foods Control Point (HACCP) and food safety prevention 

systems 

John Cerveny Oscar Meyer (retired) Microbiology and food safety consultant 

Pat Curtis Auburn University Food scientist with experience in the quality and 

microbial safety of poultry and egg products 

Catherine Cutter Penn State University Food microbiologist with expertise in meat science and 

technology 

P. Michael Davidson University of Tennessee Extensive knowledge in applied food microbiology 

Ann Draughon University of Tennessee Food microbiologist with expertise in microbiological 

food safety 

Kelly Getty Kansas State University Meat scientist with experience in beef and pork 

Dana Hanson NC State University Meat extension specialist 

William Henning Penn State University Meat extension specialist with experience in beef and 

pork 

John Henson California State 

University, Fresno Meat scientist with industry experience 

Ann Hollingsworth Better Built Foods Meat scientist with industry experience 

Lee-Ann Jaykus NC State University Food scientist with expertise in molecular methods for 

the detection of foodborne pathogens 

Kevin Keener Purdue University Food processing engineer with experience in beef, 

pork, and poultry 

Chris Kerth Auburn University Meat scientist and biologist 

Lynn Knipe Ohio State University Meat extension specialist with experience in processed 

meats 

Mohammad 

Koohmaraie USDA ARS Animal physiologist with expertise in enhancing meat quality and safety 

Anne Marie 

McNamara Silliker. Inc. Food scientist with industry and regulatory experience 

Benjy Mikel Mississippi State 

University 

Meat extension specialist with experience in beef, 

pork, and poultry 

Donald Schaffner Rutgers, The State 

University of New Jersey 

Food scientist with experience in predictive 

microbiology of meat products 

Medical epidemiologist with expertise in the 

Robert Tauxe CDC FDDB surveillance and prevention of bacterial enteric 

infections 

Food microbiologist with expertise in red meat 

Bruce Tompkin ConAgra (retired) microbiology, processing, HACCP, sanitation, and 

hygiene 

Don Zink FDA CFSAN Food scientist with extensive experience in food 

safety, microbiology, and food processing 
Matthew Michael 

September 22, 2005 

Page 3 

• developed a timeline for conducting the expert elicitations, including scheduling 

conference calls and delivering documents by e-mail. 

Materials developed for use in conducting the expert elicitation included a project 

description and an elicitation worksheet (see Attachment A). The project description, 

developed by RTI, was provided to the panelists prior to agreeing to participate in the 

expert elicitation process. The document described the reason why we were conducting an 

expert elicitation and what the experts would be expected to contribute. FSIS developed the 

expert elicitation worksheet and provided it to RTI. The elicitation worksheet included 

• the assumptions for scoring, 

• instructions for the panelists, 

• the scoring sheet with the 24 products that the panelists were to score, and 

• a list of product examples for each product type. 

RTI conducted additional pretesting of the worksheet and, in consultation, with FSIS 

prepared the final worksheet. 

After RTI recruited the 23 experts to serve on the panel, we conducted the following 

activities: 

• scheduled and hosted teleconferences with the experts to discuss the purpose of the 

data collection, review the worksheet, and respond to questions; 

• requested that the experts complete the worksheets using approximately 1 day of 

consulting time within 7 days; 

• responded to questions raised by the panelists for which RTI needed clarification 

from FSIS (see Attachment B for a list of the questions and answers provided to the 

experts); and 

• obtained the completed worksheets and lists of cited references from the 

consultants. 

Once we obtained the completed worksheets, we aggregated the responses into the tables 

described later in this memorandum. 

SUMMARY OF RESULTS 

The data collected from the experts will be used to measure the relative risks posed by 

various types of processed meat and poultry products. To ensure consistency of product 

definitions, FSIS provided a list of example products for each of the 24 product categories. 

Using the list of example products for each product category, the panel was asked to score 

each of the product categories according to the relative risk of illness, per serving that each 

product category poses. The experts could assign a score of 1 to multiple products if they 

believed that multiple products posed the minimum level of risk of illness per serving. The 
Matthew Michael 

September 22, 2005 

Page 4 

experts were asked to assign their scores assuming typical production and consumption 

practices. 

Table 2 summarizes the results of the elicitation. The average rank was calculated by first 

calculating the sequential rank of scores for each expert and then averaging the sequential 

ranks for each product. Highlights of the results are as follows: 

• The maximum scores assigned by individual experts ranged from 3 to 300,000,000 

with a median value of 10.3 

• Raw products were generally assigned the highest scores, and ready-to-eat (RTE) 

products were generally assigned the lowest scores with the exception of RTE fullycooked poultry. 

• Poultry products generally were assigned higher scores than red meat products. 

• RTE meat fully cooked without subsequent exposure to the environment and 

RTE poultry fully cooked without subsequent exposure to the environment 

were scored by many experts as the products with least risk of illness. 

• Raw ground, comminuted, or otherwise nonintact chicken and raw ground, 

comminuted, or otherwise nonintact turkey were scored by many experts as the 

products with the highest risk of illness. 

The individual scores were provided to FSIS in a separate Excel worksheet. The worksheet 

also contains brief comments provided by the experts on their individual scores. 

3Because the maximum score of 300,000,000 was substantially higher than the maximum scores 

assigned by other experts, we verified over the phone that the expert who assigned this maximum 

score correctly understood the interpretation of the scoring. The expert indicated complete 

understanding and believes the scores accurately reflect the relative risks of these products. 
Matthew Michael 

September 22, 2005 

Page 5 

Table 2. Relative Risk of Illness per Serving Among 24 Types of Processed Meat and 

Poultry Products 

Average 

Sequential Median Minimum Maximum 

Finished Product Type Rank Score Score Score 

Raw intact beef 11 5.0 1.0 3,000 

Raw intact pork 9 4.0 1.0 1,000 

Raw intact meat— 

other than beef or pork 13 5.0 2.0 1,000 

Raw intact chicken 19 8.0 2.0 5,000 

Raw intact turkey 20 9.0 2.2 5,000 

Raw intact poultry— 

other than chicken or turkey 17 8.0 2.0 5,000 

Raw ground, comminuted, or otherwise 

nonintact beef 21 10.0 3.0 100,000 

Raw ground, comminuted, or otherwise 

nonintact pork 16 8.0 1.5 50,000 

Raw ground, comminuted, or otherwise 

nonintact meat—other than beef or pork 18 9.7 2.0 50,000 

Raw ground, comminuted, or otherwise 

nonintact chicken 23 10.0 2.0 200,000 

Raw ground, comminuted, or otherwise 

nonintact turkey 23 10.0 2.0 150,000 

Raw ground, comminuted, or otherwise 

nonintact poultry—other than chicken or 

turkey 22 10.0 2.5 150,000 

Raw otherwise processed meat 14 7.0 1.0 50,000 

Raw otherwise processed poultry 15 7.0 1.2 50,000 

RTE acidified/fermented meat 

(without cooking) 7 2.0 1.0 40,000,000 

RTE acidified/fermented poultry 

(without cooking) 8 2.0 1.0 40,000,000 

RTE dried meat 4 2.0 1.0 200,000,000 

RTE dried poultry 6 2.0 1.0 300,000,000 

RTE salt-cured meat 3 2.0 1.0 600,000 

RTE salt-cured poultry 5 2.0 1.0 2,500,000 

RTE fully-cooked meat 9 3.0 1.0 125 

RTE fully-cooked poultry 12 3.0 1.0 10,000 

RTE meat fully cooked without subsequent 

exposure to the environment 1 1.0 1.0 5.0 

RTE poultry fully cooked without subsequent 

exposure to the environment 2 1.0 1.0 30,000 
Matthew Michael 

September 22, 2005 

Page 6 

Attachment A 

PROJECT DESCRIPTION 

RELATIVE RISKS OF MEAT AND POULTRY PRODUCTS: 

AN EXPERT ELICITITATION 

RTI Project No. 08610.000 

Description 

The U.S. Department of Agriculture, Food Safety and Inspection Service (USDA, FSIS) has 

contracted with RTI International (RTI) to assist in conducting an expert elicitation. The 

purpose of the expert elicitation is to measure the relative risks posed to public health by various 

types of processed meat and poultry products. We are asking for your assistance as a participant 

in the expert elicitation process to score each of 24 product categories according to the relative 

risk of illness, per serving, that you believe the product category poses. While scoring the 

categories, we will ask that you consider the biological, chemical, and physical hazards inherent 

to both the source material and the processes used to produce the products in the category. 

What We Would Need From You 

If you agree to participate in the expert elicitation process, you will need to do the following: 

• complete the accompanying Interest Form; 

• review the one-page worksheet you will be completing for the expert elicitation; 

• participate in a 20-minute teleconference to discuss the worksheet and ask questions 

about the process; 

• using resources at your disposal, complete the worksheet providing your best 

estimates of the needed information within one week of the initial teleconference; and 

• deliver your responses to the worksheet and a list of citations to RTI by Federal 

Express or e-mail. 

To compensate you for your time, we will pay you an honorarium of $250 for completing the 

worksheet. If you are not able to accept the honorarium, we would still like to encourage your 

participation. 
Matthew Michael 

September 22, 2005 

Page 7 

For additional information on this project, you can contact: 

Shawn Karns 

Food and Agricultural Policy Research Program

RTI International 

Research Triangle Park, NC 27709-2194 

E-mail: karns@rti.org 

Phone: 919-541-6380 

RTI is an independent, nonprofit organization that serves clients in government, industry, 

academia, and public service throughout the United States and abroad. Our headquarters are 

located on a 180-acre campus in Research Triangle Park, NC, and we employ a worldwide staff 

of more than 2,000 people. The Food and Agriculture Policy Program at RTI has been 

conducting analyses of the economic effects of food safety and nutrition regulations for USDA 

and FDA for more than 15 years. 
Matthew Michael 

September 22, 2005 

Page 8 

WORKSHEET FOR RELATIVE RISKS OF MEAT AND POULTRY PRODUCTS: 

AN EXPERT ELICITATION 

The purpose of this expert elicitation is to measure the relative risks posed to public 

health by various types of processed meat and poultry products. Please score each of the 

24 cells according to the relative risk of illness, per serving, that you believe the 

corresponding product category poses. Consider the biological, chemical, and physical 

hazards inherent to both the source material and the processes used to produce the 

products in the category. Take into account all that you know about meat and poultry 

science, food processing, food transport, consumer handling, and foodborne illness, but 

assume the following when scoring: 

• Each of the 24 cells represents finished products that will reach the consumer 

without further processing after it leaves the producing plant, e.g. raw ground 

chicken or canned meat product. Examples of each of the finished product types are 

provided in the attached table titled “Finished Product Type Examples.” 

• Do not account for products that are prepared (sliced, ground, cooked, etc.) at the 

retail or institutional level. Consider preparation only by the producing plant and the 

consumer. 

• Each product is produced in a USDA-regulated processing plant that operates under 

sanitation standard operating procedures (SSOP) and a Hazard Analysis Critical 

Control Point (HACCP) system. 

• The incoming source material (raw meat or poultry and other ingredients) comes 

from a slaughter plant, trim producer, grinder, or other firm with average or typical 

food safety controls. 

• The processing plant’s food safety controls are average or typical; do not think of 

extreme or unusual processing situations. 

• The products receive typical handling by all parties from the time the products leave 

the processing plant through the time they are consumed (so you may account for 

safe handling or mishandling if you believe either to be typical). 

• The consumers are healthy adults. 

• None of the products are irradiated. 

• In regard to the ready-to-eat (RTE) products: 

o Unless specifically stated, all are exposed to the environment during handling 

after lethality treatment(s); 

o None contain an additive to inhibit growth of L. monocytogenes; 

o None receive any post-lethality treatment to destroy L. monocytogenes. 
Matthew Michael 

September 22, 2005 

Page 9 

SCORING INSTRUCTIONS 

1. First, place a “1” in the cell corresponding to product that poses the least risk of 

illness per serving. 

2. Second, choose the cell corresponding to product that poses the greatest risk of 

illness per serving and give it a value proportionate to the increase in risk over that 

posed by the product posing the least risk. For example, if you believe that the 

riskiest product poses four times more risk than the product posing the least risk, 

place a “4” in its cell; if you believe it poses twenty times more risk, place a “20” in 

its cell. You may use fractions. 

3. You now have established your range of values. Now assign values to the remaining 

products. Remember that each value is proportionately relative to any other. A 

product receiving a score of “10” for example, would be considered to be ten-times 

as risky as the product posing the least risk. A product receiving a score of “20” 

would be considered twice as risky as the product that received a “10.” You may 

score two or more products with equal values (including your highest or lowest 

scores) if you believe the risk they pose per serving is equal. 

Please complete your scores independently without discussing them with the other 

panelists. If you have questions, please contact Shawn Karns at 919-541-6380 or 

karns@rti.org. 
Matthew Michael 

September 22, 2005 

Page 10 

Relative Risk of Illness per Serving among 24 Types 

of Processed Meat and Poultry Products 

Finished Product Type Score Brief Explanation of Your Score (If Needed) 

Raw intact beef 

Raw intact pork 

Raw intact meat – other than beef or pork 

Raw intact chicken 

Raw intact turkey 

Raw intact poultry – other than chicken or turkey 

Raw ground, comminuted, or otherwise nonintact beef 

Raw ground, comminuted, or otherwise nonintact pork 

Raw ground, comminuted, or otherwise nonintact meat – other than beef or pork 

Raw ground, comminuted, or otherwise nonintact chicken 

Raw ground, comminuted, or otherwise nonintact turkey 

Raw ground, comminuted, or otherwise nonintact poultry – other than chicken or turkey 

Raw otherwise processed meat 

Raw otherwise processed poultry 

RTE acidified/fermented meat (without cooking) 

RTE acidified/fermented poultry (without cooking) 

RTE dried meat 

RTE dried poultry 

RTE salt - cured meat 

RTE salt - cured poultry 

RTE fully - cooked meat 

RTE fully - cooked poultry 

RTE meat fully-cooked without subsequent 

exposure to the environment 

RTE poultry fully-cooked without subsequent 

exposure to the environment 
Matthew Michael 

September 22, 2005 

Page 11 

Please return the completed worksheet by email, fax, or FedEx: 

Email: coglaiti@rti.org 

Fax: 919-541-6683, Attn: Michaela Cimini Coglaiti 

FedEx: Michaela Cimini Coglaiti 

RTI International 

REPR, Hobbs 123 

3040 Cornwallis Rd. 

Research Triangle Park, NC 27709-2194 

(919) 990-8498 
Matthew Michael 

September 22, 2005 

Page 12 

Finished Product Type Examples 

Finished Product Type Product Examples 

Raw intact beef Steaks, roasts 

Raw intact pork Chops, roasts, ribs, loins 

Raw intact meat – other (sheep, goat) Chops, roasts 

Raw intact chicken Whole bird not stuffed and stuffed, parts (including 

necks/feet and giblets), boneless/skinless parts 

Raw intact turkey Whole bird not stuffed and stuffed, parts (including 

necks/feet and giblets), boneless/skinless parts 

Raw intact poultry – other (ducks, geese, Whole bird not stuffed and stuffed, carcass parts 

squab) 

Raw ground, comminuted, or otherwise non- Ground, restructured, tenderized/injected/marinated, 

intact beef AMR 

Raw ground, comminuted, or otherwise non- Ground, restructured, tenderized/injected/marinated, 

intact pork AMR, MS pork 

Raw ground, comminuted, or otherwise non- Ground, restructured, tenderized/injected/marinated 

intact meat – other (sheep, goat) 

Raw ground, comminuted, or otherwise non- Ground, restructured, tenderized/injected/marinated, 

intact chicken MS chicken 

Raw ground, comminuted, or otherwise non- Ground, restructured, tenderized/injected/marinated, 

intact turkey MS turkey 

Raw ground, comminuted, or otherwise non- Ground, restructured, tenderized/injected/marinated, 

intact poultry – other (ducks, geese, squab) MS Poultry 

Raw otherwise processed meat Batter set nuggets and tenders, char marked patties 

Raw otherwise processed poultry Batter set nuggets and breaded parts, partially cooked 

rolls and loaves 

RTE acidified/fermented meat (without Genoa salami, hard salami, pepperoni 

cooking) 

RTE acidified/fermented poultry (without Turkey pepperoni 

cooking) 

RTE dried meat Dried beef, jerky, landjager, meat sticks, some chorizo 

RTE dried poultry Jerky (basically turkey) 

RTE salt-cured meat Country ham, prosciutto, coppa, capocolla, basturma, 

bresaola 

RTE salt-cured poultry Ducks, geese 

RTE fully-cooked meat Hot dogs, deli meats, roasts 

RTE fully- cooked poultry Whole birds, parts, hot dogs, deli items, roasts 

RTE meat fully-cooked without subsequent Cooked in package (canned ham (not shelf stable), 

exposure to the environment cook-in-bag), hot packed (chili, sauces, soups) 

RTE poultry fully-cooked without subsequent Cooked in package (cook-in-bag), hot packed (soups) 

exposure to the environment 

AMR = Advanced Meat Recovery. 

MS = Mechanically seperated. 
Matthew Michael 

September 22, 2005 

Page 13 

Attachment B 

Questions Raised by Panelists with FSIS’s Responses 

Question: When considering the risk of illness, should we take into account the severity of 

illness or just the incidence of illness? 

Response: Do NOT consider severity of illness. Just consider the incidence or risk of 

illness. Severity of illness will be addressed in any risk analysis that uses this data. 

Question: Would the agency clarify the RTE statements about additives and post-lethality 

treatments for L. monocytogenes? 

Response: Assume all RTE products are manufactured using Alternative 3 of the final rule 

(Use only sanitation measures to prevent LM). The results of the elicitation will be used as 

a “ceiling” for the RTE products. 

Question: Does RTE fully-cooked meat and RTE fully-cooked poultry include cured and 

uncured product? 

Response: Yes. 

Question: Does “RTE meat (poultry) fully-cooked without subsequent exposure to the 

environment” include shelf-stable products? 

Response: The categories include the example product types which might be considered 

“shelf-stable”. They do NOT include thermally processed, commercially sterile products. 

Question: Should allergies be considered when scoring the relative risk of products? 

Response: No. Allergies will be addressed in any risk analysis that uses the expert 

elicitation data. 

Question: Is a healthy pregnant woman a “healthy adult”? 

Response: The woman is a healthy adult but the fetus is part of the at-risk population. Do 

NOT consider pregnant women as part of the healthy adult population. Please consider 

typical healthy adults as the population. FSIS plans to use the elicitation data as a baseline. 

The data will be scaled or weighted for at-risk populations or other extreme circumstances.