# Presentation: 582511

## MEASURING THE EFFECTIVENESS OF THE NATION’S FOODSERVICE AND RETAIL FOOD PROTECTION SYSTEM

## PRESENTATION OBJECTIVES

**Purpose of FDA’s Study**
**Review What the Study ****IS**** ****and ****IS NOT**
**Review the Design of the Study**
**Preview of 2003 Results**
**Present Recommendations**

**Notes:**
This presentation will provide information on the purpose and scope of the 2004 FDA Report on the Occurrence of Foodborne Illness Risk Factors in Selected Institutional Foodservice, Restaurant, and Retail Food Stoe Facility Types

The results contained the FDA 2004 Report are part of a larger long-term effort to observe and document food safety practices and employee behaviors that have the potential to contribute to foodborne illness.  In order to interpret and use the results presented, one will  need to understand what the study is designed to cover and what it is not designed to do.

The presentation will cover:
 The methodology for collecting the data;
 How the data was analyzed;
 How IN Compliance and Out of Compliance percentages are determined and what they refer to; and
- Recommendations for both regulatory and industry food safety professionals for enhancing the effectiveness of the nation’s foodservice and retail food protection system

## BACKGROUND

**FDA National Retail Food Steering Committee**
**Government Performance and Results Act (1993)****“*****performance plans ... measurable indicators*****”**

**Notes:**
The 1996 report, Reinventing Food Regulations [National Performance Review] concluded that foodborne illness caused by harmful bacteria and other pathogenic microorganisms in meat, poultry, seafood, dairy  products, and a host of other foods is a significant public health problem in the United States  In response to the 1996 National Performance Review Report, FDA established the National Retail Food Steering Committee including representation from the Centers for Food Safety and Applied Nutrition (CFSAN), Office of Regulatory Affairs (ORA), Divisions of Federal/State Relations (DFSR), Division of Human Resource Development (DHRD) and the Interstate Travel Program (ITP) Field Team.  The Steering Committee is responsible for reviewing  retail food program objectives and coordinating program activities.

The 1993 Government Performance and Results Act required Federal agencies to develop performance plans that included measurable goals and performance indicators.  The level of foodborne illnesses would be the ideal retail food program performance indicator.  The occurrence of foodborne illness, however, is grossly underreported which makes it an unreliable program measurement.  As an alternative, the occurrence of foodborne illness risk factors was selected as the performance indicator.

## INTRODUCTION AND PURPOSE

**Agency effort to change behaviors / practices related to foodborne illness**
**Identify food safety practices and employee behavior most in need of attention**

**Notes:**
This project was designed to collect data on the occurrence of foodborne illness risk factors in selected foodservice and retail food establishments at five-year intervals.  These data collection efforts are designed to get an accurate picture of the extent to which foodservice and retail food operations have control over the risk factors during each data collection period.  

Using multiple data collection periods, FDA hopes to evaluate trends and determine whether progress is being made toward the goals of reducing the occurrence of foodborne illness risk factors.

## INTRODUCTION AND PURPOSE

**Establish a national baseline on the occurrence of foodborne illness risk factors**
**Effort to collect sufficient data to measure patterns in the occurrence of foodborne illness risk factors over time**

**Notes:**
By establishing the 1998 Baseline on the occurrence of foodborne illness risk factors, FDA has a performance measure against which industry and regulatory efforts to change behaviors and practices directly related to foodborne illness will be measured.  This study is intended to fill a void that currently exists in the assessment of program effectiveness for controlling these foodborne illness risk factors.

In order to detect trends of improvement and/or regression in this study, it is necessary to collect data from at least three perferably more, periods in time.

## STUDY TIME LINE

_**1998 1**__**ST**__** Data Collection**_***Report of the FDA Retail Food Program Database of Foodborne Illness Risk Factors (2000)***
**Established a Baseline for 9 different foodservice and retail food facility types**

**Notes:**
Using the results of the 1998 data collection, the 2000 Report of the FDA Retail Food Database of Foodborne Illness Risk Factors established the first-ever national baseline on the occurrence of foodborne illness risk factors within institutional foodservice, restaurant, and retail food store facility types.

Using the results from the 1998 data collection as a baseline, FDA’s Natioanl Retail Food Steering Committee has established a goal of reducing the Out of Compliance percentage of observations of the original 42 data items related to foodborne illness risk factors in institutional foodservice, restaurants, and retail food store facility types by 25% by 2010

## STUDY TIME LINE

_**2003 2**__**nd**__** Data Collection**_***FDA Report on the Occurrence of Foodborne Illness Risk Factors in Selected Institutional Foodservice, Restaurant and Retail Food Store Facility Types (2004)***
**Begins the process for establishing multiple data collection periods**

**Notes:**
The 2004 FDA Report of the Occurrence of Foodborne Illness Risk Factors in Selected Institutional Foodservice, Restaurants, and Retail Food Store Facility Types summarizes the results from the second data collection period conducted in 2003.  The results contained in this report provide insight into the effectiveness of industry management systems and food safety regulatory programs in controlling foodborne illness risk factors in certain retail food and foodservice operations.  

This report should be read and the data interpreted as a stand-alone report.  Additional data are needed before any meaningful assessments of trends can be made for each of the facility types.  This report makes no attempt to draw comparisons between the results of the 1998 and 2003 data collections.

## STUDY TIME LINE

_**2008 3**__**rd**__** Data Collection will be conducted**_**Report prepared for 2009 will begin the process of analyzing the three data sets (1998, 2003, and 2008) to assess improvement or regression changes from the 1998 Baseline**

**Notes:**
A third data collection is planned for 2008.  Having 3 data collection periods is a necessary condition for the determination of a trend.  While it is theoretically possible to find  a trend with three data collections periods, the impact of certain limitations related to the observations made of food safety practices and behaviors from one study to the next may require additional data collection points. 

At the beginning of the study design it was anticipated that the number of observations for each data item would vary.  The total number of observations for each data item is likely to change from one data collection period to another.  When analyzing data from two data collections periods, the variation in the number of observations makes it difficult to draw statistical conclusions between overall compliance percentages or risk factor compliance percentages.

Changes in the number of observations of data items may be attributed to such things as sample variation –sampling establishments within the dame facility type that have different food products and/or procedures OR changes in industry practices.  For example, establishments shifting from the use of raw shell eggs to using in shell pasteurized eggs will reduce the number of observations related to inadequate cooking from one data collection period to the next.

## WHAT THE REPORT IS AND IS NOT

**IS**** – a national assessment of the relative strengths & weaknesses of food safety systems designed to control the occurrence of foodborne illness risk factors**
**IS NOT**** – a correlation of the occurrence of foodborne illness risk factors with the actual incidence of human illness**

**Notes:**
The FDA 2004 Report contains insights into the effectiveness of industry management systems and food safety regulatory programs in controlling fooborne illness risk factors in foodservice and retail food store facility types.  The results contained in the report are intended to focus attention on foodborne illness risk factors associated with food preparation procedures and employee behaviors in most need of improvement by the industry.

Pathogens may enter the food supply at any  point in the farm-to-table continuum.  All industry sectors within this continuum have a responsibility for ensuring safe food.  The 2004 Report covers only those facility types that comprise institutional foodservice, restaurant, and retail food store operations.  

## WHAT THE REPORT IS AND IS NOT

**IS****– an assessment that used the 1997 Food Code provisions as the standard upon which observations of food safety practices are based**
**IS NOT****– a determination of the industry’s OR a specific establishment’s regulatory compliance with prevailing laws & regulations**

**Notes:**
One of the purposes of the study is to track changes related to industry’s effort to control foodorne illness risk factors over time.  For this type of study, the standard of measurement used to evaluate these comparisons – in this case the 1997 Food Code  - must remain constant for each of the data collection periods.

Suppose the recommended standard in the Food Code was relaxed between data collection periods in the study and the data item was changed accordingly.  Let’s say that the next data collection shows an improvement in the IN Compliance percentage for that data item.  One would not know if the improvement was due to an improvement in employee food safety practices or whether it was due to the change in the Food Code standard.

No attempt was made to determine if an establishment would have been found to be substantially IN Compliance with prevailing state, local or  tribal regulations.  For some data items, the standard of measurement was different.  Foodservice industry practices observed may have been IN Compliance with less stringent state or local laws even though the report notes they were not IN Compliance with the 1997 Food code

The study is designed to track changes in industry’s active managerial control of food safety practices and employee behaviors not simply compliance with regulatory standards

## METHODOLOGY

- Scope: _3 Industry segments / 9 facility types_**:**
- _Institutions_** **
**Hospitals**
**Nursing Homes**
**Elementary Schools**

**Notes:**
This research  project is actually nine separate studies.  One for each of the nine facility types.  Due to the different food products and processes that will be found in each of these facility types, it is inappropriate to make comparisons between them.  

The nine facility types were chosen from three industry segments:  Institutional Foodservice, Restaurants and Retail Food Stores.  The segments were chosen because they represented both general and highly susceptible populations and they incorporated the overwhelming majority of establishments we find at the foodservice and retail level – over a million varied and diverse establishments.

Of the nine facility types, three were associated with institutional foodservice:

Hospitals
Nursing Homes and
Elementary Schools (K through 5) 

## METHODOLOGY

- Scope: _3 Industry segments / 9 facility types_**:**
- _Restaurants_
**Fast Food**
**Full-Service**

**Notes:**
The restaurant segment was comprised of two facility types:

Fast Food Restaurants and
Full Service Restaurants

## METHODOLOGY

- Scope: _3 Industry segments / 9  facility types_:
-   _Retail Food Stores_  -- _4 departments_:
**Deli**
**Meat and Poultry**
**Seafood**
**Produce**

**Notes:**
Four facility types were departments of retail food stores and independent specialty operations related to:

Delis
Meat and Poultry
Seafood and
Produce

## METHODOLOGY

**2003 Data Collection:**
** ****926 Inspections **
** ****15,516 Observations**
** ****Standardized Specialists **** conducted the inspections**

**Notes:**
FDA Regional Retail Food Specialists collected the data.  In order to reach the sample size goal for the desired statistical assessment, 90 inspections of each of the facility types had to be performed.  This was accomplished by completing 926 inspections, resulting in 15,516 observations of specific food safety practices and behavior.

## METHODOLOGY

**Selection of Project Locations**
**Selection of Establishments (Comparison Lists)**

**Notes:**
The geographic distribution of Specialists throughout the country allowed for a broad sampling of establishments through all regions of the U.S.  The choice of data collection locations, therefore, was based on the Specialists’ geographical areas of responsibility and provided a reasonably convenient design for estimating national risk-related behaviors and practices.

An improved design for measuring trends within the retail food industry would be one based on probability sampling of the whole nation in which the number of establishments inspected within any given location would be related to the volume of food consumed within that location.  This would have required the development of comprehensive establishment lists for randomly selected locations around the country and excessive travel, which would have been cost-prohibitive. 
Comparison set establishment lists was the approach used by Specialists to select establishments.  Using establishment inventories, most often provided by State or county health agencies, each Specialist developed 5 Comparison Set Establishment Lists for each of the 9 facility types.  A comparison set list usually included 10-20 establishments.  After preparing the 5 Comparison Set Establishment List, a table of random numbers was used to select one establishment from each list for inspection.  The random selection of establishments from comparison set lists reduces the potential for selection bias.

## METHODOLOGY

**Confidentiality of Selected Establishments**
**Observational vs. Regulatory inspections**

**Notes:**
The demographic and inspectional data drawn for use in the study are confidential.  Ensuring confidentiality related to the establishments selected for the study enhances the reliability of the data collected.

Data Collection visits are intended to be observational rather than regulatory – observations of actual food safety practices and employee behaviors.

## BACKGROUNDCONTRIBUTING FACTORS 

**Food from Unsafe Source**
**Inadequate Cooking**
**Improper Holding/Time-Temp.**
**Contaminated Equipment/Protection from Contamination**
**Poor Personal Hygiene**
**Other “Chemical Hazards”**

**Notes:**
Some of these reported contributing factors related to food safety concerns within foodservice and retail food store facility types and include:

Food from Unsafe Sources
Inadequate Cooking
Improper Holding/Time and Temperature
Contaminated Equipment/Protection from Contamination
Poor Personal Hygiene

FDA added another category to assess control of potential chemical hazards.  For the purposes of this study FDA designated these six categories as foodborne illness risk factors.  

It is important that one not confuse the term risk factor as it is used in this FDA study with another use of the same term which is to describe characteristics of the person, food, or location that are statistically linked to an illness by an epidemiological investigation.  The results of the FDA 2004 Report should not be used to infer a correlation between the occurrence of a foodborne illness risk factor with the actual occurrence of human illness.

## Slide 18

**Notes:**
The data collection tool used to record the information for the study includes 42 individual data items sorted among the 5 foodborne illness risk factor categories and the “Other” chemical category.  This slides depicts a section of the data collection tool pertaining to reheating potentially hazardous foods for hot holding.  This section is comprised of 4 individual data items pertaining to different reheating  practices or scenarios.  Dividing each of the foodborne illness risk factors into individual data items allows the study to assess what specific food safety practices and employee behaviors are in most need of attention.

## METHODOLOGY

**Data Collection –**** ****Records status for all individual data items **
**	****IN ****    – In Compliance**
**	****OUT**** – Out of Compliance**
**	****N.O.**** – Not Observed**
**	****N.A.**** – Not Applicable**

**Notes:**
For each individual data item, the data collector determined whether it was:

IN – In Compliance – the observation was IN Compliance with the applicable FDA Food Code provision;
OUT –Out of Compliance – the observation was Out of Compliance with the applicable FDA Food Code provision;
N.O. – Not Observed.  The N.O. marking is used when an item is a usual practice in the foodservice or retail operation, but the practices was not observed during the time of inspection; or
N.A. – Not Applicable.  The N.A. marking was used when an item was not part of the foodservice or retail operation 

The same 42 data items will be assessed for each of the data collection periods in the study (1998, 2003, and 2008).  The critical limits and standards contained in the 1997 FDA Food Code will be used to assess each of these 42 data items for all 3 data collection periods.

## DATA ANALYSIS

**Three levels of data analysis are conducted for each of the 9 facility types**
**	**
**	****A.  **_**Out of Compliance**_** percentage**** ****for each**** ****      **_**individual data item**__** **_
**	****B.****  **_**Out of Compliance**_** percentage**** ****for each**** ****      **_**foodborne illness risk factor**_
**	****C.****  **_**Overall IN Compliance**_** percentage**** ****for ****      **_**all 42 data items**_
**	**

**Notes:**
Three levels of data analysis are conducted for each of the 9 facility types:

An Out of Compliance percentage is determined for each of the individual data items;
-An Out of Compliance percentage is determined for each of the foodborne illness risk factors;
-An Overall IN Compliance percentage for all the 42 data items is determined.

Each of one of these analyses will be addressed separately in the following slides

## Determining the Out of Compliance percentage for each DATA ITEM

**Notes:**
The Out of Compliance percentage for an individual data item is calculated by dividing the total number of observations that are Out of Compliance by the total number of observations (both IN and Out).  Not Observed and Not Applicable markings are not used to determine Out of Compliance percentages.  This is because the study is designed to be an assessment of actual observations.

## DeterminingOut of Compliance percentages for each DATA ITEM

**The **_**proportion of establishments**_
**where the data item was found Out**
**of Compliance at least once when **
**the practice or procedure could be **
**observed**

**Notes:**
The Out of Compliance percentage for a data item represents the proportion of establishments where that data item was found Out of Compliance at least once when the practice or procedure could be observed.  For example, the report shows a 40% Out of Compliance percentage for proper adequate handwashing in nursing homes.  This percentage should NOT be interpreted to mean that food workers were not washing their hands 40% of the time.

What the 40% Out of Compliance percentage means is that in 4 out of 10 nursing home establishments where handwashing was observed, there was at least one observation of improper or inadequate handwashing.

## Determining Out of Compliance Percentages FOODBORNE ILLNESS RISK FACTORS

**Observations of DATA ITEMS**
**FOODBORNE ILLNESS RISK FACTOR**

**Notes:**
The second level of analysis is to determine the Out of Compliance percentage  of the foodborne illness risk factors for each of the 9 facility types.  Each of the risk factors is comprised of a number of individual data items.  The collective observations of all the individual data items that comprise a risk factor are used to determine the compliance percentage.

## Slide 24

**Notes:**
This slide summarizeS the number of data items that comprise each foodborne illness risk factor category.  The number of data items that comprise a risk factor is presented in the middle column,  

For example, 7 individual data items are assessed under the Food from Unsafe Source foodborne illnees risk factor.  For the Inadequate Cooking risk factor, 12 individual data items are assessed and so on down the table. 

## Determining the Out of Compliance percentage for each FOODBORNE ILLNESS RISK FACTOR

**Notes:**
The Out of Compliance percentage for each foodborne illness risk factor is determined by dividing the total number of Out of Compliance observations of all the data items that comprise a risk factor by the total number of observations (both IN and Out) made of those data items.

## ExampleOut of Compliance Percentage   Poor Personal Hygiene - Risk Factor

_**Poor Personal Hygiene**_
**  ****5 ****DATA ITEMS**** – Data collection Form**
**12A****       ****13A****       ****14A****       ****15A****       ****15B**
**Poor Personal Hygiene - RISK FACTOR**

**Notes:**
For example, the Poor Personal Hygiene risk factor is comprised of 5 data items.  The numbers in the boxes 12A through 15B correspond to the data item number on the data collection form.  The Out of Compliance percentage for the Personal Hygiene risk factor is determined by dividing the total Out of Compliance observations of these 5 data items by the total number of observations (both IN and Out) of the 5 data items

## Determining Overall IN Compliance Percentages - FACILITY TYPES -

**6**** RISK FACTOR CATEGORIES**
**  **
**Overall In Compliance % **
**Facility Types**

**Notes:**
The thIRd level of data analysis is to determine the Overall IN Compliance percentage for each of the 9 facility types.  This is done by dividing the total number of IN Compliance observations of all 42 data items by the total number of observations (both IN and Out) made of those data items.

## Slide 28

**Notes:**
This table may be familiar to some of you.  It summarizes the results from the first data collection period conducted in 1998.  The 1998 Overall IN Compliance percentage represents the study’s Baseline measurement for each of the 9 facility types.  The column at the far right shows FDA’s 2010 improvement goal which is a 25% reduction in the occurrence of Out of Compliance observations of food safety practices and employee behaviors for the combine 42 individual data items.

For example, the Hospital facility type has an Overall IN Compliance (Baseline measurement) of 80%.  The improvement goal is to reduce the 20% Out of Compliance percentage for Hospitals by 25%.

25% of 20% = 5%.  Hence the FDA Improvement goal for Hospitals is to achieve an 85% IN Compliance percentage by 2010.

The Overall IN Compliance percentage for 2003 is not included in the table because there is not enough data at this time to support any comparisons with the 1998 Baseline percentages.  

## FDA Report on the Occurrence of 

***Foodborne Illness Risk Factors in Selected Institutional Foodservice, Restaurant and Retail Food Store Facility Types******(2004)***

**Notes:**
So what are some of the results that can be derived from the 2004 FDA Report on the Occurrence of Foodborne Illness Risk Factors in Selected Institutional Foodservice, Restaurant and Retail Food Store facility Types.

## 2004 Report

**Report should be read and the data interpreted as a separate stand-alone assessment**** **
**The report provides insights into food safety practice and employee behaviors needing priority attention**

**Notes:**
The FDA 2004 Report is intended to be read as a stand alone report with a focus on reducing the occurrence of those foodborne illness risk factors and data items identified as needing priority attention 

## Foodborne Illness Risk Factorin Need of Priority Attention (2000 Report)

**Improper Holding/Time and Temperature**
**Poor Personal Hygiene**
**Contaminated Equipment / Protection from Contamination**

**Notes:**
One of the most important findings from the original 2000 Report was that for all 9 facility types 3 foodborne illness risk factors were identified as priority areas:

-Improper Holding/Time and Temperature
-Poor Personal Hygiene; and
-Contaminated Equipment/Protection from Contamination

The data results in the 2004 Report once again identifies these 3 foodborne illness risk factors and the individual data items that comprise them as the priority areas for all the facility types.

This is not to be interpreted as a comparison of the results in the 2004 Report with those presented in the 2000 Report.  This is meant to highlight that as 2 separate stand alone studies, they identify the same foodborne illness risk factors and data items as needing priority attention.  

## 2004 ReportData Results and Discussion

**For each of the **_**9 facility types**_** the Data Results will present:**** **
**	****A.****	****Overall Out of Compliance ****	****percentage for each of the ****	****foodborne illness risk factors**
**	****B.  Individual data Items that ****	****comprise each of the risk factors ****	****that are in need of priority ****	****attention**
**	****	**

**Notes:**
Section 3 of the 2004 FDA Report summarizes the results of the 2003 data collection period.  The data results are presented separately for each of the 9 facility types.

For each of the facility types, the report presents:

-A bar graph displaying the Out of Compliance percentage for each of the foodborne illness risk factors;
-A bar graph listing the individual data items comprising each of the foodborne illness risk factors that are in most need of attention; and

## 2004 ReportData Results and Discussion

**For each of the **_**9 facility types**_** the Data Results will present:**** **
**	**
**	****C.  A national strengths/weaknesses **** ****	****assessment of management systems ****	****for controlling the occurrence of ****	****foodborne illness risk factors ****	**

**Notes:**
-A summary table of the risk factors and individual data items in need of priority attention.

As you view the upcoming slides, some of you will begin to focus on the out of compliance percentages for the risk factors and the data items. This fixation on compliance percentage numbers can draw attention away from one of the most important uses of the data results.  The 2004 Report  should be used as a strengths/weaknesses assessment of you own industry food safety management system or regulatory retail food protection program.

For industry, when the study identifies a food safety practice or employee behavior as having a high Out of Compliance percentage, the question to ask is what procedures or processes are currently in place in my operation to ensure active managerial control of that specific item?  An assessment should be made of existing food safety management systems to ensure there are not any gaps in control measures for the specific item identified.

For regulatory food safety professionals, an examination should be made of current food establishment inspection protocols to ensure that food safety practices and employee behaviors with high Out of Compliance percentages are inspection priorities.  Is the appropriate time being taken to assess industry’s active managerial control of these areas?

## Slide 34

**Notes:**
Using the facility type Full Service Restaurants as an example, this bar graph indicates that failure to control product holding  temperatures and times was the risk factor with the highest Out of Compliance percentage.  Poor personal hygiene, contaminated equipment/protection from contamination and chemical contamination also had notable Out of Compliance percentages.  Management systems that were implemented to ensure foods were adequately cooked and from safe sources appeared to be effective when compared to the other risk factors evaluated during this data collection period.

The Report is designed to look at each of these foodborne illness risk factor categories as areas that should have a food safety management system in place to ensure their control.

## Slide 35

**Notes:**
As discussed earlier, each of these foodborne illness risk factor categories is comprised of individual data items.  These data items provide insights into what aspects of active managerial control for s risk factor should be assessed and verified to ensure appropriate measures are in place to control the food safety hazards of concern.

For full service restaurants, improper holding/time and temperature control was identified as the foodborne illness risk factor with the highest Out of Compliance percentage.  If we break this risk factor down into specific food safety practices, there is a need to review food safety systems related to:

Maintaining cold holding temperatures for potentially hazardous food;
-Cooling potentially hazardous food;
-Date marking of ready-to-eat, potentially hazardous foods made on site and open containers of commercially processed, ready-to-eat potentially hazardous food
-Maintaining hot holding temperatures for potentially hazardous food

## Slide 36

**Notes:**
With respect to the Poor Personal Hygiene risk factor in full service restaurants, the food safety procedures  that are most in need of attention include:

-Proper adequate handwashing; and
-Prevention of hand contamination – or in other words – eliminating bare hand contact with ready-to-eat foods.

Assessing procedures related to employee hygienic practices, such as eating, drinking and smoking in food preparation areas is also worthy of attention.

## Slide 37

**Notes:**
For the Contaminated Equipment/Protection from Contamination risk factor, food safety procedures most in need of attention in full service restaurants include:

-Cleaning and sanitizing food contact surfaces and utensils;
-Preventing contamination by separating raw animal foods from ready-to-eat foods;
-Separating different species of raw animal foods; and
-Protecting food products from environmental contamination – such as leaking waste water from a condensate drain.

## Slide 38

**Notes:**
At the end of each of the facility types sections, there is a summary table of the foodborne illness risk factors and the individual data items in need of priority attention.  The summary table displayed on this slide is for Deli operations.  

For each of the foodborne illness risk factors noted in the left hand column, a corresponding list of data items for each risk factor is presented in the right hand column.  In short, these summaries provide a national gap analysis – strengths/weaknesses assessment – of the food safety management system for each of the facility types.

Reducing the number of Out of Compliance observations found in these areas will have the most positive impact on the achievement of the 2010 FDA Improvement Goals for each of the facility types.

## Project Limitations

_**Field Limitations**_
**	****-- Time of the Inspection**
**	****-- Length of the Inspection**

**Notes:**
The FDA 2004 Report, like all scientific reports, includes some discussion of its own limitations.  Two limitations were imposed by the nature of the inspection process.

First the timing of the inspection could not always coincide with some of the procedure we wanted to observe, such as an establishments receiving potentially hazardous foods from their suppliers OR end-of-the-day breakdown and storage of potentially hazardous foods.

Second, we had to limit the length of our inspections.  We didn’t always have enough time to spend several hours verifying the cooling process for foods that had been cooked or reheated. 

## Project Limitations

_**Statistical Limitations**_
**  ****Some comparisons not supported by statistical Design**
**	****--  ****	****Region vs. Region, State ****	****	****vs. State**

**Notes:**
From a statistical use of the data the study does not support comparing one Region of the country with another, or one State with another State.  Our samples of each facility type are good for assessing national food safety priorities but they are too small to subdivide them further without losing their validity.  It would be wrong to divide the results by state or region of the U.S. and compare them.  The comparisons would not be reliable.

## Project Limitations

_**Statistical Limitations**_
**  ****Some comparisons not supported by statistical Design**
**	****--  ****	****Subcategories of Facility ****	****	****Types (e.g., chains of ****	****	****	****restaurants, retail food ****	****	****stores)**

**Notes:**
Similarly, there are not enough data on any one chain of restaurants or chain of grocery stores to justify a summary for just that chain.  The results would be too imprecise.

## NEW AREAS OF STUDY- Supplemental Data Items -

**Study is designed to track improvements/regression in food safety practices and employee behaviors**
**Supplement items designed to track impact of changes in the Food Code on compliance percentages in the study**

**Notes:**
The data collection form used for the study contains a section of supplemental data items.  These additional items were added in an effort to assess some industry food safety practices not addressed in the first report and also assess the impact any Food Code changes may have on the compliance percentages presented in the Report.  Only a couple of data items within the original study contain criteria or critical limits that have been changed since the release of the 1997 Food Code.

## Slide 43

**Notes:**
To illustrate, let’s look at the change in the hot holding temperature for potentially hazardous foods.  For the 1998 Baseline data collection, the critical limit for hot holding potentially hazardous food was 140o F.  The critical limit for hot holding was lowered to 135o F in the 2003 FDA Food Code Supplement.  To measure the potential impact this change might have on the hot holding IN Compliance percentages, a supplemental data item was added to assess hot holding at 135o F. 

The lowering of the critical limit to 135o F had a minimal impact on the IN Compliance percentage related to industry’s control of this data item.  None of the differences for the IN compliance percentages displayed in the far right hand column of the table meet the criteria for statistical significance.  More simply put, it can not be determined whether the differences noted are due to the lowering of the temperature or sample variation between the establishments selected for the study.  

There were not enough observations to draw any conclusions for the meat and poultry, seafood and produce facility types. 

## NEW AREA OF STUDY

**Impact of a Certified Food Protection Manager**** **
**	****4 facility types had Overall IN Compliance percentages that were significantly higher in establishments with certified managers**

**Notes:**
During the 2003 data collections, Specialists determined whether the facility being inspected has a Certified Food Protection Manager on site.  For the purposes of this study, a Certified Food Protection Manager would be defined as a person who had been certified through one of the five programs whose examinations had been recognized by the Conference for Food Protection at the time the data collection effort started.

The 2004 data results indicate that the presence of a Certified Food Protection Manager has a positive effect on the overall percent IN Compliance within some facility types

## NEW AREA OF STUDY

**-  Fast Food Restaurants****-  Full Service Restaurants****-  Meat & Poultry Departments****-  Produce Departments**

**Notes:**
-Fast Food Restaurants;
-Full Service Restaurants;
-Meat and Poultry Departments; and
-Produce Departments 

had overall IN Compliance percentages (all 42 data items combined) that were significantly higher in establishments with a Certified Food Protection Manager than in establishments without a Certified Food Protection Manager.

## Slide 46

**Notes:**
The table in this slide displays the foodborne illness risk factors for which the percent IN Compliance for establishments with a Certified Food Protection Manager was significantly higher than those without a Certified Food Protection Manager.

For Fast Food Restaurants, a Certified Food Protection Manager had a significant positive impact on the control of the:
-Improper Holding/Time and Temperature and
-Contaminated Equipment/Protection from Contamination risk factors

For full service restaurants, the presence of a Certified Food Protection Manager had a significant positive impact on the control of the:
-Poor Personal Hygiene; and
-Contaminated Equipment/Protection for Contamination risk factors

Meat and Poultry Departments and Produce Departments indicated a significant positive impact on Poor Personal Hygiene compliance percentages

## NEW AREA OF STUDY

**Impact of a Certified Food Protection Manager**
- 	**There was ****NO**** risk factor for which the IN Compliance % for establishments **_**without**_** a certified manager exceeded the % for establishments **_**with**_** a certified manager in a statistically significant manner**

**Notes:**
There was NO foodborne illness risk factor for which the IN Compliance percentage for establishments without a Certified Food Protection Manager exceeded the percentage for establishments with a Certified Food Protection Manager in a statistically significant manner.  

It is important to note, however, that though we are seeing some positive impacts the presence of a Certified Food Protection Manager has on the control of certain risk factors, we are not saying that those positive impacts are strictly  due to the presence of a Certified Manager.  There could be other things happening within the operation, such as the implementation of a comprehensive food safety employee training program or an effective food safety management system that may be contributing to the increase control of the risk factors as well,  

## NEW AREA OF STUDY

_**DATA COLLECTION TIME**_**-  determined for each facility type****-  only includes actual time in the ****   facility****-  does not include travel or report ****   writing**

**Notes:**
Many State and local regulatory agencies are planning to conduct similar foodborne illness risk factor studies within their jurisdictions.  Their intention is to establish performance measures and improvement goals specific to their retail food protection program.  To assist them in the planning process, FDA tracked the actual time spent in each of inspected establishments.

An average actual time per inspection was obtained for each of the 9 facility types.  Travel time and off-site report preparation were not included in the FDA time assessment.  Only the actual inspection time at the facility is included in the average.

## Slide 49

**Notes:**
The table in this slide presents the average data collection time, in minutes, for each of the facility types.  The average times that appear in the table are not meant to establish standards as to how long the data collection in these facilities should take.  The purpose of this information is to provide some general guidance to regulatory retail food protection programs planning to implement such a stufy.

## Pathways to Reach the Goals 

**Risk Factors **
**+**
**Intervention**
**Strategies**
** ****+ **
**Performance**
**Measures**

**FDA **
** ****Foodborne **
**        ****Illness**
**       ****Risk Factor                  **
**          ****Study**

**Standardization &**
**Certification **

** ****Program**
**    ****Standards**

**Notes:**
Several existing pathways provided by FDA interrelate to achieve a reduction in, or  prevention of, foodborne illness.  The risk factors and Food Code interventions link each of these “paths”.  

The 1st path, the FDA Foodborne Illness Risk Factor Study is  the source of the data and is used to assess thee level of occurrence of  food safety  practices and employee behaviors that  we know relate to foodborne illness.  

The 2nd  pathway is through the use of the FDA Voluntary National Retail Food Regulatory Program Standards (Program Standards).  The Standards provide a framework for assisting regulatory retail food programs to identify their inherent strengths and weakness.  Conducting self-assessments based on a set of nationally recognized standards, regulatory jurisdiction facilitate a continuous improvement process focused on the reduction of  foodborne illness risk factors.

The 3rd  pathway is through the use of FDA's Procedures for Standardization and Certification of Retail Food Inspection/Training Officers  - This document is to be used during the process of standardization and certification of retail food inspection personnel.  It provides a uniform system of measurement that is based on the Food Code, reflects current Food Code provisions, and has a focus on the risk factors, Food Code interventions, and application of the principles of Hazard Analysis Critical Control Point (HACCP).  
 This procedure measures a candidate’s knowledge and expertise in the application and interpretation of the interventions, the risk factors, HACCP principles, the use of essential inspection equipment, and lastly, necessary communication skills in conducting food establishment inspection.

## RECOMMENDATIONS FOR INDUSTRY

**ACTIVE MANAGERIAL **
**CONTROL OF FOODBORNE**
**ILLNESS RISK FACTORS**

**Notes:**
Efforts to control foodborne illness risk factors need to continue to evolve.  A different way of looking at foodservice and retail food operation is needed – one in which Active Managerial Control is used

## ACTIVE MANAGERIAL CONTROL

***Purposeful incorporation of***
***specific actions or procedures***
***by industry management ***
***to attain control of***
***foodborne illness risk factors***

**Notes:**
Although the term “Active Managerial Control” may be new to some, the underlying management principles inherent to this approach are already being used in te day-to-day operations of many foodservice and retail food establishments.  Active Managerial Control involves the purposeful incorporation of specific action or procedures by industry management to attain control of foodborne illness risk factors.

If the safety of food us to be significantly improved, foodservice and retail food store managers must establish effective managements systems to control food safety practices and employee behaviors that have the potential to contribute to the occurrence of food related illnesses. 

## RECOMMENDATIONS FOR INDUSTRY

**Develop and implement Standard Operating Procedures (SOPs) to address FBI risk factors**

**Notes:**
Food safety management systems can take many forms.  Every establishment has some type of pattern of procedures, even if it is simply described as “the way we do things.”  Some establishments have implemented effective food safety management systems by establishing controls for food preparation methods and monitoring procedures.  Many others, however, rely on vague unmonitored procedures.

Every foodservice or retail food store operation should develop and implement written standard operation procedure (SOPs) that address each of the risk factors.  These SOPs should detail procedures specific to the operation for time/temperature control of potentially hazardous food, personal hygiene, and measures to prevent food from becoming contaminated.

## RECOMMENDATIONS FOR INDUSTRY

**Provide employees with specific training and equipment to implement the SOPs**

**Notes:**
Provide the necessary resources, equipment, and supplies to implement SOPs.  Items such as temperature measurement devices, preferably thermocouples, temperature logs, chemical sanitizers, test papers, hand soap and paper towels or air drying device are crucial to the successful control of specific risk factors.

## RECOMMENDATIONS FOR INDUSTRY

**Incorporate critical limits and measurable standards for control of FBI risk factors in SOPs**

**Notes:**
Critical limits and measurable standards for control of the risk factors should be incorporated into SOPs.  At a minimum, an operator’s food safety management system should be based on achieving the same level of safety established by the critical limits in the Food Code.  Critical limits provide a means for measuring the effectiveness of an establishment’s food safety procedures.

## RECOMMENDATIONS FOR INDUSTRY

**Establish monitoring procedures that focus on critical processes and practices**

**Notes:**
Establish monitoring procedures that focus on critical processes and practices.  Monitoring procedures will only be effective if employees are given the knowledge, skill and responsibility for food safety.  

## RECOMMENDATIONS FOR INDUSTRY

**Identify methods to routinely assess the effectiveness of the SOPs**

**Notes:**
Identify method to routine assess the effectiveness of the SOPs.  This assessment should verify that key elements of a sound and effective SOP are in place.  These include:

- What process is to be controlled
- How  the processis to be controlled
- Who is responsible for ensuring the process is done correctly
- When will the process be conducted and at what times will monitoring     occur.
 Why it is important to the operation to conduct the process as specified in the    SOP

This assessment could be based on an internal review, regulatory inspection results, or third party evaluation

## RECOMMENDATIONS FOR INDUSTRY

**FDA guidance document for industry*****Managing Food Safety:  A Guide for Voluntary Use of HACCP Principles  for Operators of Foodservice and Retail Establishments***
**www.cfsan.fda.gov/~dms/hret-toc.html**

**Notes:**
To assist industry in the development of food safety management systems designed to control foodborne illness risk factors, FDA has developed a guidance document, Managing Food Safety:  A Guide for Voluntary Use of HACCP Principles for Operators of Food Service and Retail Establishments.  The FDA work directly with the Conference for Food Protection to obtain review and input from industry and regulatory food safety professionals around the country.  

The direct FDA web site link to this document is displayed at the bottom of this slide.

## RECOMMENDATIONS FOR REGULATORY

**Use risk-based inspection methodology****-  risk factors should be the primary focus of every inspection**
**Provide flexible work schedules**

**Notes:**
An assessment of an operation’s control of the foodborne illness risk factors should be the primary focus of every inspection.  The inspection methodology should be based on what food processes are occurring at the time of inspection.  Every effort should be made to check
:
Critical time/temperature relationships.  It will be important to know, at the beginning of the inspection, what items are planned for, or are in the process of cooking OR what items might be in the process of cooling.
All aspects of an effective personal hygiene program including handwashing. Prevention of bare hand contact with ready-to-eat foods and managements employee health program and policies.
Cleaning and sanitizing procedures for food contact surfaces and utensils.
Storage and preparation of raw animal foods and ready-to-eat foods.

Regulatory food safety professionals must understand the public health reasons, and be able to correctly apply the underlying regulatory requirements for the control of each risk factors.

The time of day that inspections are conducted may need to be varied based on the operation.  Inspection staff may need flexible schedules to ensure observations of critical food preparation practices that routinely occur before or after traditional working hours.  Observing and documenting practices related to risk factors, such as the cooling of potentially hazardous foods and proper handwashing, may require slightly longer inspections.

## RECOMMENDATIONS FOR REGULATORY

**Properly train and equip field personnel**
**Document compliance determination (IN; OUT; N.O.; N.A.)**

**Notes:**
Use of an inspection form that directs the inspector to routinely document the compliance status (IN, OUT, Not Observed, Not Applicable) for food safety practices and employee behaviors related to the risk factors is recommended.  Staff should be equipped with other necessary inspection tools, such as accurate temperature measuring devices (thermocouples) and sanitizer test kits, to assess an establishments compliance with the control of foodborne illness risk factors.

## RECOMMENDATIONS FOR REGULATORY

**Establish a dialog with industry foodservice and retail food store operators**
**Recognize existing industry Quality Assurance Systems and Training Programs**

**Notes:**
In order to gain a solid understanding of the food processes and food safety management systems in place within foodservice and retail food store operation, a dialog must be established between inspection staff and industry operators.  Inspectors should discuss out-of control risk factors and appropriate corrective actions with management and employees.  

In assessing any foodservice or retail food facility it is important for inspection staff to have an understanding of an establishment’s existing quality assurance and employee training programs and reinforce those components that lead to active managerial control of risk factors with their front-line employees.

## RECOMMENDATIONS FOR REGULATORY

**Take appropriate corrective action**
**Obtain immediate corrective action at the time of inspection for risk factors found out of compliance**** **

**Notes:**
Immediate corrective action for risk factors found to be Out of Compliance with regulatory requirements should be verified and documented before the inspector leaves the facility.  Efforts should be made to establish a solution that results in long-term control of food safety practices and employee behaviors that have the potential to contribute of foodborne illnesses.

## RECOMMENDATIONS FOR REGULATORY

**Take appropriate corrective action**
**Assist in developing Standard Operating Procedure and Risk Control Plans designed to attain long-term managerial control of risk factors**

**Notes:**
Many foodservice and retail food store operations have not established food safety management systems to ensure control for foodborne illness risk factors is in place.  Regulatory authorities should assist operators in developing management plans that describe the specific actions to be taken to achieve long-term control of Out of Compliance risk factors.

For example, a risk control plan is a concisely written management plan developed by the retail store or foodservice operator with input from the regulatory authority.  Risk control plans can be used as both education and compliance tools for addressing out of control risk factors.

## IMPLEMENT A CONSISTENT AND EFFECTIVE ENFORCEMENT PROTOCOL

**Develop enforcement procedures**
**Ensure credibility**

**Notes:**
There will be a few instances where facilities repeatedly fail to achieve compliance with requirements that are essential to control hazards that are serious public health concerns.  Regulatory authorities should have a well-defined, step-by-step enforcement procedure in place.  The enforcement process must be applied fairly and uniformly and with consistency when risk factors are repeatedly found to be out of control.

## VOLUNTARY NATIONAL RETAIL FOOD REGULATORY PROGRAM STANDARDS

**Standards of Excellence for **
**Continuous Improvement**

**Notes:**
FDA has developed a strategic plan directed toward establishing a national retail food program model that regulatory agencies can used to:

-identify essential food program performance requirements;
-assess strengths and gaps in the design, structure, and delivery of program services;
-establish program priorities and intervention strategies focused on reducing the occurrence of foodborne illness risk factors; and
-create a mechanism that justifies program resources and allocates them to program areas that will provide the most significant public health benefits.

The FDA Voluntary National Retail Food Regulatory Program Standards provide a framework for public health administrators to assess their retail food protection program using national recognized criteria.  Endorsed by the Conference for Food Protection, the Program Standards are not based on minimum criteria but rather set a high bar for an effective public health food safety program to aspire to.

## Voluntary National Retail Food Regulatory  Program Standards

**1. ****Regulatory Foundation**
**2. Staff Training**
**3. HACCP Principles-based Inspection  **** Program**
**4. Inspection Uniformity**
**5. Foodborne Illness and Food Security**
**	**** ****Preparedness and Response**

**Notes:**
The Program Standards are comprised of nine separate Standards.  The criteria established for each of these program areas are designed to address important retail food protection issues:

Standard No. 1 – Regulatory Foundation – Are regulatory requirements that apply to food establishments based on science and sound public health interventions?  Does the agency have the statutory authority to effectively enforce its requirements?

Standard No. 2 – Trained Regulatory Staff – Does the program ensure that personnel are properly trained and standardized in the essential elements of food safety and effective inspection and enforcement principles?

Standard No. 3 – Inspection Program Based on HACCP Principles 
Do facility inspections focus on the status of key foodborne illness risk factors and the correction of out-of-control risk factors through active managerial control by the operator?

Standard No. 4 – Uniform Inspection Program
Does tha agency have a quality assurance program that promoted uniform interpretation and application of the regulatory requirements and policies?

Standard No. 5 – Foodborne Illness and Food Security Preparedness and Response
Are reports of foodborne illness, injury, and/or intentional contamination investigated, analyzed, and documented in an effective manner?

## Voluntary National Retail Food Regulatory  Program Standards

**6. Compliance and Enforcement**
**7. Industry and Community Relations**
**8. Program Support and Resources**
**9. Self Assessment**

**Notes:**
Standard No. 6 – Compliance and Enforcement
Do agency compliance procedures result in the timely correction of out-of-control risk factors?  Are appropriate enforcement actions taken when necessary and are they applied consistently across the industy?

Standard No. 7 – Industry and Community Relations
Does the program foster communication and information exchange between it and the regulated industry, consumers, and other stakeholders?

Standard No. 8 – Program Support and Resources
Does the program have funding, staff, and equipment needed to support an effective risk-based retail food protection program?

Standard No. 9 – Program Assessment
Does the program strive for excellence in the services it provides?  Does the program employ the “best practices” to achieve a reduction in foodborne illness risk factors?

## Voluntary National Retail Food Regulatory  Program Standards

**Identify program areas where an agency can have the ****greatest impact**** on retail food safety.**
**Promote wider application of effective FBI risk factor ****intervention strategies**

**Notes:**
The Program Standards criteria are not intended as pass/fail criteria, but instead they direct an enrolled agency toward a desired set of performance outcomes.  By working toward the stated performance outcomes, an agency will position itself to achieve the greatest public health benefit given the resource constraints under which it operates.  

A primary focus of the program standards is to promote wider application of intervention strategies designed to reduce the occurrence of foodborne illness risk factors.

## Voluntary National Retail Food Regulatory  Program Standards

**Assist in identifying ****program areas most in need of additional resources**

**Notes:**
When regulatory agencies track the occurrence of foodborne illness risk factors within their own jurisdiction they are able to quantify the potential food safety risks in their communities.   By developing strategic initiatives and intervention strategies design to reduce these potential risks, each enrolled agency has in essence performed a cost/benefit analysis.  This type of cost/benefit analysis can serve as an important tool for identifying program areas most in need of additional resources. 

## Standards as a Tool for Continuous Improvement

**Standards as a ****Tool**** for Continuous Improvement**

**	****      *****Self Assess**                  ** Improve**

**Plan**

**Measure**

**Notes:**
The self assessment and baseline survey allows regulatory jurisdictions to determine where their program stands in relation to all the requirements of the Standards.  Then it’s up to them to choose which Standard or Standards to begin working to meet.  They decide where to place their resources to achieve the next level of improvement.  The idea is to work toward achieving all the Standards over a period of time.  What period of time?  That’s up to the regulatory jurisdiction and the resources they have available.

The Standards are tools that can be used in a continuous improvement cycle.  Self Assess, then select the aspect of the program you want to work to improve, meet that Standard.  Then self assess and move to the next Standard on your priority list.

## FDA Retail Food Program Resource Disk

** ****Contains all FDA’s technical guidance **** for retail food protection programs:**
**National Retail Food Program ****   Standards**** **
**FDA’s 2000 Baseline Report**** **
**Software Program & Guidance documents for jurisdictions to use to conduct their own foodborne illness risk factor studies**** **

**Notes:**
A resource disk is know available that contains all FDA’s technical guidance documents for retail food protection programs.  This includes:

-The FDA Voluntary National Retail Food Regulatory Program Standards
-The FDA 2000 Baseline Report on the Occurrence of Foodborne Illness Risk Factors;
-A software program for jurisdictions to use to enter and analyze data from their own foodborne illness risk factor studies.  The software mirrors the FDA study and will produce all the needed reports for assessing the occurrence of foodborne illness risk factors by facility types

## FDA Retail Food Program Resource Disk

**2001 Food Code (AND 2003 Food Code Supplement)**** **
**FDA Standardization Procedures**** **
**HACCP @ Retail Guides**** **
**Plan Review Guide**** **
**Temporary Events Guides**** **

**Notes:**
Other documents included on the disk include:

-the 2001 Food Code and the 2003 Supplement
-the FDA Standardization Procedures
-the agency’s HACCP Principles Guides for both industry and regulatory professionals responsible for retail food safety
-Guidance documents for plan reviews and temporary events are also contained.

The disk is available at no cost through the agency.  If you are interested in obtaining a copy of the disk for your agency, contact your FDA Regional Retail Food Specialist.

## PROGRAM GOAL

**ACTIVE MANAGERIAL CONTROL**
**OF**
**FOODBORNE ILLNESS RISK FACTORS**

**Notes:**
We should collectively keep in mind that the performance goal for enhancing the effectiveness of the nation’s retail food safety system is the reduction of foodborne illness risk factors.  One of the most effective ways to postively impact continuous progress toward this goal is to ensure the individual operators responsible for the day-to-day management of foodservice and retail food store facilities implement food safety program that attain active managerial control of foodborne illness risk factors.