# Presentation: 110467

## Assessing a Food Safety Behavior Questionnairefor Criterion Validity

- Patricia Kendall, Mary Schroeder,
- Kelly Sinclair & Anne Elsbernd
- Colorado State University
- Lydia Medeiros and Gang Chen
- The Ohio State University
- Virginia Hillers and Verna Bergman
- Washington State University

## Introduction

- Food safety education programs generally rely on self-reported behavioral questions administered pre and post education to measure program impact.
- Observational studies indicate that errors in food handling are more common than reported on questionnaires (Jay et al. 1999; Anderson et al., 2000)

## Purpose

- Develop a validated bank of food safety behavior questions that could be used with confidence when evaluating consumer education programs.
- Determine if self-reported behaviors can be a valid way to assess behavioral outcomes of food safety education programs among low-income groups.

- Medeiros, Hillers, Kendall 1999-2001
- UDSA grant #99-35201-8126

## Development of Behavioral Questions

- Sub-group (n=8) from Expert Panel developed behavioral questions for each of 29 behaviors identified by Expert Panel as being important in reducing risk of foodborne illness in the home.
- First draft reviewed for content and face validity by tri-state team, faculty in three states, and 2 groups from target audience.
- Questionnaire revised and shortened based on feedback received.

## Question Bank

- Question Bank to be tested for reliability and validity contained 52 behavior questions:
- Practice personal hygiene (5 questions)
- Cook foods adequately (12)
- Avoid cross contamination  (7)
- Keep foods at safe temperatures (12)
- Avoid foods from unsafe sources (16)

## Questionnaire

- Two part questionnaire addressed food safety issues for the general public and those specific to pregnant women
- Contained a variety of question types:
- 5 point Likert scale (20 questions)
- Dichotomous Y/N (41)
- Multiple choice (1)

## Reliability Testing

- Test/retest:
- Target audience members (n=20) took questionnaire at 2 time points; responses correlated and compared via paired t-tests
- Questions considered reliable if:
- P-value > 0.05 & r  0.70 or Agreements/Agreements 				             +Disagreements  70%
- Internal consistency:
- Assessed using Cronbach alpha; run on all items within a particular construct
- Questions with   0.60 considered internally consistent (Osterhof, 2001; Taylor et al., 2001; Murphy et al., 2001)

**Notes:**
Target audience members included WIC paritcipants (7), Greeley EFNEP participants (9) and University Village residents in a parent/toddler group (4)

## Reliability Results

- Test/Retest:
- 47 of 52 questions met reliability criteria
- Internal consistency:
- 				           Cronbach alpha
- Personal Hygiene: 			        .60
- Cook Foods Adequately: 	  	        .90
- Avoid Cross-contamination: 	        .46
- Keep Foods at Safe Temperatures:          .76
- Avoid Unsafe Foods: 		  	        .06

**Notes:**
The questions with no variability were those for which respondents answered the same way on test and retest; it seemed silly that these questions could not be reliable despite high correlation coefficients (1.00), A/A+D values (100%) or both. 

Alphas for  the CC and AF constructs were low because of the nature of the questions- AF included a variety of food preference questions for which people who knew the items were unsafe had varying levels of concern – yes I will eat raw cookie dough but no, I will drink raw milk or untreated water from a stream

CC- all 50 subjects reported “most of the time” or “always” handwashing after handling raw meat but fewer subjects reported washing their fruits (38) or vegetables (47) “most of the time” or “always” before eating them

## Validity

- Degree to which an instrument measures what it is intended to measure
- Assessed several types of validity:
- Content Validity: Reflects domain of content to be measured
- Face Validity: Measures what intended to measure
- Criterion Validity: Correlates with other more accurate instrument

## Validity Testing

- Criterion Validity – focus of this study.
- Established by comparing questionnaire response to observed behavior and interview responses during a kitchen activity session held ~ one week later.

## Validation Study Subjects

- 70 FSNEP and EFNEP participants in CO, WA, and OH
- 50 post education only
- 20 pre and post education
- Primary food preparers
- Had completed an education program that included a 30- to 60-minute food safety component

## Study Design  Post-Education

-   Week  4            Week  5           Week  6 to 7

## Study Design  Pre-Post Education

-     Week  2               Week 3              Week 4            Week 5           Week 6-7

- Recruit &
- Take questionnaire

- Observation & Interview session

- Observation
- & Interview session

-  Food
- Safety
-  Class

-  Take
- Question-
- naire

## Kitchen Activity Session

- Cooking Observation in Community Kitchen:
- Cook a chicken breast to desired doneness
- Slice an apple to garnish the chicken
- Cook a hamburger to desired doneness
- Slice a tomato to go with the hamburger
- In-depth interview
- Asked the same questions on the questionnaire in a conversational, open-ended manner

## Kitchen Activity Protocols

- Extensive training of research assistants on conducting cooking observations and interview sessions.  Mock interviews conducted & videotaped.
- Two research assistants conducted each session (safety reasons).
- Each subject provided with same food items, utensils, equipment, instructions.
- Cooking sessions videotaped and interviews audio-taped.
- Actions and responses coded by research assistant who conducted session, then re-coded by one researcher in Colorado and responses compared to ensure comparability of data.  Differences in coding reviewed by 3rd party and resolved.

## Limitations

- Non-randomized design
- Observations not performed in homes
- No interruptions
- Subjects could focus on food preparation/cooking
- Intervention for pre/post design wasn’t controlled

**Notes:**
Behaviors performed incorrectly:

Hand washing after contact with raw poultry or hamburger (76% and 43%), countertop washing after food preparation (76%) incorrectly, cutting board washing after raw poultry (21%) or after raw meat (17%) and before ready to eat fruit or vegetables 

People do not clean their countertops before preparing food........even those with pets wandering around the house all day

Only 18% of subjects used a thermometer with chicken and 20% used a thermometer with hamburger while the majority of subjects( greater than 90%)  cooked their chicken or hamburgers to 160 degrees F.
In Anderson et al, only 5% of subjects used a thermometer and fewer cooked these items to 160 degrees F (165F for the chicken) = 52-58% of subjects

## Validation Design

- *Validation criteria:
-   Observable behaviors: 70 % agreement among all 3 instruments
-                                          (counting correct & incorrect behaviors)
-   Non-observable behaviors:  70 % agreement between
-                                                 questionnaire & interview

## Validity Results

- Observable Questions:
- 54.5% (6 of 11) met validity criteria
- Non-observable Questions:
- 66% (27 of 41) met validity criteria

## Validity Results

| Control factor | # of valid questions | # of invalid questions |
| --- | --- | --- |
| Personal hygiene | 5 | 0 |
| Cook foods adequately | 6 | 6 |
| Cross-contamination | 3 | 4 |
| Safe temperatures | 8 | 4 |
| Avoid foods | 11 | 5 |
| Total questionnaire | 33 | 19 |

**Notes:**
This table shows the results for each control factors, detailing the number of questions that met our validity criteria

## Instrument Sensitivity

- Potential ceiling effect:
- Good questions are those that capture range of responses
- Looked for questions that 20-80% gave less desirable response at pre education
- Change in mean scores from pre to post education (n=20)
- 	                         (Parmenter and Wardle, JNE 32:269; 2000)

## Instrument Sensitivity Results

- Among validated questions, several in 4 of 5 control factors showed  good response variety pre-workshop, with room for change.
- Pre and post scores on Cross-contamination questions generally high, but improvements needed in skill level.
- Improvements in behavior pre to post seen for washing hands prior to cooking and not leaving meat on counter.

## Conclusions

- 33 of 52 behavioral questions met reliability and validity criteria (70% agreement), including several questions from each pathogen control factor.
- Agreement between observed and self-reported behaviors was better when incorrectly performed behaviors were included.
- Further study is needed using these questions in educational settings with controlled interventions.

## Thank You !