Examining the use of retrospective pretest-posttest design to evaluate the behavioral outcomes of adult participants in the Expanded food and nutrition education program
Creators: name: Chen, Qing
Publisher: name: No Publisher Supplied
Year: 2023

description: The Expanded Food and Nutrition Education Program (EFNEP) is a federal assistance program that provides nutrition education to American families and youth with limited financial resources to assist them in improving their nutritional health and well- being. EFNEP’s outcome evaluations use the pretest-posttest method, wherein participants self-report their behaviors, in part, by responding to the 30-item Adult EFNEP Questionnaire (AQ). One major threat to the validity of the pretest-posttest approach is the potential for response shift bias, which can lead to underestimation of the program outcomes. A retrospective pretest-posttest approach can be used to mitigate the possible effects of response shift bias associated with conventional pretests-posttests. However, the feasibility and validity of using a retrospective pretest-posttest methodology in evaluating EFNEP’s behavioral outcomes have not been tested before. Therefore, the aim of this dissertation was to assess the feasibility, validity, and reliability of using a retrospective pretest-posttest design to measure the behavioral outcomes among EFNEP adult participants via three separate studies.First, to examine the suitability of using a retrospective pretest-posttest design, cognitive interviews with adult participants of EFNEP (n=40) were conducted between November 2019 and September 2022. The interviews used an abridged retrospective survey, which included both retrospective pretest (i.e., “Before EFNEP”) and posttest (i.e., “Now”) versions of 5 questions that represented each of the main educational domains addressed by EFNEP (Diet Quality [referred to as Diet in this dissertation to differentiate this domain from one of its subscales which was also named Diet Quality; see Chapter 5], Physical Activity, Food Resource Management, Food Safety, and Food Security). The results suggested that most participants could appropriately differentiate the timeframes of the retrospective pretest and posttest questions, and they preferred the side-by-side (versus the separate-page) retrospective survey design because of its ease of use.Informed by the cognitive interviews, a retrospective AQ was developed for the second study. The aim of this study was to assess the overall performance of the retrospective pretests-posttests in comparison to the conventional pretest-posttest surveys when used for EFNEP’s behavioral outcome evaluations. Two different designs, one presenting the retrospective pretest questions and the other presenting the posttest questions first, were pilot tested in adult EFNEP lesson series between March and September 2022. Results based on the secondary data analyses among 252 adult participants suggested the presence of response shift bias in the pretest-posttest surveys. The results indicated that the retrospective AQ captured the response shift bias better when the survey design showed the retrospective pretest questions first and the posttest questions next, rather than the version showing the posttest questions first. This retrospective evaluation method demonstrated adequate reliability and convergent validity and may be a viable alternative measurement approach to the conventional pretests-posttests for EFNEP’s impact assessments.The last study described in this dissertation further investigated potential response shift bias through reconceptualization and reprioritization, and motivational biases (i.e., social desirability and effort justification) regarding the pretest-posttest surveys. Follow- up cognitive interviews with adult EFNEP participants (n=13) who had attended English lesson series were conducted between December 2022 and May 2023. The results indicated the potential presence of response shift bias primarily through reconceptualization, as well as social desirability bias.Collectively, these three studies provided evidence to support the use of the retrospective pretest-posttest design in evaluating the behavioral outcomes among adult EFNEP participants. The results suggested that the retrospective pretests-posttests in a side-by-side format were preferred by the EFNEP participants. When presented with retrospective pretest questions first, this survey format seemed to better capture participants’ behavioral changes in cases where response shift bias is likely to be present. Overall, retrospective data collection appeared to be a suitable alternative to the conventional pretest-posttest surveys for evaluating EFNEP’s behavioral outcomes. Future research should include the examination of criterion validity and time-efficiency of the retrospective pretest-posttest design.