Response burden and bias in the National Household Food 

Acquisition and Purchase Survey (FoodAPS-1): 

an empirical analysis based on the respondent feedback survey 

Xingyou Zhang 

John Kirlin 

Mark Denbaly 

Elizabeth Larimore 

Elina Page 

Economic Research Service (ERS), USDA 

FCSM Research and Policy Conference 

Washington, DC 

March 7-9, 2018 
The findings and conclusions in this presentation are those of the authors and 

do not necessarily represent the official position of Economic Research 

Service, or U.S. Department of Agriculture. 
Outline 

Introduction: FoodAPS-1 data collection 

FoodAPS feedback survey questionnaire 

Research questions 

Model results for response bias, behavior change and response 

burden 

Implications for future FoodAPS data collection 
FoodAPS Data Collection Week 

Mon Tues Wed Thu Fri Sat Sun Mon Tue 

0 1 2 3 4 5 6 7 8 

• Screen 

• Train HH 

Initial 

• Pick up 

• • Final Household tracks food acquisitions 

Interview in booklets and with scanner Interview 

(CAPI) (CAPI) 

•Feedback 

Survey 

PR call to report PR calls to report PR call to report 

4 
FoodAPS response rates 

• Screening interview 

• 70.4% (unweighted) 

• 71.0% (weighted)* 

• Main study 

• 44.5% (unweighted) 

• 43.2% (weighted) 

*Weighted with unadjusted sampling weights (inverse of 

probability of selection). 

• FoodAPS feedback survey 

• 97.6%(unweighted, 4,712 out of 4,826) 

5 
 

 

Feedback survey questionnaire 

• How often did you • How easy or difficult was 

complete the meals and it to keep track of the 

snack forms? (N=4624) foods you got? (N=4668) 

– Everyday (72.2%) – Very easy (41.7%) 

– More than once but not – Easy (38.2%) 

every day (18.1%) – Neither easy nor difficulty – Once before the end of the (14.3%) 

week (1.8%) – Difficult (4.7%) – Once at the end of the – Very difficult (1.2%) week (4.2%) 

– Did not complete at all 

(3.7%) 

6 
Feedback survey questionnaire 

• During the past week, did – Ate out more often (2.4%) 

you (or other household – Ate out less often (3.6%%) 

members) change the – Did more food shopping (2.5%) 

way you got food because – Did less food shopping(3.5%) 

you were taking part in – Bought a specific item(s) just to 

be able to scan it (1.2%) this study? (N=4,666) – Avoided specific items so you 

wouldn’t have to scan them

(0.5%) 

– Other changes (2.7%) 

– No, did not change (89.3%) 

7 
 

 

Research questions 

• Is the FoodAPS Primary Respondent (PR)’s delay in

food acquisition reporting associated with his/her 

actual response burden? 

• Does the PR’s participation in FoodAPS data 

collection change his/her food purchase and 

acquisition behaviors? 

8 
Response bias and response burden 

• FoodAPS PR’s response bias (binary outcome) is

defined as failing to report his/her meals and 

snacks everyday (delay in reporting or no 

reporting) 

– 27.8% PRs did not report meals and snack daily 

• FoodAPS PR’s actual response burden is defined as

feeling difficult or very difficult to keep tract of the 

food he/she got? 

– 5.9% PRs had difficulties in tracking their food. 

9 
 

Multilevel logistic models (MLM) for the associations 

between response bias and response burden 

▪ Model (I) with only PR’s response burden

– Having difficulties in tracking food (yes vs no) 

▪ Model (II): Model I plus PR’s demographics

– Sex: male vs female 

– Age (years): 18-24, 25-29, ..., 80+

– Race/ethnicity: Non-Hispanic white, black, AIAN, Asian, HNPI, 

other single race, two or races, Hispanic 

▪ Model (III): Model II plus county-level urbanrural status 

– NCHS county urban-rural continuum (2013): 

• Metro: large central, large fringe, medium and small 

• Non-metro: Micropolitan, noncore 

10 
Fixed effect odds ratio (OR) and random effect variance estimates of MLM 

for the associations between response bias and response burden 

Model I II III 

Fixed effects 

Predictor OR (95% CIs) OR (95% CIs) OR (95% CIs) 

Having difficulties 

(Yes vs No) 4.31 (3.30, 5.64) 4.07 (3.11, 5.33) 4.07 (3.10, 5.33) 

Random effects 

Cluster Variance (SE) Variance (SE) Variance (SE) 

county 0.103 (0.038) 0.059 (0.030) 0.044 (0.026) 

tract(county) 0.071 (0.039) 0.040 (0.036) 0.045 (0.037) 

11 
Implication for FoodAPS data collection 

▪ PR’s actual response burden, independent of his

demographics, is a significant barrier for FoodAPS PR 

to report meals and snacks daily; the delay in recalling 

meals and snacks could introduce substantial bias in 

reporting events (missing, over-reporting, and 

underreporting 

▪ Compared to rural counties (PSU), PRs from large 

central metro and micropolitan counties are more likely 

to delay in reporting meals and snacks, more field 

interventions are needs to minimize this adverse effect 

in data collection 

12 
PR food acquisition behavior change 

• FoodAPS PR food acquisition behavior change 

(binary outcome) is defined as those PR reporting 

one or more food acquisition behavior changes 

during foodAPS data collection 

– 10.7% PRs reported food acquisition behavior 

13 
 

Multilevel logistic models (MLM) for the associations 

between behavior change and response burden 

▪ Model (I) with only PR’s response burden

– Having difficulties in tracking food (yes vs no) 

▪ Model (II): Model I plus PR’s demographics

– Sex: male vs female 

– Age (years): 18-24, 25-29, ..., 80+

– Race/ethnicity: Non-Hispanic white, black, AIAN, Asian, HNPI, 

other single race, two or races, Hispanic 

▪ Model (III): Model II plus county-level urbanrural status 

– NCHS county urban-rural continuum (2013): 

• Metro: large central, large fringe, medium and small 

• Non-metro: Micropolitan, noncore 

14 

14 
Fixed effect odds ratio (OR) and random effect variance estimates of MLM 

for the associations between behavior change and response burden 

Model I II III 

Fixed effects Predictor OR (95% CIs) OR (95% CIs) OR (95% CIs) 

Having difficulties 

(Yes vs No) 1.90 (1.38, 2.64) 1.80 (1.30, 2.51) 1.77 (1.27, 2.46) 

Random effects Cluster Variance (SE) Variance (SE) Variance (SE) 

county 0.078 (0.038) 0.041 (0.032) 0.014 (0.028) 

The behavior change binary outcome did not any meaningful census tract-level correlations, 

thus census tract-level random effects were dropped for all MLMs for behavior change. 

15 
 

Implications for FoodAPS data collection 

▪ PR’s actual response burden is significantly associated

with behavior changes in food purchase and 

acquisition, which could introduce reporting bias in field 

data collection 

▪ Compared to rural counties (PSU), PRs from large 

central metro counties were likely to experience food 

purchase and acquisition behavior changes 

16 

16 
 

What is next? 

▪ As expected, PR’s actual response burden could

increase underreporting and introduce substantial bias 

in field data collection. 

▪ Additional analysis is needed to explore and identify 

what individual and/or local community factors could 

reduce and minimize the adverse effects of response 

burden during data collection. 

– Interviewer effects? 

– Employment status, language, household with children, or house 

size? 

17 

17 
Thanks you! 

Questions? 

Xingyou.zhang@ers.usda.gov 

18 
 

Fixed effect odds ratio (OR) estimates of MLM for the associations 

between response bias and response burden 

19 

Model I II III 

Predictor Subgroup OR 95% CIs OR 95% CIs OR 95% CIs 

LCL UCL LCL UCL LCL UCL 

Sex Male 1.26 1.08 1.47 1.27 1.09 1.48 

Female (ref.) 

Age (years) 18-24 (ref.) 

25-29 0.76 0.56 1.02 0.76 0.56 1.02 

30-34 0.95 0.71 1.27 0.95 0.71 1.27 

35-39 0.79 0.58 1.07 0.79 0.58 1.08 

40-44 0.69 0.51 0.94 0.69 0.51 0.94 

45-49 0.68 0.50 0.92 0.67 0.50 0.91 

50-54 0.67 0.49 0.91 0.67 0.49 0.91 

55-59 0.52 0.38 0.72 0.53 0.38 0.73 

60-64 0.55 0.39 0.77 0.55 0.39 0.78 

65-69 0.68 0.47 0.97 0.67 0.47 0.97 

70-74 0.89 0.59 1.34 0.89 0.59 1.35 

75-79 0.81 0.49 1.33 0.81 0.50 1.33 

80+ 0.60 0.37 0.99 0.60 0.37 0.98 

Race/Ethnicity White (Ref.) 

Black 1.51 1.23 1.86 1.49 1.22 1.83 

AIAN 1.33 0.61 2.90 1.28 0.59 2.78 

Asian 1.30 0.91 1.88 1.28 0.88 1.84 

HNPI 0.56 0.15 2.14 0.53 0.14 2.05 

Other race 1.08 0.48 2.45 1.04 0.46 2.35 

Multiple race 1.21 0.72 2.02 1.18 0.71 1.98 

Hispanic 1.80 1.48 2.18 1.71 1.40 2.09 

Having difficulities Yes 4.31 3.30 5.64 4.07 3.11 5.33 4.07 3.10 5.33 

No (ref.) 

County 

Urban 

Rural 

Status 

Large central metro 1.49 1.07 2.08 

Large fringe metro 1.29 0.92 1.79 

Medium metro 1.12 0.79 1.58 

Small metro 1.24 0.84 1.84 

Micropolitan 1.67 1.17 2.38 

Noncore (ref.) 

19 
 

Fixed effect odds ratio (OR) estimates of MLM for the associations 

between behavior change and response burden 

20 

20 

Model I II III 

Predictor Subgroup OR 95% CIs OR 95% CIs OR 95% CIs 

LCL UCL LCL UCL LCL UCL 

Sex Male 1.22 0.99 1.50 1.21 0.98 1.50 

Female (ref.) 

Age (years) 18-24 (ref.) 

25-29 0.76 0.51 1.12 0.75 0.50 1.11 

30-34 0.97 0.66 1.42 0.97 0.66 1.41 

35-39 0.88 0.59 1.31 0.87 0.58 1.30 

40-44 0.65 0.43 0.99 0.65 0.43 0.98 

45-49 0.69 0.46 1.04 0.68 0.45 1.03 

50-54 0.52 0.33 0.81 0.51 0.33 0.80 

55-59 0.65 0.42 1.00 0.65 0.42 1.00 

60-64 0.49 0.30 0.81 0.50 0.31 0.82 

65-69 0.71 0.44 1.17 0.71 0.44 1.17 

70-74 0.47 0.24 0.93 0.48 0.25 0.95 

75-79 0.51 0.24 1.12 0.51 0.23 1.11 

80+ 1.14 0.64 2.04 1.14 0.64 2.04 

Race/Ethnicity White (Ref.) 

Black 1.76 1.35 2.29 1.71 1.31 2.22 

AIAN 0.61 0.15 2.57 0.59 0.14 2.50 

Asian 1.07 0.64 1.79 0.97 0.58 1.62 

HNPI 2.17 0.59 7.97 1.94 0.53 7.13 

Other race 3.56 1.58 8.03 3.18 1.41 7.21 

Multiple race 1.86 1.00 3.45 1.78 0.96 3.29 

Hispanic 1.59 1.23 2.06 1.43 1.10 1.86 

Having difficulities Yes 1.90 1.38 2.64 1.80 1.30 2.51 1.77 1.27 2.46 

No (ref.) 

County Large central metro 1.89 1.22 2.91 

Urban Large fringe metro 1.37 0.88 2.14 

Rural Medium metro 1.56 0.99 2.45 

Status Small metro 1.47 0.89 2.43 

Micropolitan 1.30 0.80 2.12 

Noncore (ref.)