Examining Response Fatigue and Noncooperation in the National Household 

Food Acquisition and Purchase Survey 

(FoodAPS)

Consumer Expenditure Survey

Survey Methods Symposium

July 12, 2016

John A. Kirlin, USDA/ERS
2

2

Acknowledgements and Disclaimer

• Co-authors Brady T. West, Mengyao Hu, and 

Garrett Gremel (University of Michigan)

– Cooperative assistance grant #59-5000-5-0008

• The views expressed are those of the authors 

and should not be attributed to the Economic 

Research Service or the U.S. Department of 

Agriculture.
3

3

What’s the issue?

• FoodAPS is a diary survey designed to collect data on all 

food acquisitions by household members over a 7-day 

period.

– “Acquisition” is an event in which food is purchased or 

otherwise acquired

• Potential for non-reporting of food acquisitions in FoodAPS 

by household members

– Refuse to participate on given day or entire week

– Report some but not all food events on a given day

– Report some but not all food items for a given event
4

4

Today’s focus

• Potential for non-reporting of food acquisitions 

in FoodAPS by household members

–Refuse to participate on given day 

or entire week (at person level)

– Report some but not all food events on given day

– Report some but not all food items for a given event
5

5

Research questions

• How often did members refuse to 

participate?

• Who refused to participate?

• Was there variation in the rate of refusals 

over data collection week?

– Any evidence of response fatigue?
6

6

Data source?

• Survey paradata from FoodAPS

• But first, need to understand overall 

structure of data collection in 

FoodAPS
7

7

Data collection in FoodAPS

• Primary respondent (PR) identified as 

the person “who does most of the 

grocery shopping or who is the main 

meal planner”

• 2 in-person interviews

• All members 11 or more years old track 

their food acquisitions

• PR gathers info and calls survey center 3 

times during week
8

PR calls 

to report

PR calls to 

report

Mon Tues Wed Thu Fri Sat Sun Mon Tue

0 1 2 3 4 5 6 7 8

PR calls to 

report

Initial Interview and 

training

Members track food acquisitions 

in booklets and with scanner

Final Interview

Screener

Overview of planned data collection week

8

Feedback Survey
9

9

Phone calls provide paradata

• For each household member on each 

day, whether status was:

– Acquired food (confirmed yes)

– Did not acquire food (confirmed no)

– Refused to provide info (refused)

– PR didn’t know (unconfirmed)

• 14,317 members → 100,219 person-days
10

10

Results

• How often do members refuse to 

participate?

• Who refuses?

• Is there variation over data 

collection week (response fatigue)?
11

11

How often do members refuse to 

participate?

50.7%

38.3%

3.3%

7.7%

0%

10%

20%

30%

40%

50%

60%

Food No food Refused Don’t know
12

12

Collapsed distribution

88.9%

3.3% 7.7%

0%

20%

40%

60%

80%

100%

Confirmed Refused Don’t know
13

13

To address questions of who and 

week-day effects...

• Model member reporting status

– Multinomial logistic regression

– Confirmed (reference), refused, 

unconfirmed
14

14

Covariates

• Sex, age, education, race, marital status, 

income, BMI, completed Meals & 

Snacks form

• Relationship to PR

• Household size, days since last SNAP 

issuance

• Self-reported financial condition, 

frequency of bill review, frequency of 

paying bills on time
15

15

Paradata covariates

• Difficulty to get members to participate, 

ease of keeping track of foods 

• Numbers of inbound and outbound calls

• Start-date cohort (day of week)

• Fielding day (continuous and centered 

as 0-6)

• Interaction of start-date and fielding day
16

16

Who refuses to participate?

... by marital status

2.5% 2.9% 2.9% 1.7%

6.4%

6.1%

9.5%

5.8% 7.5%

9.6%

0%

5%

10%

15%

20%

Refused Unconfirmed
17

17

Who refuses to participate?

... by marital status

2.5% 2.9% 2.9% 1.7%

6.4%

6.1%

9.5%

5.8% 7.5%

9.6%

0%

5%

10%

15%

20%

Refused Unconfirmed
18

18

Who refuses to participate?

... by relation to PR

0.1% 4.2%

11.4%

16.3%

5.9%

7.8%

15.5%

18.8%

0%

10%

20%

30%

40%

Refused Unconfirmed
19

19

Who refuses to participate?

... by age group

1.3% 3.3% 6.3% 6.4%

3.0% 2.4%

10.3% 7.0%

9.7% 9.4%

7.2%

5.6%

0%

5%

10%

15%

20%

Refused Unconfirmed
20

20

Who refuses to participate?

... by race

2.8%

5.3% 4.4%

0.3%

6.4%

11.1% 12.8%

7.8%

0%

5%

10%

15%

20%

Refused Unconfirmed
21

21

Who refuses to participate?

... by ease of getting members to participate

1.4% 4.2%

14.0%

4.8%

5.8%

9.3%

16.8%

9.7%

0%

5%

10%

15%

20%

25%

30%

35%

Refused Unconfirmed
22

22

Who refuses to participate?

... by ease of keeping track of foods

2.9% 4.8% 4.4% 5.5%

5.8%

13.6% 15.6%

16.7%

0%

5%

10%

15%

20%

25%

30%

Refused Unconfirmed
23

Temporal effects and response fatigue

• Graphs are now predicted probabilities 

of reporting status by fielding day

• Covariates are:

• Fielding day (continuous and centered), 

• Start-date (Sun, Mon,..etc), and 

• Interaction of fielding day and start-date
24

Predicted refusals by fielding day

Friday (ref) Sunday **

Saturday **
25

Predicted confirmeds by fielding day

Sunday Sunday Monday

Wednesday
26

Predicted unconfirmeds by fielding day

Sunday

Wednesday

Monday
27

Again, predicted refusals by 

fielding day

Friday (ref) Sunday **

Saturday **
28

Reality check (same scale)

Confirmed Refused

Not confirmed
29

29

Recap – Variation over time?

• Some evidence of response fatigue

• But seems to be focused on those 

households that start data collection 

on a Friday, Saturday or Sunday

• Survey protocol led to many 

unconfirmed days of no reported 

acquisitions near end of week
30

30

Recap - Who most likely to refuse?

• Older teens, adults and seniors

• Never married

• African-Americans

• Non-relatives and “other” relatives 

of PR
31

Implications for Diary Surveys 

• Day-by-day variation in response rates may be 

important and informative

• Response fatigue exists in FoodAPS, but not to 

a great degree

• Is there a cohort effect? Maybe, but why? 

Worthy of additional research.
32

Implications (cont) 

• For next FoodAPS, need to work on reducing 

the unconfirmeds

• Provide extra training, reminders, or other 

inducements to those expected to have 

higher rates of non-participation.
33

Thank you!

John A. Kirlin

jkirlin@ers.usda.gov

http://www.ers.usda.gov/data-products/foodaps-nationalhousehold-food-acquisition-and-purchase-survey.aspx
34

Extra slides, if needed
35

35

Key Characteristics of FoodAPS

• Fielded from April 2012 to January 

2013

• Oversampled households participating in 

the Supplemental Nutrition Assistance 

Program (SNAP) and other low-income 

households

• Nationally representative, N ≈ 5,000
36

FAH vs. FAFH

• FAH

• Groceries and other foods and drink s you 

brought home

• FAFH

• Meals, snacks, and drinks you got outside 

your home
37

37

Who refuses to participate?

... by days since SNAP issuance

4.1% 4.3% 5.7%

11.8%

3.0%

12.3% 10.8%

12.4%

14.7%

6.9%

0%

5%

10%

15%

20%

25%

30%

Refused Unconfirmed
38

38

Status by fielding day

3.0% 3.3% 3.4% 3.5% 3.4% 3.3% 3.2%

5% 5% 7% 7% 8% 10% 12%

0%

5%

10%

15%

20%

1 2 3 4 5 6 7

Fielding Day

Refused Unconfirmed
39

39

Who refuses to participate?

... by inbound calls