Subsampling to Reduce Nonresponse Bias in the National Food Acquisition and Purchase Survey (FoodAPS) Jeffrey M. GonzalezEconomic Research ServiceDarcy Miller and Joseph RodhouseNational Agricultural Statistics Service 2021 Federal Committee on Statistical Methodology Research and Policy Conference Session G-4. Data Quality-Nonresponse Bias  Wednesday, November 3, 2021

Disclaimer: The findings and conclusions in this presentation are those of the authors and should not be construed to represent any official USDA or U.S. Government determination or policy.

Necessary steps in the development of responsive designs

From Groves and Heeringa (2006):

“...the field needs to study how the survey statistician should best model paradata from early phases. In a real sense, responsive designs are model-assisted designs, not just on sample design issues, but on all the aspects of the data collection. These models, as all models, are imperfect characterizations of the world. They need 

development, 

sensitivity analyses for alternative specifications, 

diagnostic scrutiny, 

studies of the meaning of outliers, etc.”

‹#›

National Household Food Acquisition and Purchase Survey

Relevance: First nationally representative survey to collect unique and comprehensive data on household food purchases and acquisitions

FoodAPS-1: April 2012-January 2013

All acquisitions over a 7-day period, including FAH, FAFH, and free foods

Details on food items and acquisition events

Factors that affect food purchase decisions

Focus on SNAP and low-income households

Supports research and informs policy-making on topics related to food demand, diet quality, obesity, food security, food assistance, food access, food waste, and climate change.

‹#›

FoodAPS-1 constructs, measurement, and collection protocols

‹#›

4

Households

Income

Food assistance program participation

Individuals

Demographics

Events

Place

Payment type

Items

Food security

Diet and nutrition knowledge

Total paid

Descriptions

Quantities

Expenditures

Eating occasions

School attendance

Dietary restrictions

Prices

Data quality, nonresponse, and burden in FoodAPS-1

Major findings:

Nonresponse and underreporting of food acquisition events tended to increase across the survey week (Hu et al 2017; Maitland and Li 2016)

Large households had significantly lower response rates to in-person interviews and telephone calls  (Yan and Maitland 2016) 

Large households reported significantly fewer per-person food events and food items than smaller households (Yan and Maitland 2016) but characteristics like income and SNAP-status were significantly associated with higher expenditure reporting (Hu et al 2017) 

Several socio-demographic factors, including race, ethnicity, and education, are related to the likelihood of refusing to report or failing to confirm no acquisitions (Maitland and Li 2016)

Key challenge for FoodAPS-2 is to find ways to minimize burden and increase reporting over the survey period.

‹#›

Goals for FoodAPS-2

Capture higher quality data and reduce backend processing time:

Leverage smartphone technologies (e.g., built-in camera and GPS location services)

Implement survey methodological enhancements (e.g., tailored communications and collection strategies, incentives) 

Same survey concepts as FoodAPS-1 plus new for FoodAPS-2:

12-month data collection period

Expanded target population to cover Alaska and Hawaii 

WIC as a sampling domain leading to larger WIC sample

Better identification of SNAP, WIC, and school meal program participants

Addition of new survey concepts – Feeding America, food pantry usage, veterans’ status, online ordering, etc.

Native smartphone application, FoodLogger

‹#›

6

Primary research question

Are responsive design or subsampling strategies feasible in FoodAPS-2 to achieve further reductions in response burden and improve data quality? 

Strategies conditional on participating in the seven-day food log and:

Sample characteristics and linked administrative data

Household- and individual- level characteristics

Respondent behaviors and tasks (e.g., save receipt, barcode scanning)

Survey items (e.g., report status, total amount paid)

Motivation: Certain response behaviors are less burdensome but may still yield complete and accurate data

‹#›

FoodAPS-1 data description and other sample characteristics

14,317 individuals living in 4,826 households with an overall response rate of 41.5%

Paradata includes contact and call history and acquisition report status
| Distribution of the number of days on which individuals are ...   |                |           |                |           |
|-----------------------------------------------------------------|----------------|-----------|----------------|-----------|
|                                                                 | Confirmed      |           | Unconfirmed    |           |
| Day                                                             | Percentage (%) | Count (n) | Percentage (%) | Count (n) |
| 0                                                               | 5.9            | 683       | 76.0           | 8,781     |
| 1                                                               | 2.4            | 279       | 9.4            | 1,089     |
| 2                                                               | 2.3            | 267       | 5.0            | 573       |
| 3                                                               | 2.0            | 227       | 2.0            | 225       |
| 4                                                               | 2.1            | 245       | 1.5            | 177       |
| 5                                                               | 4.9            | 566       | 1.7            | 198       |
| 6                                                               | 9.0            | 1,040     | 1.4            | 156       |
| 7                                                               | 71.4           | 8,245     | 3.1            | 352       |

Adapted from: Hu et al 2020

Confirmed acquisition indicates individuals who indicated that they did or did not acquire any food on that day.

Estimated posterior probabilities of being confirmed 

‹#›

FAH and FAFH event- and item-level information
| Method in which item was reported (FAH events) |        |         |
|------------------------------------------------|--------|---------|
|                                                | Count  | Percent |
| Scanned UPC code on package                    | 88,084 | 61.58   |
| Scanned food book barcode                      | 9,386  | 6.56    |
| Survey book                                    | 11,070 | 7.74    |
| Receipt                                        | 34,510 | 24.12   |

Total number of FAH events: 15,998

Total number of FAH items: 143,050
| Saved receipt (FAFH events) |        |         |
|-----------------------------|--------|---------|
|                             | Count  | Percent |
| No receipt was provided     | 25,290 | 66.24   |
| Indicated saved             | 10,574 | 27.70   |
| Itemized provided           | 96     | 0.25    |
| Provided, condition unknown | 2,218  | 5.81    |

Total number of FAFH events: 39,120

Total number of FAFH items: 116,074 	

‹#›

Next steps and simulation setup

Three simulation conditions

Completely random

Conditional on respondent characteristics

Conditional on observed response behaviors

Layers of evaluation

Potential for burden reduction

Traditional survey sampling features, e.g, design effects and CVs on average aggregate weekly expenditures, number of events, items per event

‹#›

Foundation for FoodAPS-2

Field Test to be conducted in 2022:

Serve as a dress rehearsal for the full collection

Incentives experiment to evaluate the effect of doubling the daily incentive for Food Log reporting at Day 4

Obtain data survey costs 

Analyze and model paradata to propose interventions for the full collection

Examples of planned paradata for collection:

Is GPS enabled?

Taps at the “Help Button” 

Receipt upload

Food item picture upload

‹#›

# image6.png

# image7.png

# image4.jpeg

# image5.jpeg

# image8.png

# image9.png

# image10.png

# image11.png

# image12.png

# image1.png

# image2.png

# image3.png