Correlates of Item Nonresponse to Open-Ended Web Probes

Zachary Smith1,2,3, Kristen Cibelli Hibben1,2,3, Benjamin Rogers2,3, Valerie Ryan1,2,3, Paul Scanlon1,2,3, 

and Travis Hoppe3

1Collaborating Center for Questionnaire Design and Evaluation Research

2Division of Research and Methodology

3National Center for Health Statistics

Federal Committee on Statistical Methodology Research and Policy Conference

Washington, DC

October 25-27, 2022

National Center for Health Statistics

The findings and conclusions in this presentation are those of the authors and do not necessarily represent the official position of 

the National Center for Health Statistics, Centers for Disease Control and Prevention. 
Outline

 Item nonresponse in open-ends: the problem

 A semiautomated model for detecting item nonresponse

 Web probes evaluated and model validity

 Subgroup variation in model-coded nonresponse

 Implications for question evaluation
Item nonresponse in open-ends: the problem
Value and challenges of working with open-text data

 Wide range of methodological uses (Singer & Couper, 2017)

 Because responses are unconstrained, particularly useful when little is 

known about a topic (Neuert et al., 2021, Scanlon, 2019; 2020) 

 However: 

• More burdensome to respondents

• Prone to nonresponse or inadequate, non-codable responses

• Coding and analysis is time- and labor-intensive for researchers
A semiautomated model for detecting item 

nonresponse
Building a model to detect nonresponse

 Prior work:

• Categorizing item nonresponse (Behr et al., 2012; Meitinger et al., 2021)

• Detecting item nonresponse via rule-based approach (Kaczmirek et al. (2017); available on 

GitHub)

 Leveraging advances in data science to build a more accurate detector

 Trained a natural language processing (NLP) model

 Refined with human coding (active learning)

 EvalAnswer: https://git.gesis.org/surveymethods/evalanswer

 Bidirectional Transformer for Language Understanding (BERT): https://arxiv.org/abs/1810.04805

 Simple Contrastive Sentence Embedding (SimCSE): https://arxiv.org/abs/2104.08821
Taxonomy of responses

 The model assigns a score (0-1) for the extent to which a response falls 

into each of the item non-response categories

• Complete non-response: Blank text box

• Gibberish or nonsensical: “dfgjh”

• Don’t knows: “I don’t know”; DK; idk

• Refusals: “no comment”; “Because”; “none”

• Other, high-risk: non-useful response, non-codable

• Valid: useful response, codable

 Several rounds of arbitration and refined training produced latest model 

version
Web probes evaluated and model validity
Data source

 NCHS’s Research and Development Survey (RANDS) During COVID-19 

https://www.cdc.gov/nchs/rands/index.htm

• Three-round web/phone survey

• Focused on health, impacts of pandemic, behaviors

 Conducted using NORC at the University of Chicago’s Amerispeak®, a 

probability-based panel representative of the US adult, English-speaking, 

non-institutionalized household population

 Round 3 fielded May-June 2021: 5,458 Completes

• 7,852 NORC’s AmeriSpeak probability-based sample = 11.8% weighted cumulative 

response rate / 69.5% completion rate
Web probes evaluated

 Vaccine hesitancy:

• Please list the reasons you say you [are / are not] hesitant about vaccines in 

general.

 Social distancing:

• When you were answering about social distancing in the previous questions, what 

were you thinking about?

 Religion:

• Currently, how important is religion in your daily life? (Very, somewhat, not 

important)

• Why do you say that?
Model validity

 Manually evaluated all coded soft nonresponses and 1,000 randomly 

sampled coded valids to determine model sensitivity and specificity

 Overall:

• Sensitivity: 83.6%

• Specificity: 86.5%

 Hesitancy:

• Sensitivity: 77.7%

• Specificity: 89.7%

 Distancing:

• Sensitivity: 81.9%

• Specificity: 95.6%

 Religion:

• Sensitivity: 90.1%

• Specificity: 70.9%
Subgroup variation in model-coded nonresponse
Subgroup variation: age
Subgroup variation: gender
Subgroup variation: race/ethnicity
Subgroup variation: education
Subgroup variation: device
Pulling all the data together

 Logistic regression to estimate odds of nonresponse by subgroup for a) all 

probes and b) the social distancing probe

 Reference categories:

• Age: 18-29

• Gender: Male

• Race/ethnicity: Non-Hispanic White

• Education: Some college or less

• Device: Desktop computer

• Interaction of race/ethnicity and education: NH White, Some college or less

 95% confidence intervals shown

 Analysis run in R 4.2.0 and Rstudio 2022.02.03 using tidyverse and sjPlot

packages
Odds of Nonresponse by Subgroup, All Probes
Odds of Nonresponse by Subgroup, Social Distancing
Implications for question evaluation
How can this model assist in question evaluation?

 Speedily categorizes open-text data with reasonable sensitivity and 

specificity

 Clear understanding of demographics of non-responders – potential for 

insight into patterns of nonresponse that can improve question design

 But, some dangers: reliance on the coded valid dataset excludes:

• Potentially valid responses missed by the model (false negatives)

• The voices of groups systematically more likely to be categorized as nonresponse
For more information, contact CDC

1-800-CDC-INFO (232-4636)

TTY: 1-888-232-6348 www.cdc.gov

The findings and conclusions in this report are those of the authors and do not necessarily represent the 

official position of the Centers for Disease Control and Prevention.

Thank you!

For more information contact: Zachary Smith, zsmith@cdc.gov

Q-Bank: providing access to survey question evaluation reports, question design and performance https://wwwn.cdc.gov/qbank/

Q-Notes: designed to facilitate the management and analysis of cognitive 

interviews https://www.cdc.gov/nchs/ccqder/products/qnotes.htm