The Importance of Nonresponse Follow-up 

in Probability Panels 

2022 FCSM Research & Policy Conference

Session B-6: Leveraging Probability Panels for Federally Sponsored 

Statistical Data Collections 

Ipek Bilgen, Sr. Research Methodologist, NORC

J. Michael Dennis, SVP, NORC

David Dutwin, SVP, NORC
INTRODUCTION 2

Federal surveys for decades have rightly held high standards for low 

nonresponse

• In today’s age of low response rates, an effective strategy for maximizing a weighted 

response rate is a nonresponse follow-up (NRFU) protocol 

• Within the last decade, probability-based panels are growing in acceptance by the 

research community

– Used in variety of fields – Social research, market research, medical research, election studies, etc.

– Nonresponse during the recruitment stage is as a big danger to accurate estimates in probability-based 

web panels

• This study explores the benefits of NRFU during recruitment on AmeriSpeak Panel 

composition and data quality

– What is the impact of NRFU on panel response rates (AAPOR RR3)?

– Does NRFU improve demographic representation of the panel sample?

– Does NRFU have an impact on the study estimates?
AMERISPEAK RECRUITMENT METHODOLOGY

3

Initial Recruitment • Pre-notification postcard • +5 days, 9 x 12 recruitment packet w/ pre

-

incentive, study brochure, and privacy 

policy.

• +11 days, reminder postcard • +18 days, reminder postcard • Call-ins allowed throughout • +25 days, call-outs to matched telephones

NRFU Recruitment • Federal Express study brochure and 

enhanced pre-incentive

• In-person recruitment
WEIGHTED AAPOR RR3 - FOR RECRUITMENT YEARS WITH NRFU 4

A little more than half of AmeriSpeak panelists 

have been recruited during the NRFU recruitment stage

NRFU boosts 

response rate

by 3 times 

9.9%

28.6%

31.7%

0%

5%

10%

15%

20%

25%

30%

35%

Initial NRFU Overall

Response Rate (AAPOR RR3, Weighted)
PANEL RECRUITMENT METHODOLOGY : OVERSAMPLING INNOVATIONS (2014-PRESENT) 5

Oversample Census Tracts with 

33%+ Hispanic and NH Black and 

12%+ young adult share

Oversample Hispanics, NH 

Blacks, and young adults 

identified by commercial 

auxiliary data

2017 ABS Frame Supplement:

Oversample HHs identified by 

commercial auxiliary data as likely 

Spanish-speaking HHs
Panel Composition Analysis
PANEL COMPOSITION (FOR RECRUITMENT YEARS WITH NRFU) 7

Unweighted Panel Distribution by Panelist Type and Education

0%

10%

20%

30%

40%

50%

Some high school High school grad Some college

or above

BA or above

Initial Recruits

NRFU Recruits

Overall Recruits

ACS benchmark

NRFU brings in lower 

educated panelists

Initial Recruits over 

represent higher education
PANEL COMPOSITION (FOR RECRUITMENT YEARS WITH NRFU) 8

Unweighted Panel Distribution by Panelist Type and Age

0%

5%

10%

15%

20%

25%

18-24 25-34 35-44 45-54 55-64 65+

Initial Recruits

NRFU Recruits

Overall Recruits

ACS benchmark

Initial Recruits are older 

(65+)

NRFU brings in younger 

panelists (18-34)
PANEL COMPOSITION (FOR RECRUITMENT YEARS WITH NRFU) 9

Unweighted Panel Distribution by Panelist Type and Race/Ethnicity

0%

10%

20%

30%

40%

50%

60%

70%

White, NH Black, NH Asian, NH Other, NH Multi, NH Hispanic

Initial Recruits

NRFU Recruits

Overall Recruits

ACS benchmark

NRFU brings in more 

Hispanic Minorities
Study Estimate Analysis
11 STUDY DATA SOURCES : EVALUATING THE IMPACT OF NRFU PANEL RECRUITMENT ON THE MEASUREMENT 

OF ATTITUDES AND OPINIONS 

* University of Michigan “NASA Scientific Literacy Study”

* Johns Hopkins University “Public Support for Gun Violence Prevention Policies 

Among Gun Owners and Non-Gun Owners”

AARP Retirement Study

* General Social Survey 

AmeriSpeak Study

Legal Services Corporation

“’Justice Gap’ Survey”

All analyses are conducted with weighted data reflecting the 

probabilities of selection and post-stratification weighting
SUMMARY OF BIVARIATE ITEM DIFFERENCES BETWEEN NRFU AND INITIAL SAMPLE 12

NRFU Produces Different Estimates for Measurement 

of Attitudes and Opinions 

Items different 

at p = 0.05

Gun Control 10/39 = 25.6%

GSS 5/29 = 17.2%

NASA 4/17 = 23.5%

Overall 19/85 = 22.4%

Statistical significance is determined by a χ2 Test (NRFU vs. Initial)
13

0%

10%

20%

30%

40%

50%

60%

70%

Improving and protecting the nation’s health

Social security

Highways and bridges

Improving and protecting the environment

Solving the problems of the big cities

Supporting scientific research

Initial: Mail/Phone NRFU: Fed-Ex/Face to Face

Initial% - NRFU% > 4pp

Improving the conditions of blacks 

Mass transportation

Source: AmeriSpeak Panel - General Social Survey (Weighted) 

NRFU Recruits Trend Towards Conservative Positions 

on Government Spending

NATIONAL SPENDING PRIORITIES: % REPORTING THE COUNTRY SPENDS “TOO LITTLE”
HOW INTERESTED ARE YOU IN CURRENT NEWS EVENTS? 14

0% 20% 40% 60%

Not at all interested

Moderately interested

Very interested

Source: University of Michigan “NASA Scientific Literacy Study” (Weighted) ** p = 0.003

Mail/Phone recruits 

tend to be more 

interested in current 

events

Mail/Phone

FedEx/F2F

Fewer FedEx/F2F 

recruits ‘very 

interested’ in 

news
SCIENCE AND TECHNOLOGY ARE MAKING OUR LIVES HEALTHIER, EASIER, AND MORE COMFORTABLE 15

Mail/Phone

FedEx/F2F

* p = 0.011

0% 20% 40% 60%

Strongly disagree

Disagree

Agree

Strongly agree

Mail/Phone recruits tend to be 

more pro-science, compared to 

FedEx/F2F recruits

Source: University of Michigan “NASA Scientific Literacy Study” (Weighted) 
SUMMARY – IS NRFU RECRUITMENT MAKING A DIFFERENCE? 16

Panelists recruited via NRFU tend to have different policy 

positions on issues – even after weighting
SUMMARY – IS NRFU RECRUITMENT MAKING A DIFFERENCE? 17

• NRFU respondents are less likely believe the country “spends too little” on: 

– Improving and protecting the nation’s health 

– Solving the problems of the big cities 

– Highways and bridges 

– Improving conditions of African-Americans, and other priorities

• NRFU panelists are less likely to be:

– Very interested in the current news 

– Pro-science 
Questions?
Thank you.