Strategies to Recruit and Represent Low 

Incidence Population Groups and Engender

Sample Diversity in a Probability Panel

2022 FCSM Research & Policy Conference

Session B-6: Leveraging Probability Panels for Federally Sponsored 

Statistical Data Collections 

David Dutwin, SVP, NORC

Ipek Bilgen, Sr. Research Methodologist, NORC
INTRODUCTION 2

Federal surveys have a “higher calling” which includes deep 

representativeness

• Particularly today, a focus on diversity in surveys means sample and consequently 

respondent diversity

• Particularly today, a focus on equity in surveys means:

– Ensuring that all persons are equally able to participate

– Ensuring that our survey designs encourage as many people, equally and equitably as possible, to, in 

fact, participate

• This study serves as a case study in this regard: how the AmeriSpeak panel has 

made substantial efforts toward sample and panelist diversity and equity

– What have we done with regard to sampling?

– Special efforts to increase Spanish speakers, AANHPI

– Impact of a new “equitable” design of recruiting materials.

– Efforts to engage respondents we fear may disengage
Big Data to find hard to recruit 

populations
BIG DATA SAMPLING FOR DIVERSITY 4

Big Data Classifying

Predicting ahead of time whether a household has a person who is “x” (young, poor, 

Spanish speaking, smokes, etc.)

Potential Utility: 

– To develop indictors far more effective than sample appends/vendor flags and/or geographic targeting.

– Significantly higher incidence = lower survey costs. 

– Significantly higher coverage = lower weighting variances.

Anything can be modelled so long as there is an available training dataset with self-reporting on the 

variable of interest, address, and some indicator of respondent (name, or demo combinations like 

gender + age).

Models will vary in effectiveness: many explain more than half of indictor variance, but others have 

not been shown to be effective.
BIG DATA SAMPLING FOR DIVERSITY 5

What Big Data?

There are a range of Big Data available, and at differing levels of measurement:

• Governmental data including Census Planning Database, the ACS, data from FEMA, the EPA, etc.

• Voter Registration data

• Consumer data:

• Financial data:

• Investment activities

• Credit statistics

• Personal finances and spending

• Lifestyle indicators (hobbies)

• Automobile purchases

• Channel indicators (media use, subscriptions, etc)

• Housing/Real estate data

• Demographics

• Travel

• Sports use/interest

• Music interests

• Retail transactional data

• Health indicators

• Social media use
BIG DATA SAMPLING FOR DIVERSITY 6

Process

• Acquire and Match Big Data 

• Merge onto a large, high-quality survey 

• Apply Data Science to Train a model

• Test the model

• Apply (Score) to a survey project

Acquire

Merge

Train

Test

Apply
BIG DATA SAMPLING FOR DIVERSITY 7

Impact of Big Data Classifier Sample Stratification on Recruiting

37%

6%

23%

9%

64%

16%

19%

4%

47%

30%

3%

20%

6%

40%

7% 6%

1%

21%

0%

20%

40%

60%

80%

Child $200k HSLess 18-24 Under 50 AA Hisp Span Male

2021 Results w/ Big Data Sampling vs. 2020 Recruits

2021 2020
Increasing Spanish Language
BIG DATA SAMPLING FOR DIVERSITY 9

The Challenge of Spanish Language

In short, you can target such households with Big Data classifiers, but they still 

may not choose to join as a Spanish speaker:

• Multiple studies find that Spanish speakers participate less, and when they do, they often 

strive to complete in English (for ex: CHIS 2019).

• Added 2,000 addresses identified as having the highest likelihood of containing Spanish 

speaking residents. 

• Specifically in CA (LA, Fresno, San Diego, Modesto areas), TX (Houston, Dallas), FL (Miami, 

Orlando), NY (New York), and IL (Chicago) (approx. 60% of predicted Spanish speakers 

nationwide)

• Sent a 2-day FedEx packet followed by in-person visits in the fall of 2021. Hired 20 additional 

Bilingual field interviewers (~50 in total)

• An average of 2.5 visits were made to each household
TARGETED RECRUITMENT MATERIALS 10

Spanish Dominant FedEx Packet Brochure:
TARGETED RECRUITMENT MATERIALS 11

Bilingual/Spanish Leave Behinds: Door hangars, Sorry I Missed You cards, & other
SPANISH RECRUITING 12

Case Status N %

TOTAL 2,000 100%

Complete 432 22%

Hispanic 1,801 90%

Spanish: Fluent Read/Speak 1,702 84%

Spanish survey takers 980 49%

Results of Spanish Language Recruiting Sample 2021

22%

49%

0%

10%

20%

30%

40%

50%

60%

Mail In-Person

Percent Spanish Recruit w/in 

Spanish Big Data Classifer by 

Modality of Recruiting
Increasing AANHPI
BIG DATA SAMPLING FOR DIVERSITY 14

Turning AANHPI from “Other” to a Robustly Represented Group

• AmeriSpeak working with AARP and other major funders to build an AANHPI Panel that would 

supplement AmeriSpeak but also be able to support its own standalone surveys of the 

AANHPI population

• Aiming to support 1,000 panel interviews in early 2023 and 2,000 by end of 2023; 3,000 end 

of 2024.

• Pilot in 2022 resulted in 341 screened AANHPI interviews of which 292 agreed to future 

survey participation/empanelment

• Sample design leverages Big Data Classifiers + medium and high incidence CGBs.

• Multiple major funders already committed; continuing to seek additional funders to realize 

the full build
BIG DATA SAMPLING FOR DIVERSITY 15

Largely representative of the AANHPI population:

75.7%

24.3%

65.0%

34.0%

0.0%

20.0%

40.0%

60.0%

80.0%

Yes No

Speak Non-English 

Language At Home

ACS

S1 Realloc

21.2%

23.7%

16.6%

4.4%

8.0%

9.8%

11.7%

0.9%

0.6%

0.2%

1.1%

1.8%

18.3%

24.4%

13.3%

11.9%

7.1%

7.3%

4.0%

2.7%

0.6%

0.9%

0.3%

9.2%

0.0% 10.0% 20.0% 30.0%

Asian Indian

Chinese

Filipino

Japanese

Korean

Vietnamese

Other Asian

Native Hawaiian

Samoan

Fijian

Other Pacific Islander

Multi-AANHPI

Heritage

ACS

S1 Reallocated

32%

29%

22%

16%

25%

29% 29%

17%

0%

10%

20%

30%

40%

18 - 34 35 - 49 50 - 64 65 Plus

Age

ACS

S1 Realloc
Creating Equitable Materials
BIG DATA SAMPLING FOR DIVERSITY 17

AmeriSpeak recognizes the need for equitable designs

• Most Americans don’t read or speak like survey researchers do!

• Surveys historically fail getting at low education, young, and non-white respondents

• More currently, some concern with strong Republicans
BIG DATA SAMPLING FOR DIVERSITY 18

Results: Improvements in race/ethnicity and age

73.8%

67.1%

60.0%

65.0%

70.0%

75.0%

80.0%

NH White

Race Ethnicity

EN Control EN EQ

24.2%

58.2%

17.7%

32.9%

51.6%

15.6%

0.0%

20.0%

40.0%

60.0%

80.0%

18-34 35-64 65ov

Yield by Age Category

EN Control EN EQ

34.2%

65.8%

35.3%

64.7%

0.0%

20.0%

40.0%

60.0%

80.0%

HS or Less Some College

Yield by Education Category

Bilingual Spanish

30.6%

39.8%

27.7% 30.4%

39.6%

27.8%

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

Democrat Independent Republican

Yield by Ideological Category

Bilingual Spanish
Retainment is Just as Important and 

Recruitment
BIG DATA SAMPLING FOR DIVERSITY 20

AmeriSpeak has made significant efforts to increase retainment 

and participation over time

• 1st 6 months are critical: 50% of attrition occurs at this time.

• Therefore, testing 1st 6th month mail and e-mail retainment communications

• Specific campaign to engage Hispanics, who attrit the most in the 1st 6th months

• Additional mailings

• Modelling future participation (see Stas!)
Questions?
Thank you.