A Test of a “Federal” Approach to Increase 

Survey Response and Fit-for-purpose of

Probability Panels for Federal Data Collection

2023 FCSM Res earch & Policy Conference

Session A-1: Leveraging Probability Panels for Federally Sponsored 

Statistical Data Collections 

David Dutwin

Ipek Bilgen

J . Michael Dennis
AMERISP EAK FEDERAL 2

Why an AmeriSpeak Federal?

AmeriSpeak Federal is a new NORC panel with features designed to better fit 

expectations within Federal clients with regard to response rate and data 

quality. Its main differentiators to AmeriSpeak “main” are:

 A modified sampling geared more toward NRFU, and

 Greater effort in fielding

 BOTH are meant to secure higher response rates

AmeriSpeak AmeriSpeak 

Federal Panel

AAPOR Recr Hous ehold Recruitment Rate 22% 32%

Estimated Cooperation Rate 25% 32%

Estimated AAPOR Response Rate 5.5% 10.2%

Estimated Retention Rate 80% 80%

Estimated NORC AAPOR Response Rate 4.4% 8.2%
AMERISP EAK FEDERAL 3

Methodological and Operational Contrasts

Standard Federal

Main Recruiting 4 mailings, phone 4 mailings, phone

NRFU Fed Ex, in person Fed Ex, in person

NRFU Sampling Rate 20% 40%

Standard Study Sample Full panel 6-month Active Panelists

Study Incentive Standard Double

Study Contacts Email for CAWI Email for CAWI

Phone for CATI Phone for CATI (2 add'l call attempts)

Prenote PC for lowest 50% CR

Nonresponse PCs

Data Collection Field 2 weeks 4 weeks
AMERISP EAK FEDERAL 4

Sample makeup and cost

Some modest differences in panelists:

17.6%

14.9%

23.4%

16.9%

0.0%

10.0%

20.0%

30.0%

His panic Black

Race/ Ethnicity

AmeriSpeak AmeriSpeak Federal

23.1% 22.6%

26.8%

18.2%

0.0%

10.0%

20.0%

30.0%

18-34 65 and older

Age

AmeriSpeak AmeriSpeak Federal

Cost: Approximately 26% more than a comparable AmeriSpeak quote
INTRODUCTION 5

Res earch Ques tions

• In practice, what is the relative difference in res pons e rates ?

• In practice, what is the relative difference in s tatis tical power

– Given a) larger NRFU sampling fraction and b) demographics closer to benchmarks, Federal should 

have a lower design effect and therefore a lower margin of error

• Are there s ignificant differences in interviewee demographics (unweighted)

• Are there s ignificant differences in weighted point es timates ?

Data and Method

• AmeriSpeak Omnibus tes t with two independent s amples (s tandard and Federal)

– Field in September 2023, N = 1,518

– Each condition weighted identically and to the same sum of weights (1,000)
Results
BIG DATA SAMP LING FOR DIVERSITY 7

Response Rates and Design Effect 

Federal attains response rates over double AmeriSpeak 

standard; 25% of completes come in “overtime”

• Recruitment rate is 60% higher

• Cooperation rate is 30% higher in full sample (notably, despite 

a higher proportion of less active panelists)

• Design effect is 15% lower

Standard Federal

Recr 21.1% 33.8%

Comr 18.3% 23.8%

Cumr 3.9% 8.0%

Deff 1.85 1.60 

MoE (n=1,000) 4.2% 3.9%

Standard Federal

Recr 21.1% 33.8%

Comr 49.6% 72.9%

Cumr/NORC RR 10.5% 24.6%

Full Sample Active Sample

Standard Federal

Main Field 538 731

Extra 2 Weeks 0 249

Total 538 980
BIG DATA SAMP LING FOR DIVERSITY 8

Demographic Differences

In a Word: Small

– AS Federal slightly more likely to acquire 

“hard to reach” respondents

• Non-white

• Young

• Male

• Renters

– Differences are minor

17%

39%

56%

28%

33%

24%

61%

30%

29%

18%

37%

50%

32%

38%

25%

65%

31%

32%

0% 10% 20% 30% 40% 50% 60% 70%

Speak Other Lang Home

Urban

Female

18-34

Non-White

HS or Less

Employed

$30k or Less

Rent

Unweighted Demographics

Federal Standard
BIG DATA SAMP LING FOR DIVERSITY 9

Additional Demographic Differences

Some political and religious 

differences

– AS Federal leans more right

• More republican

• More conservative

– Religious differences

• More Catholic

• Less “Nones ”

32%

28%

17%

34%

16%

37%

33%

20%

27%

14%

0% 10% 20% 30% 40%

Republican

Conservative

Catholic

No Religion

Fair/Poor Health

Weighted Demographics

Federal Standard
BIG DATA SAMP LING FOR DIVERSITY 10

Substantive Differences

Small and generally infrequent

– 141 variables dichotomous variables 

tested (t-tests)

• Vaccine behavior and attitudes

• Travel/Int’l relations

• Reading behavior

• Car purchase intent

– 28 significant differences (19%)

• P < .001: 4

• P = .001 thru .01: 9

• P = .01 thru .05: 15

7%

15%

64%

37%

2%

1%

10%

7%

8%

11%

34%

21%

58%

1%

28%

4%

18%

60%

45%

4%

2%

14%

4%

5%

7%

40%

16%

64%

2%

34%

0% 10% 20% 30% 40% 50% 60% 70%

Did not get flu vaccine: Already got the flu

Relationship bet Mexico and US friendly

Relationship bet Canada and US friendly

Impt for US to advance womens rights in Afghanistan

No Covid booster: Don’t know was eligible

Relationship bet Afghanistan and US friendly

Buy car next 12 months

Not getting flu vaccine afraid of shots

Get Covid vax info from social media

Relationship bet Iran and US friendly

Ever interacted with Mexicans

Not getting flu vaccine not concerned about the flu

US should work to reduce inequality in Mexico

Not getting flu vaccine allergic

Havent gotten flu have not gotten around to it

Substantive Differences: Top 15

Standard Federal
Thank you.