© 2023 Ipsos 1

Megan A. Hendrich & Randall K. Thomas

Ipsos U.S. Public Affairs

Give it Time?: 

Sample Composition by Completion Date

2023 FCSM Research & Policy Conference

October 24 – 26, 2023

Hyattsville, MD
© 2023 Ipsos 2

People who respond earlier to a survey invitation can differ from those 

who respond later (e.g., Fricker & Tourangeau, 2010). 

For example, we often find that older, female, urban, and White 

respondents are more likely to respond earlier in a field period. 

Though younger, male, rural, Black, and Hispanic respondents respond in 

lower proportions early on, these harder-to-reach groups tend to 

increase participation with longer time in field. 

Study Background
© 2023 Ipsos 3

In political polling, many have advocated for leaving a study in field for a 

longer amount of time to try to ensure that hard-to-reach respondents 

might be more likely to complete the survey. This then provides more 

diversity and representativeness to a sample. 

Others have suggested that many people change their minds about 

candidates closer to an election (such as in the 10 days prior to election 

day) and that the bulk of the sample should complete the survey within 

this shorter time frame for better predictions of election outcomes. 

Study Background
© 2023 Ipsos 4

Since address-based sample (ABS) studies typically employ mailing 

invitations, often with mail-back surveys, ABS studies may be more 

difficult to field and complete in a manner in which the data are still 

fresh and predictive of voters’ choices.

In collaboration with the New York Times, we conducted an ABS study 

on the 2022 midterm election in Wisconsin to investigate how early 

responders differ from the overall sample in terms of sample 

composition and vote choice. We were interested in the election 

outcomes and the representativeness and bias of the sample overall and 

at different points during the fielding period.

Study Background
© 2023 Ipsos 5

Method
© 2023 Ipsos 6

We conducted a study of Wisconsin adults (age 18+) using address-based 

sample (ABS). 

Sample was selected randomly from all residential addresses, with an 

oversample of rural addresses (38% rural and 62% urban; benchmarks 

indicated 28% rural and 72% urban). 

Field period: 9/9/2022 to 11/7/2022

Each sample member received 4 mailings:

1. Invitation letter with a mail-back survey & return envelope ($5 prepaid incentive)

2. Reminder postcard 1

3. Reminder invitation letter with a mail-back survey & return envelope

4. Reminder postcard 2

Study Design: Fielding Details
© 2023 Ipsos 7

ABS modes of completion:

• Mail-back paper survey: Enclosed in mailings 1 and 3 with a 

return envelope – 1,497/1,610 completed by mail (93%)

• Online survey: Offered in mailings 3 and 4 (reminder invite and 

reminder postcard 2) – 113/1,610 completed online (7%)

Survey incentive: $20 for responding to the survey

Study Design: Fielding Details
© 2023 Ipsos 8

Study Design: Mail-back Questionnaire
© 2023 Ipsos 9

We randomly divided the sample into 3 replicates for mailings:

Study Design: Fielding Details

Replicate 1 Replicate 2 Replicate 3 Overall

Sample Mailing Size 610 2,745 2,745 6,100

First Mailed 09/09/2022 09/19/2022 09/23/2022 09/09/2022

First Returned Complete 09/12/2022 09/23/2022 09/28/2022 09/12/2022

50% Response Date 09/22/2022 09/30/2022 10/03/2022 10/03/2022

75% Response Date 10/04/2022 10/11/2022 10/14/2022 10/11/2022

100% Response Date 11/01/2022 11/07/2022 11/07/2022 11/07/2022

Total Completes 171 699 740 1,610

Cooperation Rate 28.0% 25.5% 27.0% 26.4%
© 2023 Ipsos 10

We categorized respondents in each replicate as early responders using 

two different criteria: 

• Comparison sample 1: First 50% of each replicate

• Comparison sample 2: First 75% of each replicate

We then compared these “early responder samples” to the overall 

sample to assess how they compared on sample composition and bias.

Each sample was weighted using demographic values for Wisconsin for 

the following: Age by Gender, Race-Ethnicity, Education, Income, and 

Region of State.

Study Design: Analysis
© 2023 Ipsos 11

We examined differences in the sample composition of the early 

responder samples to the overall sample by comparing demographics

and responses to behavioral questions, including vote choice. 

We then assessed bias by calculating the average absolute deviation 

from 15 benchmarks for Wisconsin:

• Calculated the absolute difference between each benchmark and 

the estimate from each sample

• Averaged across the absolute differences for each benchmark to 

obtain the average absolute deviation for each sample

Study Design: Analysis
© 2023 Ipsos 12

Results
© 2023 Ipsos 13

Do demographics used for weighting become more representative with a 

longer field period? First, we looked at the average absolute deviation by 

category among the weighting variables.

Results: Effect on Weighting Demographics

Weighting Variables Extent of Completeness Bias 

50% 75% 100% Reduction?

Age-Gender (8 categories) 5.9% 5.5% 5.0% Yes

Race-Ethnicity (3 categories) 3.1% 2.7% 2.3% Yes

Education (3 categories) 7.0% 7.4% 7.9% No

Income (4 categories) 3.9% 4.0% 4.2% –

Region of State (7 categories) 2.4% 2.4% 2.3% –

Average Absolute Deviation from Unweighted Data 

Across Variable Categories
© 2023 Ipsos 14

A summary measure of demographic imbalance can be obtained by 

looking at the weighting efficiency values of the weights we computed 

for each sample. The full sample had highest weighting efficiency.

Results: Effect on Weighting Efficiency

61%

65% 67%

40%

50%

60%

70%

80%

50% 75% 100%

Weighting Efficiency
© 2023 Ipsos 15

How do the samples compare on 15 weighted estimates 

for Wisconsin benchmarks?

Results: Effect on Bias in Other Variables

Absolute Deviation from 

Benchmark

Benchmark Benchmark

Value

Extent of Completeness Extent of Completeness

50% 75% 100% 50% 75% 100%

Veteran 06.6% 09.1% 08.5% 08.2% 2.5% 1.9% 1.6%

Valid concealed carry license 10.6% 13.6% 13.5% 11.9% 3.0% 2.8% 1.3%

Has a motorcycle 13.2% 12.5% 11.9% 11.4% 0.8% 1.4% 1.8%

Valid hunting license 13.9% 22.5% 22.2% 20.8% 8.6% 8.3% 6.9%

At least 1 child under 18 in household 27.6% 32.4% 32.4% 32.3% 4.8% 4.8% 4.7%

Vote for Governor for 2022 WI – Republican 28.7% 29.4% 30.4% 30.4% 0.7% 1.7% 1.7%

Vote for Senator for 2022 WI – Republican 30.3% 30.7% 31.3% 31.3% 0.4% 1.0% 1.0%

Valid fishing license 30.5% 29.4% 31.1% 30.5% 1.1% 0.6% 0.0%

Valid U.S. passport 42.4% 50.1% 48.7% 49.0% 7.8% 6.3% 6.6%

Works full time 50.1% 56.8% 57.2% 56.4% 6.7% 7.1% 6.3%

Currently married 52.8% 56.1% 56.0% 55.5% 3.3% 3.2% 2.7%

Born in Wisconsin 70.8% 71.0% 72.4% 71.5% 0.2% 1.6% 0.7%

Own their own home 75.2% 77.9% 76.5% 75.3% 2.7% 1.3% 0.1%

Currently registered to vote in Wisconsin 81.7% 82.2% 81.8% 81.0% 0.5% 0.1% 0.7%

Received at least 1 dose of a COVID-19 vaccine 83.7% 76.8% 77.0% 77.6% 6.9% 6.8% 6.1%
© 2023 Ipsos 16

How do the samples compare on 15 weighted estimates 

for Wisconsin benchmarks?

Results: Effect on Bias in Other Variables

Absolute Deviation from 

Benchmark

Benchmark Benchmark

Value

Extent of Completeness Extent of Completeness

50% 75% 100% 50% 75% 100%

Veteran 06.6% 09.1% 08.5% 08.2% 2.5% 1.9% 1.6%

Valid concealed carry license 10.6% 13.6% 13.5% 11.9% 3.0% 2.8% 1.3%

Has a motorcycle 13.2% 12.5% 11.9% 11.4% 0.8% 1.4% 1.8%

Valid hunting license 13.9% 22.5% 22.2% 20.8% 8.6% 8.3% 6.9%

At least 1 child under 18 in household 27.6% 32.4% 32.4% 32.3% 4.8% 4.8% 4.7%

Vote for Governor for 2022 WI – Republican 28.7% 29.4% 30.4% 30.4% 0.7% 1.7% 1.7%

Vote for Senator for 2022 WI – Republican 30.3% 30.7% 31.3% 31.3% 0.4% 1.0% 1.0%

Valid fishing license 30.5% 29.4% 31.1% 30.5% 1.1% 0.6% 0.0%

Valid U.S. passport 42.4% 50.1% 48.7% 49.0% 7.8% 6.3% 6.6%

Works full time 50.1% 56.8% 57.2% 56.4% 6.7% 7.1% 6.3%

Currently married 52.8% 56.1% 56.0% 55.5% 3.3% 3.2% 2.7%

Born in Wisconsin 70.8% 71.0% 72.4% 71.5% 0.2% 1.6% 0.7%

Own their own home 75.2% 77.9% 76.5% 75.3% 2.7% 1.3% 0.1%

Currently registered to vote in Wisconsin 81.7% 82.2% 81.8% 81.0% 0.5% 0.1% 0.7%

Received at least 1 dose of a COVID-19 vaccine 83.7% 76.8% 77.0% 77.6% 6.9% 6.8% 6.1%
© 2023 Ipsos 17

How do the samples compare on 15 weighted estimates 

for Wisconsin benchmarks?

Results: Effect on Bias in Other Variables

Absolute Deviation from 

Benchmark

Benchmark Benchmark

Value

Extent of Completeness Extent of Completeness

50% 75% 100% 50% 75% 100%

Veteran 06.6% 09.1% 08.5% 08.2% 2.5% 1.9% 1.6%

Valid concealed carry license 10.6% 13.6% 13.5% 11.9% 3.0% 2.8% 1.3%

Has a motorcycle 13.2% 12.5% 11.9% 11.4% 0.8% 1.4% 1.8%

Valid hunting license 13.9% 22.5% 22.2% 20.8% 8.6% 8.3% 6.9%

At least 1 child under 18 in household 27.6% 32.4% 32.4% 32.3% 4.8% 4.8% 4.7%

Vote for Governor for 2022 WI – Republican 28.7% 29.4% 30.4% 30.4% 0.7% 1.7% 1.7%

Vote for Senator for 2022 WI – Republican 30.3% 30.7% 31.3% 31.3% 0.4% 1.0% 1.0%

Valid fishing license 30.5% 29.4% 31.1% 30.5% 1.1% 0.6% 0.0%

Valid U.S. passport 42.4% 50.1% 48.7% 49.0% 7.8% 6.3% 6.6%

Works full time 50.1% 56.8% 57.2% 56.4% 6.7% 7.1% 6.3%

Currently married 52.8% 56.1% 56.0% 55.5% 3.3% 3.2% 2.7%

Born in Wisconsin 70.8% 71.0% 72.4% 71.5% 0.2% 1.6% 0.7%

Own their own home 75.2% 77.9% 76.5% 75.3% 2.7% 1.3% 0.1%

Currently registered to vote in Wisconsin 81.7% 82.2% 81.8% 81.0% 0.5% 0.1% 0.7%

Received at least 1 dose of a COVID-19 vaccine 83.7% 76.8% 77.0% 77.6% 6.9% 6.8% 6.1%
© 2023 Ipsos 18

How do the samples compare on 15 weighted estimates 

for Wisconsin benchmarks?

Results: Effect on Bias in Other Variables

Absolute Deviation from 

Benchmark

Benchmark Benchmark

Value

Extent of Completeness Extent of Completeness

50% 75% 100% 50% 75% 100%

Veteran 06.6% 09.1% 08.5% 08.2% 2.5% 1.9% 1.6%

Valid concealed carry license 10.6% 13.6% 13.5% 11.9% 3.0% 2.8% 1.3%

Has a motorcycle 13.2% 12.5% 11.9% 11.4% 0.8% 1.4% 1.8%

Valid hunting license 13.9% 22.5% 22.2% 20.8% 8.6% 8.3% 6.9%

At least 1 child under 18 in household 27.6% 32.4% 32.4% 32.3% 4.8% 4.8% 4.7%

Vote for Governor for 2022 WI – Republican 28.7% 29.4% 30.4% 30.4% 0.7% 1.7% 1.7%

Vote for Senator for 2022 WI – Republican 30.3% 30.7% 31.3% 31.3% 0.4% 1.0% 1.0%

Valid fishing license 30.5% 29.4% 31.1% 30.5% 1.1% 0.6% 0.0%

Valid U.S. passport 42.4% 50.1% 48.7% 49.0% 7.8% 6.3% 6.6%

Works full time 50.1% 56.8% 57.2% 56.4% 6.7% 7.1% 6.3%

Currently married 52.8% 56.1% 56.0% 55.5% 3.3% 3.2% 2.7%

Born in Wisconsin 70.8% 71.0% 72.4% 71.5% 0.2% 1.6% 0.7%

Own their own home 75.2% 77.9% 76.5% 75.3% 2.7% 1.3% 0.1%

Currently registered to vote in Wisconsin 81.7% 82.2% 81.8% 81.0% 0.5% 0.1% 0.7%

Received at least 1 dose of a COVID-19 vaccine 83.7% 76.8% 77.0% 77.6% 6.9% 6.8% 6.1%
© 2023 Ipsos 19

How do the samples compare on 15 weighted estimates 

for Wisconsin benchmarks?

Results: Effect on Bias in Other Variables

Absolute Deviation from 

Benchmark

Benchmark Benchmark

Value

Extent of Completeness Extent of Completeness

50% 75% 100% 50% 75% 100%

Veteran 06.6% 09.1% 08.5% 08.2% 2.5% 1.9% 1.6%

Valid concealed carry license 10.6% 13.6% 13.5% 11.9% 3.0% 2.8% 1.3%

Has a motorcycle 13.2% 12.5% 11.9% 11.4% 0.8% 1.4% 1.8%

Valid hunting license 13.9% 22.5% 22.2% 20.8% 8.6% 8.3% 6.9%

At least 1 child under 18 in household 27.6% 32.4% 32.4% 32.3% 4.8% 4.8% 4.7%

Vote for Governor for 2022 WI – Republican 28.7% 29.4% 30.4% 30.4% 0.7% 1.7% 1.7%

Vote for Senator for 2022 WI – Republican 30.3% 30.7% 31.3% 31.3% 0.4% 1.0% 1.0%

Valid fishing license 30.5% 29.4% 31.1% 30.5% 1.1% 0.6% 0.0%

Valid U.S. passport 42.4% 50.1% 48.7% 49.0% 7.8% 6.3% 6.6%

Works full time 50.1% 56.8% 57.2% 56.4% 6.7% 7.1% 6.3%

Currently married 52.8% 56.1% 56.0% 55.5% 3.3% 3.2% 2.7%

Born in Wisconsin 70.8% 71.0% 72.4% 71.5% 0.2% 1.6% 0.7%

Own their own home 75.2% 77.9% 76.5% 75.3% 2.7% 1.3% 0.1%

Currently registered to vote in Wisconsin 81.7% 82.2% 81.8% 81.0% 0.5% 0.1% 0.7%

Received at least 1 dose of a COVID-19 vaccine 83.7% 76.8% 77.0% 77.6% 6.9% 6.8% 6.1%
© 2023 Ipsos 20

Average bias somewhat declines as the proportion of sample that 

completes the survey increases, both for variables needed for weighting 

and other variables, reflecting general bias.

Results: Effect on Bias Overall

4.5% 4.4% 4.3%

3.3% 3.3% 2.8%

0.0%

2.0%

4.0%

6.0%

50% 75% 100%

Bias for Variable Types

Demographic Bias Other Variable Bias
© 2023 Ipsos 21

Discussion
© 2023 Ipsos 22

Discussion and Conclusions

We examined how time in field for an ABS study could affect the extent 

of bias in the sample. Our study lasted for a total of about 8 weeks in field. 

Shorter field periods could reduce the accuracy of the results. Limiting 

time in field could increase bias for both weighting variables (affecting 

weighting efficiency and effective sample size) and for other substantive 

variables of interest. However, a shorter field period could work given 

that the reductions in bias were relatively small.

Based on these results, we would recommend longer field periods, up to 

some limit determined by the last mailing (typically, we found almost 

zero completes 8 to 10 days after the last mailing).
Megan A. Hendrich

Megan.Hendrich@ipsos.com

Thank you!

Ipsos U.S. Public Affairs is a 

Charter Member of

AAPOR’s Transparency Initiative