www.rti.org RTI International is a trade name of Research Triangle Institute. RTI and the RTI logo are U.S. registered trademarks of Research Triangle Institute.

Nonresponse Bias Mitigation Strategies

Andy Peytchev

FCSM Conference

November, 2021
Focus

2

o Nonresponse bias, not response 

rates

o Emphasis on selected findings 

and more recent developments

1. Study design features

- Level of effort

- Mode

- Incentives

2. Strategies during data collection

- Responsive Design

- Adaptive Survey Design

3. Postsurvey adjustments

- Modeling approach

- Auxiliary information
Study Design Features: Level of effort

o Curtin, Presser, and Singer (2000)

• 1-2 vs. 3+; 1-5 vs. 6+

o Numerous studies following Curtin, Presser, and Singer (2000)

• In general, little if any relationship between effort and survey estimates

• Keeter et al. (2000), with an experimental design, an exception

o Lin and Schaeffer (1995)

• Nonresponse bias could even be in the opposite direction (we return to this 

under adjustment)

o Yet, level of effort measured as contact attempts has risen over time (e.g., 

Curtin, Presser, and Singer, 2000; 2005)

3
Survey Design Features: Mode of Data Collection

o A very expansive topic, but two key points with regard to nonresponse 

bias:

• Modes vary in resulting nonresponse bias (e.g., Rothbaum and Bee, 2021, CPS 

ASEC)

• Modes that reduce nonresponse bias may induce greater measurement bias for 

some estimates (e.g., Sakshaug, Yan, and Tourangeau, 2010)

• Designs with multiple modes rarely include random assignment to allow 

estimation of mode-specific errors

4
Survey Design Features: Incentives

o Leverage-salience theory and related experiments to produce 

nonresponse (Groves, Singer, and Corning, 2000; Groves, Presser, and 

Dipko, 2004; Groves et al., 2006)

• Those who are less interested or involved in the topic are less likely to participate

• Monetary incentives are particularly effective for these groups of individuals

5
Survey Design Features: Level of Effort and Incentives

6

30%

31%

32%

33%

34%

35%

36%

37%

38%

39%

40%

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31

Stalking

Number of Calls

National Intimate Partner and Sexual Violence Survey Pilot

All Calls

All calls with NRFU

Source: Peytchev, Baxter, and Carley-Baxter, 2009
Strategies During Data Collection

o Heterogeneity

• Leverage-salience theory (Groves, Singer, and Corning, 2000)

o Responsive Design (Groves and Heeringa, 2006)

• Multiple phases and multiple protocols

• Error-sensitive indicators play a central role

o Adaptive Survey Design (Wagner, 2008; Schouten, Peytchev, and 

Wagner, 2017)

• Tailoring or protocols at the individual level

• Often involve models to assign protocols to sample members during data 

collection

7
Strategies During Data Collection: Dutch Labor Force Survey

8

Strata

o Registered unemployed

o 65+ households without employment

o Young household members without 

employment

o Non-western without employment

o Western without employment

o Young household member and 

employed

o Non-western and employed

o Western and employed

o Large households

Modes

o Web

o Telephone

o Face to Face

o Only mode vs. follow-up mode

o Standard vs. extended effort (contact 

attempts)

Source: Calinescu and Schouten, 2015
Strategies During Data Collection: Dutch Labor Force Survey

9

“W”, “T” and “F” refer to Web, telephone and face-to-face; “S” and “E” refer to standard and extended effort.

The strata are based on age, size of the household, number of registered unemployment and ethnicity.

Stratum

1 2 3 4 5 6 7 8 9

W 23.2% 23.6% 15.5% 10.8% 27.9% 27.7% 17.5% 36.7% 22.4%

TS 12.2% 31.4% 8.5% 4.7% 19.7% 13.3% 7.2% 18.1% 21.2%

TE 20.8% 41.3% 15.2% 8.6% 31.1% 23.8% 14.3% 33.3% 37.5%

F 43.5% 53.5% 42.2% 34.1% 45.1% 45.3% 35.9% 46.7% 54.6%

FE 52.4% 58.3% 51.0% 41.2% 51.2% 54.9% 46.0% 56.8% 61.4%

W→TS 28.3% 41.0% 20.2% 13.9% 36.3% 34.0% 20.8% 44.5% 23.1%

W→TE 32.8% 48.4% 23.8% 17.5% 42.1% 41.1% 25.8% 52.1% 24.4%

W→FS 46.3% 57.7% 38.6% 32.7% 50.0% 51.0% 39.3% 58.9% 50.0%

W→FE 49.8% 58.3% 43.4% 36.6% 52.6% 54.7% 44.3% 62.0% 54.2%

Estimated response propensities by mode and stratum

Source: Calinescu and Schouten, 2015
Strategies During Data Collection: High School Longitudinal Study 

of 2009 (HSLS:09) 2013 Update

o Concept of Bias Propensity (Peytchev, Pratt, and Duprey, 2020)

• Exclude paradata that are strong predictors only of nonresponse

• Include demographic and substantive variables of interest

• Allows identification of nonrespondents who contribute to nonresponse bias

o Intervene on underrepresented sample members based on the bias 

propensity

• Prepaid $5 incentive

o Evaluate nonresponse bias reduction

• Frame and prior round data

• Additional phases of data collection

10
Strategies During Data Collection: HSLS:09 2013 Update

3.1 3.0

2.3

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

End of

Phase 1

End of

Phase 2,

Control

End of

Phase 2,

Treatment

Average Absolute Bias (in 

Percentage Points)

Variables Available Only in the Survey 

Compared to Estimate from

Additional Follow-up Phases

11

Source: Peytchev, Pratt, and Duprey, 2020
Postsurvey Adjustments

o Relatively small gains from alternative statistical approaches and 

estimation methods

• Machine learning, e.g., tree-based methods (for reducing nonresponse bias)

- Some can lead to increase in variance without a commensurate reduction in bias

• Replication-based variance estimation methods (for reducing variance estimates)

• It would have to be a very rudimentary statistical method to find a substantial 

improvement

o Substantial gains are possible from additional auxiliary information

• A shift in approach from demographic characteristics to substantive variables

- Designed paradata (Groves and Heeringa, 2006)

- Data from other surveys, at lesser risk of nonresponse bias

- Administrative data

12
Postsurvey Adjustments: Observations in Several Studies

Kreuter et al. (2010). Using proxy measures and other correlates of survey outcomes to adjust for non-response: examples from 

multiple surveys. JRSS-A.
Postsurvey Adjustments

o Auxiliary variables informed by social science

• Civic duty

- Political participation is strongly related to survey participation

• Altruism

- Charitable activities are strongly related to survey participation

14
Postsurvey Adjustments: General Social Survey

15

90%

72%

57%

41%

2%

84%

64%

53%

46%

1%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Obama McCain Other

Eligible to Vote Voted, among

Eligible

Candidate Choice

Weighted Percent

GSS Weighted Estimate,

Including Also

Demographic

Characteristics

Benchmark Population

Estimate

Source: Peytchev, Presser, and Zhang, 2018
Postsurvey Adjustments: General Social Survey

16

Source: Peytchev, Presser, and Zhang, 2018

48%

26%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Volunteered

Weighted Percent

GSS Weighted Estimate, Including Also

Demographic Characteristics

Benchmark Population Estimate
Postsurvey Adjustments: GSS—Voting and Preference

17

-5% -4% -3% -2% -1% 0% 1% 2% 3% 4% 5%

Fair or poor health

Very or pretty happy

Life exciting

Most people try to be helpful

Most try to take advantage

Most people can be trusted

Family income below average

Read newspaper every day *

No religion

Donated blood in past year

Donated to charity in past year *

Support birth control to teens 14 to 16 years old

Same-sex female couple can bring up child just as well *

Oppose capital punishment

Courts are too harsh with criminals

Relative Difference in Weighted Estimates

Addition of Voting Eligibility Addition of Voting Eligibility and Voting

Addition of Voting Eligibility, Voting, and Candidate Choice

Source: Peytchev, Presser, and Zhang, 2018
Postsurvey Adjustments: GSS—Voting and Preference

18

Source: Peytchev, Presser, and Zhang, 2018

-15% -10% -5% 0% 5% 10% 15%

Fair or poor health *

Very or pretty happy

Life exciting *

Most people try to be helpful

Most try to take advantage *

Most people can be trusted

Family income below average

Read newspaper every day

No religion *

Donated blood in past year

Donated to charity in past year *

Support birth control to teens 14 to 16 years old

Same-sex female couple can bring up child just as well

Oppose capital punishment

Courts are too harsh with criminals

Relative Difference in Weighted Estimates
Summary Remarks

o NRB is a counterfactual that is difficult to study

• Contributes to a focus on response rates

o We usually know what design features can reduce NRB

• But it often involves tradeoff with other sources of survey error

o We can have much smarter data collection

• Added effort that needs to be planned

• Requires statistical expertise

o Postsurvey adjustments

• Over-emphasis of statical methods

• Relying on demographic characteristics alone may be insufficient

• Needed variables related to both nonresponse and survey variables of interest

19
Andy Peytchev

apeytchev@rti.org

20