© 2021 by the Regents of the University of Michigan

Nonresponse Bias Analysis Methods: 

A Taxonomy and Summary

James Wagner

University of Michigan – SRC

FCSM Conference

November 4, 2021
© 2021 by the Regents of the University of Michigan

Overview

ï Background

ï Five approaches

– Response rate comparisons

– Subgroup response rate variation

– Comparisons to external estimates

– Changes due to level of effort

– Contrasting alternative adjustment strategies

ï Conclusion

2
© 2021 by the Regents of the University of Michigan

Background

ï Nonresponse bias concerns have grown

ï Nonresponse bias evaluation has been a

burgeoning area of research

– Groves (2006) provides useful taxonomy

ï OMB requires nonresponse bias analysis for

surveys with response rates lower than 80%

ï FCSM Nonresponse Bias Subcommittee

report: “Best Practices for Nonresponse Bias

Reporting”

3
© 2021 by the Regents of the University of Michigan

Background

ï Nonresponse bias analysis is driven by the

available data:

– Sampling frame/auxiliary data

– Paradata

– Survey data

– Administrative data

ï Want data that are “closer to the target”

4
© 2021 by the Regents of the University of Michigan

1. Response Rates

ï Groves and

Peytcheva (2008)

– “NR rate by itself is

a poor predictor

of...NR bias”

ï Brick and

Tourangeau (2017)

reanalysis

5

ï Response rates are a valuable indicator

– Not the only, and maybe not the best
© 2021 by the Regents of the University of Michigan

2. Subgroup Response Rate Variation

ï We can also think of as comparisons ‐‐

responders vs nonresponders

ï Using sampling frame, auxiliary, and paradata

– Ideally, proxy‐Y variables

ï Controlling variation seems helpful

– Assume balanced response is better

– Assume better than simply adjusting

6
© 2021 by the Regents of the University of Michigan

2. Subgroup Response Rate Variation

ï Example indicator: R‐

Indicator (Schouten, et

al., 2009)

– Variation of estimated

response probabilities

– 沈

‐‐> 1 is

perfect balance

– Schouten et al. (2016)

ï Simulation study:

increases in sample

balance are associated

with reductions in bias

7

Coffey, et al., 2019
© 2021 by the Regents of the University of Michigan

2. Sugroup Response Rate Variation

ï Comparison of responders and sample

ï Based on administrative data:

8

Survey variable Sample Respondent

Cumulative GPA 3.18 3.26

Avg. weekly Campus Rec Facility

visits 0.78 1.02***

Avg. PE classes skipped 2.98 2.95

Greek life participant 0.2 0.18*

Residential village participant 0.49 0.54

Standish and Umbach (2019) *p < 0.05, **p < 0.01, ***p < 0.001
© 2021 by the Regents of the University of Michigan

3. Comparisons to External Estimates

ï Poststratification factors

ï Comparison so other surveys

9

Characteristic

Final

calibrated

estimate

Bench‐mark

estimate

Bench‐mark

source

Access to Internet 78.1 70.9*

NHIS

Excellent, very good, or good health 84.9 86.9

Never visited doctor 21.2 19.0*

Looked for health information on the

Internet (Internet users only) 78 57.9*

Health professionals always explain

things in a way you understand 61 61.4

MEPS In past 12 months, health

professionals always spend enough

time with you

44.6 52.4*

HINTS 4 Cycle 1

compared to  

NHIS/MEPS

(abridged from

Maitland, et al.,

2017)

*p<0.05
© 2021 by the Regents of the University of Michigan

4. Variation within survey

ï Comparison of estimates by level‐of‐effort

– Special nonresponse follow‐up studies

10

Substance Early Late

Alcohol

12 months* 77.57 83.24

Chronic risky use* 6.25 8.23

Heavy weekly use 4.69 5.55

Cannabis

Lifetime* 42.94 47.88

12 months* 13.21 16.35

Any illicit drug

Lifetime* 43.66 48.47

12 months* 13.64 16.69

ï Example:

– Early vs Late

responders in

Canadian Addiction

Survey

– CATI → Compleピon

within 1‐6 (Early) vs 7+

(Late) attempts

– Zhao, et al., (2009)

*p<0.05
© 2021 by the Regents of the University of Michigan

4. Variation within survey

ï Which design features reduce the risk of

nonresponse bias?

– Groves and Heeringa (2006): Change design when

current design no longer leads to changes in

estimates – “phase capacity”

– Peytchev et al. (2009)

ï More of the same (e.g. additional call attempts) does

not lead to changes in estimates

ï Changing the protocol in a way that addresses the

mechanism leads to changes in estimates

– Example: Reduced length questionnaire

11
© 2021 by the Regents of the University of Michigan

5. Contrasting post‐survey adjustments

ï “Sensitivity” to nonresponse and

poststratification adjustment model selection

12

ï Little, et al. (2020)

Standardized Measure of

Unadjusted Bias (SMUB)

– Using Pattern‐Mixture

Models to estimate bias

under different

assumptions about

nonrespondents,

including NMAR
© 2021 by the Regents of the University of Michigan

Lessons Learned

ï Choose design features that minimize risk of

nonresponse bias

– Reduce the impact of multiple mechanisms:

ï Topic not interesting, Too little time, etc.

ï Multiple approaches to evaluation is a best

practice

ï Check sensitivity to model assumptions

ï Allow users to evaluate risks relative to their

analyses

13
© 2021 by the Regents of the University of Michigan

Thank You!

ï Email: jameswag@umich.edu

14
© 2021 by the Regents of the University of Michigan

References

ï Brick, J. M. and R. Tourangeau (2017). "Responsive Survey Designs for Reducing Nonresponse Bias." Journal of

Official Statistics 33(3): 735‐752.

ï Coffey, S., B. Reist and P. V. Miller (2020). "Interventions on‐Call: Dynamic Adaptive Design in the 2015 National

Survey of College Graduates." Journal of Survey Statistics and Methodology 8(4): 726‐747.

ï Groves, R. M. (2006). "Nonresponse Rates and Nonresponse Bias in Household Surveys." Public Opinion Quarterly

70(5): 646‐675.

ï Groves, R. M. and S. G. Heeringa (2006). "Responsive Design for Household Surveys: Tools for Actively Controlling

Survey Errors and Costs." Journal of the Royal Statistical Society: Series A (Statistics in Society) 169(3): 439‐457.

ï Groves, R. M. and E. Peytcheva (2008). "The Impact of Nonresponse Rates on Nonresponse Bias: A Meta‐Analysis."

Public Opinion Quarterly 72(2): 167‐189.

ï Little, R.J.A., West, B.T., Boonstra, P., and Hu, J. (2020). Measures of the Degree of Departure from Ignorable

Sample Selection. Journal of Survey Statistics and Methodology, 8, 932‐964.

ï Maitland, A., A. Lin, D. Cantor, M. Jones, R. P. Moser, B. W. Hesse, T. Davis and K. D. Blake (2017). "A Nonresponse

Bias Analysis of the Health Information National Trends Survey (Hints)." Journal of Health Communication 22(7):

545‐553.

ï Peytchev, A., R. K. Baxter and L. R. Carley‐Baxter (2009). "Not All Survey Effort Is Equal: Reduction of Nonresponse

Bias and Nonresponse Error." Public Opinion Quarterly 73(4): 785‐806.

ï Schouten, B., F. Cobben and J. G. Bethlehem (2009). "Indicators for the Representativeness of Survey Response."

Survey Methodology 35(1): 101‐113.

ï Schouten, B., F. Cobben, P. Lundquist and J. Wagner (2016). "Does More Balanced Survey Response Imply Less

Non‐Response Bias?" Journal of the Royal Statistical Society: Series A (Statistics in Society) 179(3): 727‐748.

ï Standish, T. and P. D. Umbach (2019). "Should We Be Concerned About Nonresponse Bias in College Student

Surveys? Evidence of Bias from a Validation Study." Research in Higher Education 60(3): 338‐357.

ï Zhao, J., T. Stockwell and S. MacDonald (2009). "Non–Response Bias in Alcohol and Drug Population Surveys."

Drug and Alcohol Review 28(6): 648‐657.

15