RTI International is a registered trademark and a trade name of Research Triangle Institute. www.rti.org

Finding the Right Auxiliary Information for Nonresponse 

Adjustment Models

In Search of Zs with Desirable Properties

Andy Peytchev, RTI International
Outline

▪ What we are (usually) doing wrong 

▪ What we should aim to be doing 

▪ What we can be doing
1. Nonresponse adjustment models are generally bad
Lack of Auxiliary Information in Household Surveys

▪ Few studies report model fit for nonresponse weight adjustment

models 

– Generally poor model fit in general population surveys, relying on aggregate 

information (e.g., Census Block Group level) 

– Exceptions to this can be undesirable. Variables endogenous to

nonresponse are strong “predictors” of nonresponse and can do more harm 

than good (more on this in a moment) 

▪ Effort to use advanced statistical methods (e.g., machine learning 

algorithms) to identify complex interactions are limited by the lack of 

auxiliary information with desirable properties
2. Focus on explaining nonresponse is misplaced
Common Nonresponse Adjustments Model Participation

▪ Strongest predictors of nonresponse generally do not help with bias 

reduction 

– Refusal on a prior call/contact attempt, number of calls/contact attempts 

(Wagner, Valliant, Hubbard, and Jiang, 2014) 

▪ Associations with nonresponse but not with survey variables of 

interest unduly increase the variance estimates (Little and Vartivarian, 

2005)

Association with outcome

Association with 

nonresponse

Low High

Low Bias: --- 

Var: ---

Bias: --- 

Var: ↓

High Bias: --- 

Var: ↑

Bias: ↓

Var: ↓
3. Commonly used auxiliary information lacks desirable properties
Desirable Properties

▪ The ideal “Z” variable: an indicator for a common cause of both

likelihood to respond and the survey outcome of interest

Note: Partial figure. 

Source: Groves, R. M. (2006). Nonresponse rates and nonresponse bias in household surveys. Public Opinion Quarterly, 

70(5), 646-675. 
Challenge in Practice

▪ The objective is seldom met in practice. Auxiliary variables tend to be 

associated with EITHER nonresponse OR survey variables

No variables in 

this area

Source: Kreuter, F., Olson, K., Wagner, J., Yan, T., Ezzati-Rice, T. M., Casas-Cordero, C., . . . Raghunathan, T. E. (2010).

Using Proxy Measures and Other Correlates of Survey Outcomes to Adjust for Nonresponse: Examples from Multiple 

Surveys. Journal of the Royal Statistical Society, Series A (Statistics in Society), 173(2).
4. Promising avenues for improvement
Potential for Improving Nonresponse Adjustments

▪ Designed paradata 

– Interviewer observations 

– Proxy reports 

▪ Embed survey content for weight calibration 

– Consider calibration beyond traditional demographic characteristics 

▪ Administrative data 

– E.g., student information 

▪ Statistical methods 

– E.g., tree-based methods (limited utility depending on auxiliary data) 

– Multiple imputation for unit nonresponse, particularly for swiss-cheese 

pattern of missing auxiliary data and improved efficiency
Designed Paradata

▪ National Survey of Family Growth’s interviewer observations of 

sexual activity (used in nonresponse weighting adjustments)

Source: West, B. T., & Kreuter, F. (2014). A Practical Technique for Improving the Accuracy of Interviewer Observations of 

Respondent Characteristics. Field Methods.
Designed Paradata

▪ Interviewer variance in interviewer observations

Source: West, B. T., & Kreuter, F. (2013). Factors Affecting the Accuracy of Interviewer Observations: Evidence from the 

National Survey of Family Growth. Public Opinion Quarterly, 77(2), 522-548.
Designed Paradata—Another Cautionary Tale

▪ 2015 California Health Interview Survey 

– Two-stage design with screener and main interview 

– Ask about health conditions for each selected household member (often a 

proxy report) 

▪ Found substantial measurement error in the screener reports 

▪ Measurement error was correlated with nonresponse 

▪ Underreporting was correlated with nonresponse

Source: Peytcheva, E., Peytchev, A., Currivan, D. & Jans, M. (2017). Measurement error in proxy measures of key survey 

variables to estimate, reduce, and adjust for nonresponse bias. Paper presented at the ESRA annual conference, Lisbon. 
Combining Statistical Properties with Social Science

▪ What behaviors are related to processes generating nonresponse 

decisions, and are likely associated with many survey variables? 

– Highly correlated with nonresponse and substantive survey variables 

▪ Two example studies using General Social Survey data (Peytchev, 

Presser, and Zhang, 2018) 

– Voting 

– Volunteering 

▪ Calibrate survey weights using benchmark estimates from a source 

that is not subjected to high nonresponse