Best Practices for Nonresponse Bias Reporting

Prepared by the Nonresponse Bias Subcommittee of the

Federal Committee on Statistical Methodology

June 2023

Recommended citation: Madans JH, Earp MS, Blumberg SJ, Christopher EM, Thompson KJ, Sivinski RG, Frenk 

SM, Fakhouri THI, Piscopo KD. 2023. Best Practices for Nonresponse Bias Reporting, FCSM-23-01

Nonresponse Bias Subcommittee, Federal Committee on Statistical Methodology. June 2023.
Federal Committee on Statistical Methodology

Members

Ann Carson

Bureau of Justice Statistics

Chris Chapman

National Center for Education Statistics

Jennifer Childs

U.S. Census Bureau

Tala Fakhouri

U.S. Food and Drug Administration

John Finamore

National Center for Science and Engineering 

Statistics

Ellen Galantucci

Federal Maritime Commission

Michael Hawes

U.S. Census Bureau

Travis Hoppe

National Center for Health Statistics

Wendy Martinez

Bureau of Labor Statistics

Lisa Mirel

National Center for Health Statistics

Jennifer Nielsen

National Center for Education Statistics

Jennifer M. Ortman

U.S. Census Bureau

Anne Parker

Internal Revenue Service

Jennifer Parker

National Center for Health Statistics

Mark Prell (Co-Chair)

Economic Research Service

Michael Ratcliffe

U.S. Census Bureau

Rolf Schmitt

Bureau of Transportation Statistics

Darius Singpurwalla

National Center for Science and Engineering

Statistics

Bob Sivinski (Chair)

Office of Management and Budget

Matt Spence

U.S. Census Bureau

Ed Strocko

Bureau of Transportation Statistics

G. David Williamson

Centers for Disease Control and Prevention

Linda Young

National Agriculture Statistic Services

Lori Zehr (Secretary)

U.S. Census Bureau
Nonresponse Bias Subcommittee

Members

Stephen Blumberg (Co-Chair)

National Center for Health Statistics

Elise Christopher

National Center for Education Statistics

Morgan Earp (Co-Chair)

National Center for Health Statistics

Tala Fakhouri

U.S. Food and Drug Administration

Steven Frenk

National Institutes of Health

Jennifer Madans (Co-Chair)

National Center for Health Statistics

Kathy Downey Piscopo

Substance Abuse and Mental Health Services 

Administration

Bob Sivinski

Office of Management and Budget

Jenny Thompson

U.S. Census Bureau
Best Practices for Nonresponse Bias Reporting June 2023 1

Table of Contents

OBJECTIVES ....................................................................................................................................... 3

OVERVIEW ......................................................................................................................................... 5

GUIDANCE FOR DEVELOPING A NONRESPONSE BIAS REPORT.....................................10

Best Practice 1: Describe the survey that is the subject of the nonresponse bias analysis.............10

Guideline 1.1. Describe the survey target population ..................................................................10

Guideline 1.2. Describe the survey sample frame ........................................................................11

Guideline 1.3. Discuss the potential for coverage error of the frame..........................................11

Guideline 1.4. Describe the sample design ...................................................................................12

Guideline 1.5. Describe the sample unit........................................................................................12

Guideline 1.6. Describe the survey data collection modes..........................................................13

Guideline 1.7. Describe the survey’s key survey items, and identify those that will be used in 

the nonresponse bias analysis.........................................................................................................13

Guideline 1.8. Describe any nonresponse mitigation strategies tested during data collection ..13

Best Practice 2: Provide unit-level response rates for the survey and discuss potential for 

nonresponse bias................................................................................................................................. 14

Guideline 2.1. Identify and report unit-level response rates based on weights that adjust for 

selection probabilities.....................................................................................................................15

Guideline 2.2. Report unit-level response rates for geographic areas or key subgroups for 

which population-based estimates are published..........................................................................15

Guideline 2.3. Report item-level response rates for all key survey items...................................17

Guideline 2.4. Report response rates for each sampling stage that provides actionable 

information on survey participation and/or coverage...................................................................17

Guideline 2.5. Report response rates for each wave of data collection in longitudinal surveys18

Guideline 2.6. Report response rates for each group when employing responsive design or 

treatment groups..............................................................................................................................18

Guideline 2.7. Discuss the potential for nonresponse bias given weighted response rates, 

response rates by sampling stage, waves of data collection, and key subgroups .......................19

Best Practice 3: Describe the plan for evaluating and quantifying nonresponse bias and any 

mitigation strategies to be employed.................................................................................................19

Guideline 3.1. Describe and justify all nonresponse bias evaluation methods used ..................21

Guideline 3.2. Describe the analysis plan for comparing and assessing nonresponse bias levels 

across key stages and subgroups....................................................................................................21
Best Practices for Nonresponse Bias Reporting June 2023 2

Guideline 3.3. Describe post data collection nonresponse bias mitigation strategies that were 

employed and how they might reduce nonresponse bias.............................................................22

Best Practice 4: Describe and justify all available sources of auxiliary data used in the 

nonresponse bias analysis................................................................................................................... 22

Guideline 4.1. Describe all sources of auxiliary data used for the nonresponse bias analysis..24

Guideline 4.2. Discuss the characteristics, quality, and completeness of auxiliary data in terms 

of coverage, item missingness, measurement error, and timeliness............................................25

Guideline 4.3. Discuss the relationship between auxiliary data being used to evaluate 

nonresponse bias and the key survey items...................................................................................26

Best Practice 5: Describe results of nonresponse bias analysis including for all key survey items

.............................................................................................................................................................. 28

Guideline 5.1. Provide and discuss the results of the nonresponse bias analyses specified in the 

analytic plan ....................................................................................................................................28

Guideline 5.2. Describe the impact of post data collection mitigation strategies on reducing 

nonresponse bias.............................................................................................................................29

Guideline 5.3. Describe nonresponse bias before and after adjustment across key stages and 

subgroups.........................................................................................................................................29

Best Practice 6: Summarize the major conclusions of the analyses................................................30

Guideline 6.1. Summarize the results of the nonresponse bias analysis, any post-data collection 

mitigation strategies, and the final assessment of potential nonresponse bias for the full data 

collection and for key indicators....................................................................................................30

Guideline 6.2. Discuss the implications and potential causes of contradictory results..............31

Guideline 6.3. Discuss potential implications as they relate to stated objectives of the data 

collection .........................................................................................................................................32

Best Practice 7: Discuss recommendations for data collection methods and adjustment strategies 

to mitigate nonresponse bias in future waves or iterations of longitudinal and cross-sectional data 

collection ............................................................................................................................................. 32

Guideline 7.1. Discuss recommendations for modifications to sample design, questionnaires, 

or data collection strategies that may reduce the bias in collected data ......................................33

Guideline 7.2. Discuss recommendations that may improve post-data collection adjustment 

strategies..........................................................................................................................................33

REFERENCES ................................................................................................................................... 35

APPENDIX A: AUXILIARY DATA SOURCE TYPES................................................................38

APPENDIX B: AUXILIARY DATA QUALITY METRICS.........................................................41
Best Practices for Nonresponse Bias Reporting June 2023 3

OBJECTIVES

The goal of this report is to provide an overview of best practices for discussing and 

reporting on nonresponse bias in estimates obtained from survey data. Nonresponse bias is one 

of the main threats to data quality in federal surveys, as discussed in the 2020 report A 

Framework for Data Quality from the Federal Committee on Statistical Methodology (FCSM). 

Yet there are no consistent standards for reporting on nonresponse bias, a problem clearly 

demonstrated in another 2020 FCSM report: A Systematic Review of Nonresponse Bias Studies in 

Federally Sponsored Surveys (Miller et al., 2020). The current report addresses the latter 

concern. 

This report and both of the 2020 FCSM reports elaborate on the issues included in the 

2006 Office of Management and Budget (OMB) guidance Standards and Guidelines for 

Statistical Surveys. The OMB guidance focuses on several aspects of data quality and provides 

standards for describing surveys and reporting on nonresponse. The OMB guidance requires that 

all federal surveys conduct a nonresponse bias analysis if the program expects a unit-level 

response rate less than 80 percent, or an item-level response rate less than 70 percent (Office of 

Management and Budget 2006). The current report offers additional guidance for reporting on 

the methods, results, and conclusions from those nonresponse bias analyses. 

The literature on how to conduct nonresponse bias analyses is extensive and growing. 

The methods used for any data collection will be a function of the characteristics of that data 

collection and the availability of information to evaluate bias. The intent of the current report is 

not to provide a comprehensive guide to conducting nonresponse bias analyses. Rather, it is to 

provide guidance on reporting on the analyses done. To do this, the current report sometimes 

refers to various methods for conducting, and aspects of, nonresponse bias analyses. But these 

methods are not fully described. Readers will need to refer to the many existing references for 

more information on these methods. 

The current report also does not offer guidance for assessing whether the nonresponse 

bias present in any data collection is small enough to be acceptable or so large as to be 

unacceptable. Rather, the guidance in the current report is intended to yield sufficient 

documentation of the methods and results of nonresponse bias analyses so that readers can 
Best Practices for Nonresponse Bias Reporting June 2023 4

determine for themselves whether the data are of sufficient quality — that is, the data “are fit for 

their intended uses in operations, decision making, and planning” (Redman, 2001, p. 74). Survey 

programs generally cannot predict all purposes for which their data will be used. Moreover, even 

within the context of a specific intended use, assessment of “fitness for use” requires a 

multidimensional approach that considers many elements of data quality (FCSM, 2020). 

Nonresponse bias is just one (albeit important) element in one of the FCSM data quality 

dimensions (accuracy and reliability).

The guidance presented in the current report is designed for survey data collections. 

However, many of the best practices in this report can also be applied to reporting analyses of 

bias for other data sources. For example, gaps in data resulting from malfunctioning sensors in a 

large array can be thought of as equivalent to nonresponse in sensor-based data collections. Itemlevel missingness in administrative data may be analogous to item-level missingness in surveys, 

especially when items are not mandatory for completion of the form.
Best Practices for Nonresponse Bias Reporting June 2023 5

OVERVIEW

The generalizability of survey estimates, a key quality indicator (FCSM, 2020), relies on 

the premise that information is obtained from the entire universe or a randomly selected, 

representative sample of potential survey respondents selected from the population of interest. If 

information is not complete, response could be disproportionate across groups. Nonresponse bias 

can occur when the likelihood of responding to surveys or other data collection methods (or 

items) is correlated with key data collection outcomes. Put another way, nonresponse bias can 

occur when respondents differ from nonrespondents on key characteristics and key outcomes. 

Nonresponse bias affects both the accuracy and the reliability of population estimates. 

Nonresponse can occur for a variety of reasons both at the sample unit level (e.g., 

(persons, establishments, schools, farms, or hospitals) and at the item level (e.g., variables). 

Respondent knowledge and perceptions of burden, saliency, sensitivity, and/or privacy threats 

may vary across sampled units, leading to nonresponse that is non-random. When those and 

other drivers of nonresponse are related to the key survey items or characteristics being measured 

by the survey, nonresponse bias can occur. 

The relationship between the unit-level response rate and nonresponse bias is complex 

and researchers should avoid relying on response rates alone to determine data quality. Several 

studies have shown that response rates are not always correlated with nonresponse bias (Groves 

& Peytcheva, 2008; Hendra & Hill, 2019) and that boosting the response rate by bringing in 

additional cases does not necessarily decrease bias (Heerwegh, Abts, & Loosveldt, 2007). Some 

studies have even shown that efforts to increase response rates can increase bias by recruiting

subgroups that were already at a higher propensity to respond, versus those at lower propensities 

to respond, which could further exacerbate the nonresponse bias (Merkle & Edelman, 2009) or 
Best Practices for Nonresponse Bias Reporting June 2023 6

increase measurement error by coercing unwilling participants that give low quality responses 

(Barge & Gehlbach, 2012; Groves, 2006). It should also be noted that while response rates are 

often used as a proxy of overall data quality, using the response rate alone ignores all other 

possible sources of error that can cause survey estimates to differ from the underlying population 

parameters, such as coverage error, sampling error, measurement error, and processing error. 

Survey programs invest a significant amount of time and resources trying to maximize 

response rates and minimize the risk of nonresponse bias. While response rates alone are not 

indicative of data quality, the lower the response rate, the greater the opportunity is for 

nonresponse bias to occur (Groves, 2006; Groves & Peytcheva, 2008). This can be especially 

true in establishment surveys, where the impact of big retailers or producers not responding can 

have a large impact on the final estimate. 

An overarching best practice is to plan on conducting a nonresponse bias analysis as part 

of the survey life cycle. Besides setting aside time for this research and analysis, it is important 

to include questions in the data collection instrument that might help in the evaluation of 

nonresponse bias. For example, if a survey includes a longitudinal component, it may be useful 

to include questions in an early round that can be used to evaluate nonresponse during later 

follow-up data collection. Advice from program staff and contractors with subject matter and 

survey methods expertise should be solicited when developing the plan for the nonresponse bias 

analysis. Resources need to be devoted to finding comparable and high-quality auxiliary data to 

aid in the process. A useful nonresponse bias analysis is motivated by a desire to explore 

relationships between survey nonresponse – both at the unit level and at the item level – and the 

quality of the estimates produced. 
Best Practices for Nonresponse Bias Reporting June 2023 7 

Nonresponse bias analysis should be viewed as part of the core objective to mitigate total 

survey error — the difference between a survey estimate and the true value— thus enhancing the 

quality of the data. This difference results from the accumulation of all errors that arise in the 

design, collection, processing, and analysis of survey data (Biemer, 2010). Mitigation of 

response bias can be done during data collection using design elements (such as responsive 

design) or experiments that target known or hypothesized sources of error. Mitigation can also be 

done as part of the post-data collection processing where efforts are made to reduce, to the extent 

possible, any bias identified in the nonresponse bias analysis, for example by adjusting survey 

weights. 

The nonresponse bias analysis report should provide a clear description of the various 

analytic approaches used to determine the level of bias and any mitigation strategies 

implemented. As post-data collection analysis and mitigation generally are done in stages, the 

report should document each stage of analysis and mitigation. An ascertainment of residual bias 

after all mitigation strategies, if any, have been implemented is required for a final determination 

of potential bias and how it might affect the use of the data. This information enables data users 

to determine whether a particular data source is fit for their analytic purpose and for the users of 

statistical and analytic products to evaluate the appropriateness of findings.

While this report focuses on the discussion and reporting of nonresponse bias in surveys, 

nonresponse bias can even occur in the external sources used to evaluate potential nonresponse 

bias in surveys. Given that the sources used to evaluate nonresponse may also suffer from 

nonresponse bias, nonresponse rates and nonresponse bias should be reported for both the survey 

being evaluated for nonresponse bias and (if possible) the data used to evaluate nonresponse 

bias, to assess the potential for compounded or distorted bias estimates. 
Best Practices for Nonresponse Bias Reporting June 2023 8

Just as planning a nonresponse bias analysis needs to be built into the survey life cycle, 

successful mitigation strategies should be implemented for future data collections; therefore, 

recommendations for mitigating nonresponse bias based on the results of the study should be 

included in the report and should accompany the release of the data. The knowledge gained from 

conducting a nonresponse analysis should ultimately be used to mitigate future nonresponse, 

whether it is through design elements or post-survey adjustment methods (Wagner et al., 2012; 

Lepkowski et al., 2013). For recurring surveys, the findings of the nonresponse bias analysis can 

be used to target future data collection resources more efficiently and/or improve the existing 

nonresponse bias mitigation strategies (e.g., weighting or imputation). 

While survey programs often devote resources to assessing and addressing nonresponse 

bias, the results of this work are often not shared or published for public consumption. This 

creates two problems: 1) it prevents data users from understanding data quality concerns and 

biases; and 2) it prevents other survey programs from learning from each other’s experiences, 

including both successes and challenges. 

The best practices – and associated guidelines – presented in this document address two 

major concerns highlighted by the FCSM in 2020: 1) the threat of nonresponse to data quality, 

specifically accuracy and reliability (FCSM, 2020); and 2) the lack of standardized reporting and 

evaluation of nonresponse bias (Miller et al., 2020). This report provides a framework for

addressing these problems. Reporting and disseminating nonresponse bias analysis and research 

conducted by federal agencies and survey researchers more widely within a standard framework 

allows data collection entities to better plan, share, and evaluate nonresponse bias research and 

results within and across the user community.
Best Practices for Nonresponse Bias Reporting June 2023 9

The report is structured around seven best practices that provide guidance for developing 

a nonresponse bias analysis report: 1) describing the survey, 2) providing unit-level response 

rates and discussing the potential for nonresponse bias, 3) describing the plan for evaluating, 

quantifying, and mitigating nonresponse bias, 4) describing and justifying auxiliary data used to 

assess nonresponse bias, 5) describing the results of the nonresponse bias analysis, 6) 

summarizing the major conclusions of the nonresponse bias analysis, and 7) discussing 

recommendations for data collection methods and adjustment strategies to mitigate nonresponse 

bias in future data collections. The best practices are drawn from previous research and build on 

recommendations introduced in the FCSM report Transparent Reporting for Integrated Data 

Quality: Practices of Seven Federal Statistical Agencies (Prell et al., 2019). The best practices 

reinforce and are consistent with the current focus on the need to appropriately document all data 

collection procedures, including comprehensive metadata. As is the case with all guidance 

frameworks, not all best practices will apply in all situations. However, all best practices should 

be considered when developing and reporting on nonresponse bias and there should be 

appropriate justification for not following any best practice.
Best Practices for Nonresponse Bias Reporting June 2023 10

GUIDANCE FOR DEVELOPING A NONRESPONSE BIAS REPORT

Best Practice 1: Describe the survey that is the subject of the nonresponse bias analysis

There are several key elements of a survey that impact data quality (e.g., accuracy and 

reliability) and the magnitude of total survey error (e.g., coverage error, measurement error, 

nonresponse error, etc.) including the overall construct, design, administration, target population, 

sampling frame, sampling methods, and data collection modes. These elements can all impact 

data quality and should be discussed and considered before attempting to assess nonresponse 

bias. According to the FCSM report, A Framework for Data Quality (2020), “The accuracy of a 

data product reflects the accuracy of the input sources and all processing and calculations 

performed to transform those data into outputs. As these errors can accumulate throughout the 

data lifecycle, the accuracy of a data product will reflect a combination of the accuracy of its 

input sources, the processing steps applied to those inputs, and any additional calculations 

performed to transform the data into outputs” (p. 17). Given the potential for errors even without 

nonresponse, it is important to first evaluate the validity and generalizability of the key survey 

estimates given the overall survey design. The report should describe the key aspects of the 

survey design up through data collection, such as the population of inference, target population, 

sample frame, potential for coverage error, sample design, sample unit, data collection modes, 

key survey items, and any nonresponse bias mitigation used during data collection.

Guideline 1.1. Describe the survey target population

The report should describe the specific target population (people or establishments) of the 

survey and the population for which inferences will be made. For example, the target population 

for a survey might be the U.S. non-institutionalized population aged 12 or older, while other 

surveys may target agricultural operations, business establishments, or educational institutions.
Best Practices for Nonresponse Bias Reporting June 2023 11 

Guideline 1.2. Describe the survey sample frame

Probability samples utilize a frame (i.e., a finite list of sampling units in the population), 

where each sample unit has a known probability of selection. For example, an address-based 

household survey might construct a frame using the Census Bureau’s Master Address File, or by 

sending staff into the field to make lists of addresses (area listing), or by purchasing a 

commercial list of household addresses. Telephone-based household surveys may draw from 

banks of consecutive telephone numbers known to include at least one directory-listed residential 

telephone number or cell phone number. In some cases, a list of email addresses may provide a 

useful frame when all units have an email address and have access to email (e.g., all enrolled 

students at a university). For example, the Bureau of Labor Statistics uses unemployment 

insurance records as the sample frame when surveying businesses on job openings and labor 

turnover. 

The report should describe the survey frame and how it was built. Some surveys might 

rely on multiple frames in which case information on each frame used and how they are used 

together should be discussed. The report should provide a definition of the sampling units and 

clearly define how units are classified as eligible versus ineligible or out of scope. 

Guideline 1.3. Discuss the potential for coverage error of the frame

Coverage error represents the gap between the true value given the target population and 

the estimate given the sampling frame. According to the FCSM report, A Framework for Data 

Quality (2020), “For sample surveys, coverage error occurs when the sampling frame differs 

from the target population. Substantial coverage errors affect the utility of the data for inferences 

about the target population” (p. 38). For example, target household units without landline 
Best Practices for Nonresponse Bias Reporting June 2023 12

telephones would not be included in a frame consisting of landline phone numbers and would 

therefore have zero probability of being sampled in a survey of households.

Available sampling frames do not always align with the target population and may 

exclude certain populations and/or may not be complete or accurate. Even when the scope of a 

sampling frame aligns perfectly with the target population, it may include sample units that are 

no longer eligible or be missing newer sample units (e.g., recently formed households, schools, 

or business) that are eligible. Frames used for population or establishment surveys may also be 

out of date by the time the sample is selected if new housing units or establishments are formed 

between the time the frame is created and data collection begins. Any issues that impact the 

generalizability of the sample to the target population should be described for transparency and 

credibility.

Guideline 1.4. Describe the sample design

The report should provide a description for how the sample units were selected from the 

sample frame. Some surveys use simple random samples, while larger surveys meant to 

generalize to the U.S. population and have more disaggregated geographic levels (such as the 

state level) might use multi-stage stratified or cluster designs. If stratification or clustering was 

used, the report should provide a discussion of how these sampling decisions impact the 

estimates. Descriptions of any unequal probabilities of inclusion used should be included.

Guideline 1.5. Describe the sample unit

The report should describe the level(s) at which data were collected. For example, the 

report should discuss whether the survey sampled entire households as the unit or sampled 

persons within a sampled household, or both; whether the survey sampled entire educational 

institutions as the unit or sampled classes within sampled institutions and then students within 
Best Practices for Nonresponse Bias Reporting June 2023 13

sampled classes, or at all levels; or whether the survey sampled the entire business as the unit or 

sampled one or more locations for a given business.

Guideline 1.6. Describe the survey data collection modes

The report should describe the methods used to collect the data, particularly whether 

telephone, web, computer-assisted interviewing (CAI), touch tone or interactive voice response 

(IVR), data exchange, or a combination of modes (mixed mode collection) were used. Data 

collection mode may be correlated with coverage error, measurement error, and response error. 

For example, households without telephones cannot participate in telephone-only surveys 

(coverage error), whereas face-to-face data collection could potentially result in higher response 

rates (and perhaps lower nonresponse error) but also introduce measurement error if respondents 

are less likely to respond honestly to sensitive questions that are interviewer administered in 

person.

Guideline 1.7. Describe the survey’s key survey items, and identify those that will be used in 

the nonresponse bias analysis

While a survey may collect a variety of data elements, each survey is designed to produce 

specific key estimates or outcomes. The report should provide a description of the survey’s key 

estimates or outcomes and the set of variables used to produce them. Any other survey variables 

that are assessed in the nonresponse bias analysis should also be described.

Guideline 1.8. Describe any nonresponse mitigation strategies tested during data collection

The report should document and describe specific and targeted activities implemented or 

tested during data collection to increase response rates and reduce potential bias. Examples of 

such activities include experiments with incentives, respondent materials, or contact protocols. 
Best Practices for Nonresponse Bias Reporting June 2023 14

While all surveys use a variety of methods to increase response rates, this guideline calls 

specifically for specifying new, revised, or experimental data collection protocols designed to 

reduce nonresponse bias and for which an evaluation of bias reduction is warranted.

Best Practice 2: Provide unit-level response rates for the survey and discuss potential for 

nonresponse bias

Unit-level response rates are unit-level performance metrics that measure the proportion 

of the eligible sample units that responded to a survey. The report should provide a complete 

definition of what constitutes a “response.” While obtaining responses to all appropriate 

questions is the goal, breakoffs where the respondent completed a predetermined subset of 

questions that address the key elements of the survey are often considered as having responded 

to the survey. The criteria used to determine when a partial response is considered a response to 

the survey should be described. 

The unit-level response rate is often used as a proxy measure of survey data quality; 

however, it is not necessarily a strong predictor of the nonresponse bias in survey estimates

(Groves & Peytcheva, 2008; Hendra & Hill, 2019). While a high unit-level response rate is 

generally consistent with low bias, and a lower unit-level response rate is more likely to be 

associated with bias, there are some situations where the latter may not lead to bias. For example, 

response rates are associated with bias only if the likelihood of response is related to the key 

survey items. It is possible for a survey to have a low response rate but also key measures that 

are not related to survey response propensity. Therefore, those measures will not be subject to 

nonresponse bias. The unit-level response rate is the starting point for evaluating nonresponse 

bias. Conducting nonresponse bias analysis and reporting on the results is particularly important 

if unit-level rates are low (e.g., below 80 percent; Office of Management and Budget, 2006).
Best Practices for Nonresponse Bias Reporting June 2023 15

Guideline 2.1. Identify and report unit-level response rates based on weights that adjust for 

selection probabilities

The unit-level response rate measures the proportion of eligible units that responded to 

the survey. Unit-level response rates computed without survey weights provide a useful metric 

for evaluating the success of data collection for the selected units. But as a metric for evaluating 

the success of the survey with respect to the population sampled, the response rate calculations 

should incorporate the base weight for each unit. The base weight often is the inverse probability 

of selection and its inclusion in the response rate estimate results in a response rate calculation 

that uses a denominator that is the approximate total number of eligible sample units in the frame 

(population total). For establishment surveys, a measure of size should be used to calculate the 

weighted response rate to reduce the influence of small units with large sampling weights and 

appropriately represent the anticipated contribution of the large establishments on survey 

estimates. 

Unit-level response rates should be reported, to the extent possible, using standard 

definitions such as those published by the American Association for Public Opinion Research 

(AAPOR, 2023). Standard definitions have been vetted by the statistical and survey research 

communities and the justifications for the definitions are transparent and available to the user 

community. Rates that are calculated using standard definitions can also be compared over time 

for the same data collection as well as across data collections.

Guideline 2.2. Report unit-level response rates for geographic areas or key subgroups for 

which population-based estimates are published

Examining the unit-level response rates of subgroups can help identify potential sources 

of nonresponse bias, especially if it is found that the survey estimates vary by subgroup. The 
Best Practices for Nonresponse Bias Reporting June 2023 16

subgroups examined should reflect the subgroups for which key estimates are reported. For 

example, if the survey provides estimates for Census regions, lower levels of geography, or key 

subgroups, then response rates should also be provided for these subgroups. In addition, it can be 

instructive to produce response rates for subgroups that historically have lower response rates. 

Variation in response rates for subgroups not only identifies potential sources of bias but also 

alerts the user to potential limitations of the data set for that user’s intended analytic objectives. 

This variation can also be an important consideration when evaluating data through an equity 

lens. 

The ability to calculate response rates for subgroups will be limited by the data available 

at the sample level. Without a rich sampling frame or matched auxiliary data, characteristics of 

the nonrespondents may be unknown. Subgroup-specific response rates should be reported for 

key subgroups where information is available to directly calculate the response rate. 

Coverage ratios — a comparison of the sum of the sampling weights from the sampled 

units that have a particular characteristic to the same estimate (e.g., number of people with that 

characteristic) from an independent source — are sometimes used as a proxy for reporting 

response rates by key subgroups, when the information needed to identify subgroups is not 

available at the sample level. It should be noted, however, that coverage ratios calculated using 

the base weights conflate nonresponse error with coverage error. If coverage error has been 

described well (Guideline 1.3), it may be possible to use coverage ratios to infer differences in 

response rates by key subgroups.

R-indicators and partial R-indicators discussed in Schouten et al (2012) and the balance 

and distance indicators discussed in Lundquist and Särndal (2013) are additional methods of 

evaluating variation in unit-level response rates for subgroups.
Best Practices for Nonresponse Bias Reporting June 2023 17

Guideline 2.3. Report item-level response rates for all key survey items

The report should discuss the item-level response rates for all key survey items (see 

Guideline 1.7). A survey might have a very high unit-level response rate, but particular items

might have high item-level nonresponse rates, therefore rendering the item either unusable or 

needing imputation. Reporting item-level response rates is particularly important if some partial 

responses have been classified as unit-level respondents. For example, some surveys classify 

units as respondents if they just respond to one or more key survey items, which can lead to high 

item-nonresponse after the cutoff for defining when a partial response is a response to the survey. 

The item-level response rate calculations should incorporate the base weight for each 

unit. Unweighted and weighted item-level response rates may both be considered, but weighted 

rates are better indicators of potential bias. The unweighted rate describes the extent to which 

sample units provided information and reflects the success or challenges of the field work. High 

unweighted item-level nonresponse might indicate that the question was too sensitive, or the 

cognitive burden required to respond was too high, or the question was unclear, or respondents 

simply don’t have the answer. This may have implications for planned analyses or future 

surveys. However, the weighted rate is assumed to provide a better indicator of how well the 

reporting units describe the target population as it accounts for unequal probabilities of selection.

Guideline 2.4. Report response rates for each sampling stage that provides actionable 

information on survey participation and/or coverage

For surveys that employ complex sample designs with more than one stage of selection or 

clustering, response rates should be reported for each sampling stage and include a discussion of 

how nonresponse at each stage could affect the bias in the final estimates. For example, schoolbased surveys sample schools and then students within schools. Address-based surveys sample 
Best Practices for Nonresponse Bias Reporting June 2023 18

geographic areas then sample households within those areas, and persons within households. 

Establishment surveys sometimes breakdown sampling and unit response into different stages. 

For example, establishments may first be selected from a frame; then contacted to get mailing or 

email addresses and to verify frame information; then contacted to enroll them in the survey; and 

then recontacted at the start of data collection. Response rates can vary by phase of data 

collection, which could have a cumulative or counteracting impact in bias, so it is important to 

report response rates at each reporting phase, not just during the final data collection phase (see 

Earp et al., 2018).

Guideline 2.5. Report response rates for each wave of data collection in longitudinal surveys

Attrition at each wave of a panel or longitudinal survey can affect nonresponse bias as 

data collection continues. Response rates should be included for each data collection wave 

starting with the weighted and unweighted response rates for the first wave of data collection. 

Response rates should be reported for all key items in all waves (see Guideline 1.7). Response 

rates across waves can increase or decrease over time, and the variation in response rates can be 

related to key items. It should be noted whether cases can be missing for a given wave but still be 

considered part of the panel, or if they are removed following nonresponse at a certain wave, as 

this affects response rate calculations for subsequent waves.

Guideline 2.6. Report response rates for each group when employing responsive design or 

treatment groups

Adaptive or responsive designs intended to increase response from “hard to reach” units 

(i.e., population groups) are often used to reduce nonresponse bias (Wagner et al., 2012; 

Lepkowski et al., 2013). For example, treatments may be applied to all sampled units or 

subgroups to increase response (e.g., monetary incentives may be offered for completion of a 
Best Practices for Nonresponse Bias Reporting June 2023 19

survey), but not all sampled units may be offered the same treatment. However, these efforts can 

have unintended consequences and inadvertently introduce or increase bias. Response rates by 

treatment group (i.e., monetary offer amount) should be reported.

Guideline 2.7. Discuss the potential for nonresponse bias given weighted response rates, 

response rates by sampling stage, waves of data collection, and key subgroups

The potential bias introduced by overall nonresponse and by nonresponse for each of the 

key items should be discussed. Differential nonresponse by subgroup can affect bias in overall 

findings as well as findings for that subgroup, so the potential bias for subpopulations of interest 

should also be discussed. Level and observed variation in response rates by sampling stage and 

waves of data collection should also be discussed to present a more complete picture of how the 

variation may be affecting estimates. Weighted response rates should be reported and discussed 

in terms of how bias could be affected. (See Guidelines 2.1 through 2.6).

This discussion is intended to offer predictions and hypotheses to be evaluated and 

quantified when the analytic plan (Best Practice 3) is implemented.

Best Practice 3: Describe the plan for evaluating and quantifying nonresponse bias and any 

mitigation strategies to be employed

The report should include a clear description of activities used to reduce nonresponse bias 

during data collection (see Guideline 1.8), the analytic plan for evaluating nonresponse bias in 

the final data file, and, if applicable, the post-data-collection mitigation strategies (see Guideline 

3.3). The description of the plan should include the order in which analysis and mitigation are 

done. Mitigation strategies are those used to reduce biases identified in the nonresponse bias 

analysis. 
Best Practices for Nonresponse Bias Reporting June 2023 20

There are a variety of evaluation methods that can be used to assess nonresponse bias. 

The nonresponse bias assessment method(s) selected should depend on data collection 

characteristics and the availability of appropriate auxiliary sources of information (see Best 

Practice 4). To the extent possible, different methods should be used as each method focuses on 

different aspects of the effects of nonresponse and the use of multiple methods provides a more 

complete evaluation of bias. There is no standard or preferred order or manner in which different 

methods or strategies should be combined; however, the analysis and mitigation plan should 

provide the order of evaluation activities used as the effects of the methods or strategies used are 

cumulative. The report should help the reader understand the process through with mitigations 

were applied. New approaches to nonresponse bias analyses are being developed, so it is critical 

to keep informed on advances in the field. 

Groves (2006) and Wagner (2012) both suggest five main approaches to assessing 

nonresponse bias: 

1) assessing response rates over waves/time, across surveys, and within subgroups; 

2) comparing estimates across surveys or to similar estimates from other sources; 

3) studying estimate variation within the existing survey by comparing estimates within 

surveys across waves/time or via nonresponse follow-up studies; 

4) comparing statistics within a survey using rich sampling frame or matched auxiliary 

data; and 

5) assessing the impact of mitigation strategies by monitoring data collection and 

contrasting alternative postsurvey adjustment methods. 
Best Practices for Nonresponse Bias Reporting June 2023 21

These standard nonresponse bias analysis approaches fall into two main categories – those that 

use information from the data collection itself and those that use auxiliary data. 

The report should include a complete discussion of all evaluation methods used, 

including the rationale for the selection of each method along with an explanation of why some 

standard methods are not used. In addition, the discussion should address the aspect of 

nonresponse that the method will address.

Guideline 3.1. Describe and justify all nonresponse bias evaluation methods used

Evaluation methods can be direct or indirect. Direct assessment methods use data from 

the data collection such as the sampling frame or directly matched auxiliary data. Indirect 

assessment methods compare estimates from the data collection to similar estimates from other 

sources. The report should identify and describe all methods used to evaluate nonresponse bias, 

identifying which are direct and which are indirect.

Limitations of different methods are discussed by both Groves (2006) and Wagner 

(2012). The report should compare and contrast the limitations associated with all selected 

nonresponse bias assessment methods. The report should also discuss any steps taken to address 

these limitations and how the varying evaluation methods may or may not complement each 

other.

Guideline 3.2. Describe the analysis plan for comparing and assessing nonresponse bias levels 

across key stages and subgroups

The report should discuss all aspects of the plan for comparing and assessing 

nonresponse. This should include the potential impact on nonresponse bias of any observed 

differential response rates for multiple stages of data collection if applicable or for subgroups of 
Best Practices for Nonresponse Bias Reporting June 2023 22

interest (see Guideline 2.2). The plan for determining how the level and variation in response 

rates could impact bias should also be discussed.

Guideline 3.3. Describe post data collection nonresponse bias mitigation strategies that were 

employed and how they might reduce nonresponse bias

The report should describe all the strategies used after data collection was completed to 

mitigate previously identified biases, including adjustments to sample weights and imputation. 

Sufficient detail on any new post-data collection mitigation methods developed should be 

provided so that readers will understand the methods used.

Base weights can be adjusted in a variety of ways to mitigate nonresponse bias. 

Descriptions of unit-level nonresponse bias mitigation strategies such as poststratification, 

raking, and adjustment cell weighting should include an explanation of the models used, 

underlying assumptions about missingness, and discussion of the alternative approaches 

evaluated. These adjustments often involve a tradeoff between variance and bias. To the extent 

possible, decision criteria for managing these tradeoffs should be established prior to decisions 

about whether to employ mitigation strategies. Any efforts to attenuate the increased variance 

resulting from bias mitigation, such as weight trimming, should be clearly described.

Best Practice 4: Describe and justify all available sources of auxiliary data used in the 

nonresponse bias analysis

Auxiliary data are broadly defined as data that contain variables on the target population 

that are the same as or at least correlated to the key survey items. The type of nonresponse bias 

analysis as well as the type of inferences that can be drawn depend on what auxiliary sources are 

available at the time of analysis. Approaches that employ auxiliary data can provide critical 

information on nonresponse bias and identify ways to mitigate it. Because of the potential 
Best Practices for Nonresponse Bias Reporting June 2023 23

importance of auxiliary data for the evaluation and mitigation of nonresponse bias, issues related 

to the use of auxiliary data should be addressed. For example, the availability, completeness, 

timeliness, and accuracy of auxiliary data will impact the methods selected, how the analysis is 

done, and the interpretation of the results. 

The auxiliary data can be obtained from external sources or may include data collected 

through the survey but not used for reporting purposes (e.g., paradata). Auxiliary data fall into 

three broad categories:

1. Unit level: variables from another source that match (one-to-one) to the survey’s sample 

or collection units for respondents and nonrespondents. Unit-level data can come from a 

variety of sources including the survey frame, another survey, administrative records, 

paradata, or commercial data. 

2. Area level: reliable aggregate totals from subgroups or domains from another source that 

can be linked (many-to-one) to the unit-level survey data.

3. Benchmarks: reliable aggregate totals from another source that are closely related to the 

survey’s respondent-based estimates. 

As previously noted, auxiliary data that can be used to evaluate and mitigate bias must be at least 

weakly correlated to key survey outcomes. Auxiliary data that include the same concepts for the 

same target population provide the most direct assessment of nonresponse bias. Auxiliary data 

with high correlations can be used as “proxies” for key outcomes. But even variables that are 

weakly correlated may also be useful for assessing nonresponse bias. Groves (2011), Krueger 

and West (2014), Johnson and Smith (2017), Lohr (2004), and Rao (2003) suggest using 

multiple sources of auxiliary data whenever possible. The strength of the relationships between 

the auxiliary data and survey characteristics should be reported (Guideline 4.3).
Best Practices for Nonresponse Bias Reporting June 2023 24

See Appendix A for more information on selected types of auxiliary data. Advances in 

the availability and diversity of big data and in the development of artificial intelligence will 

likely lead to the identification and increasing use of new types of auxiliary data, including data 

extracted from maps, images, web scraping, and more. Any effort to develop a complete list of 

all possible types of auxiliary data would be quickly outdated.

Guideline 4.1. Describe all sources of auxiliary data used for the nonresponse bias analysis

Depending on the survey being assessed for nonresponse bias, there may be multiple 

sources of auxiliary data or very few, if any. All major potential auxiliary data sources should be 

discussed. The discussion of the auxiliary data should include whether they are available for all 

sampled units or for the entire population (Brick, 2013; Lohr, 2004; Rao, 2003; Wagner, 2012). 

If unit-level data are available for all sampled units, then quantified nonresponse bias at the unit 

level can and should be reported (Brick, 2013; Groves, 2006; Krueger & West, 2014; Rao, 2003; 

Wagner, 2012). Even if data are not available for all sampled units, geographic and demographic 

information should be used when available, and should be used to report bias at the area level or 

subdomain level (Brick, 2013; Groves, 2006, Rao, 2003; Wagner, 2012). In addition, if current 

population totals (benchmarks) are known, these should be compared to survey estimates and 

included in the report. 

Sources of auxiliary data can be obtained from the survey itself. Paradata (e.g., how long 

the interview took to complete or how long the respondent took to fill out the form) may be

actively or passively collected during the data collection process. Standardized interviewer 

observations might be tailored such that they are expected to be related to the key survey 

variables (West, 2013; West & Kreuter, 2015; West & Little, 2013). In addition, in surveys that 

screen for eligibility, adding questions to the screening instrument may provide auxiliary data 
Best Practices for Nonresponse Bias Reporting June 2023 25

that permit evaluation of nonresponse bias at the post-screening stage of recruitment (Wagner et 

al., 2012; Montaquila et al., 2013).

The report should describe the sources of the auxiliary data selected. In addition, 

information on potential sources evaluated but not selected should be provided along with 

criteria for selecting the sources used. All appropriate auxiliary sources should be used in the 

evaluation. If multiple sources of auxiliary data exist, the report should discuss nonresponse bias 

results using each data source. It is expected that sources will vary in terms of quality and 

relevance, and that results from different auxiliary sources may vary. See Appendix A for a 

discussion of different auxiliary data sources.

Guideline 4.2. Discuss the characteristics, quality, and completeness of auxiliary data in terms 

of coverage, item missingness, measurement error, and timeliness

The importance of reliable and conceptually comparable auxiliary data for all key survey 

items in nonresponse bias analysis is emphasized in the survey research methods literature; 

however, this rarely occurs in practice (Brick, 2013; Andridge & Thompson, 2015; Olson, 2013; 

Groves, 2006). A good source to review for assessing and reporting auxiliary data quality is the 

FCSM Data Quality Assessment Tool for Administrative Data (Iwig, Berning, Marck, & Prell, 

2013).

There are a variety of challenges and limitations to consider when using auxiliary data. 

The availability of auxiliary data may be very limited. The auxiliary data may also include 

nonresponse bias and measurement error. Ideally, auxiliary variables should provide non-missing 

values for all respondents and nonrespondents and be free of measurement error (Kreuter at al., 

2010; Olson, 2013; Massey & Tourangeau, 2013). This ideal is difficult to achieve. Therefore, 

the nonresponse bias analysis report should discuss the strengths and limitations of each 
Best Practices for Nonresponse Bias Reporting June 2023 26

auxiliary data source with the respect to the analysis. The most useful auxiliary data are 

themselves of high quality with low levels of missingness and measurement error. See Appendix 

B for a discussion of missing data, coverage, measurement error, and data timeliness.

Guideline 4.3. Discuss the relationship between auxiliary data being used to evaluate 

nonresponse bias and the key survey items

When assembling different sets of auxiliary data for nonresponse bias analyses, priority 

should be given to identifying variables of high quality (see Appendix B) that are expected to be 

related both to survey response propensity and the survey characteristics of interest (i.e., key 

survey items) (Andridge & Thompson, 2015; Groves, 2006; Krueger & West, 2014; Olson, 

2013; Särndal & Lundström, 2008; Kreuter et al., 2010; Massey & Tourangeau, 2013). Of the 

two, it is more important to find strong correlates of survey items in nonresponse analysis. 

Several studies show that using auxiliary variables that are weakly correlated with the survey 

variables of interest may not only result in poor nonresponse bias assessment and nonresponse 

adjustment but may also unnecessarily increase variance (if included in the adjustment 

procedure) (Kreuter et al., 2010; Little & Vartivarian, 2005; Olson, 2013; McConville & Toth, 

2019).

While a strong association between related measures is expected, the association may 

vary depending on differences in definition, collection units, and timeliness. Differences between 

the survey of interest and the benchmark do not necessarily indicate nonresponse bias but can 

also be due to other factors. In addition to subtle differences in definitions, reference populations, 

and timing, survey estimates may also differ from benchmarks because of the survey’s “social 

setting” (i.e., the location of survey administration such as the respondent’s home, a clinic, or a 

school, etc.), mode of collection (e.g., online or paper forms), context or priming effects (from 
Best Practices for Nonresponse Bias Reporting June 2023 27

other questions on the survey or the topic of the survey), and respondent fatigue (more common 

for questions near the end of the survey), all of which have been shown to influence survey 

responses in some instances. In many cases, the associations may be much weaker for survey 

measures that measure similar but related concepts. In addition, all surveys are subject to 

nonresponse and potential nonresponse bias, as well as measurement, coverage, processing, and 

sampling errors. 

When items in the survey data can be matched to auxiliary data items (e.g., the same 

variable exists for the same unit in another data source), then it is a best practice to assess the 

relationship between the items. For example, agricultural production surveys often match 

variables to similar variables collected on the Census of Agriculture to assess the potential for 

nonresponse bias. By comparing reported Census values for survey respondents and 

nonrespondents, it may be possible to determine if there is potential for nonresponse bias. 

Correlation and regression type analyses are commonly used to assess the relationship between

matched variables. Such analyses can indicate the degree to which the auxiliary data might be 

useful in understanding the amount of bias due to nonresponse in the survey data.

While it is common practice in household surveys to adjust on age, sex, race/ethnicity, 

and geographic location, relying on these characteristics alone may not be enough to adequately 

adjust for nonresponse bias. If no other auxiliary data exists, one should at the very least report 

interaction effects between key demographics in relation to survey nonresponse (Brick, 2013; 

Peytcheva & Groves, 2009; Phipps and Toth, 2012). For example, using regression trees to 

model the relationship between establishment or household characteristics and nonresponse can 

provide insight as to whether nonresponse propensities vary across demographic subgroups. 

While variation across demographic subgroups alone does not necessarily indicate bias, it can be 
Best Practices for Nonresponse Bias Reporting June 2023 28

informative if inequities in nonresponse are also thought to be related to inequities in the survey 

outcome.

Best Practice 5: Describe results of nonresponse bias analysis including for all key survey 

items

The results — that is, the extent of bias identified from the nonresponse bias analysis —

should be reported for each method used, even if the results were inconclusive. If pre or post

mitigation strategies were implemented, the results of those strategies on bias indicators and key 

survey items should also be reported. If nonresponse bias analyses reveal that the relationship 

between response propensity and key survey item values is not well understood and there are 

concerns about the potential biasing effects of nonresponse, then the report may include a 

recommendation for future nonresponse follow-up studies. However, the recommendation 

should include a caveat that these studies should only be undertaken if enough resources can be 

dedicated to ensuring a study with sufficient sample size and without similarly high rates of 

nonresponse.

Guideline 5.1. Provide and discuss the results of the nonresponse bias analyses specified in the 

analytic plan

The report should discuss results of all components of the analysis included in the 

analytic plan. The discussion should address potential bias using indicators of bias for the key 

survey items (see Guideline 1.7), for key subgroups (see Guideline 2.2), and for each sampling 

stage and wave of data collection (see Guidelines 2.4 and 2.5). The results of all evaluation 

methods (see Guideline 3.1) should be presented individually, and differences and similarities in 

findings should be highlighted.
Best Practices for Nonresponse Bias Reporting June 2023 29

Guideline 5.2. Describe the impact of post data collection mitigation strategies on reducing 

nonresponse bias

While one aim of nonresponse bias analysis is to quantify the amount and nature of the 

bias introduced by nonresponse, an equally important aim is to identify possible mitigation 

strategies based on the results of the nonresponse bias analysis and apply them to the dataset. 

Ideally, these strategies can be tested on the current collection set (or subset thereof) but may 

have to be postponed to subsequent collections (see Best Practice 7). Once those strategies are 

implemented, it is then necessary to conduct a subsequent nonresponse bias analysis on the 

mitigated data, using the same variables as in the original analysis. The impact of mitigation 

strategies on nonresponse bias reduction for key survey items should be discussed. In addition, 

the extent of bias that remains after the mitigation strategies have been applied should also be 

presented.

Post data collection adjustments to mitigate nonresponse bias often involve a tradeoff 

between variance and bias. In addition to describing the impact on reducing bias, the discussion 

should also describe how the properties of the survey weights and the survey design effects may 

have changed.

Guideline 5.3. Describe nonresponse bias before and after adjustment across key stages and 

subgroups

The results of nonresponse bias analysis done after adjustment should be reported by 

each sampling phase and for each wave of data collection in longitudinal or panel studies as well 

as for key subgroups, including those for which geographic and demographic estimates are 

published (see Guideline 2.2 to 2.6). The change in nonresponse bias before and after adjustment 
Best Practices for Nonresponse Bias Reporting June 2023 30

by stages, waves, and subgroups should be discussed to evaluate the impact of the adjustment 

strategies and to highlight the remaining bias that will need to be considered when using the data.

Best Practice 6: Summarize the major conclusions of the analyses

The report should provide a summary that includes high-level findings for the full data 

collection activity and for the key survey items, overall and by studied subgroup. If possible, the 

report should highlight post-data collection mitigation strategies that appear to be effective, as 

well as those that are less effective.

Guideline 6.1. Summarize the results of the nonresponse bias analysis, any post-data 

collection mitigation strategies, and the final assessment of potential nonresponse bias for the 

full data collection and for key indicators

All results of the nonresponse bias analysis should be summarized so that the reader has a 

good understanding of how the analyses were done and the magnitude and nature of bias that is 

not addressed through the mitigation process. This summary should not be merely an 

abbreviation of the findings, but a narrative that tells the story in a clear and concise manner.

When summarizing the results based on auxiliary data, information on the strength of the 

relationship between the auxiliary data and the survey data should be considered, as well as the 

quality of the auxiliary data. The summary should put greater emphasis on results that are 

obtained using strongly related auxiliary data as predictors or as benchmarks as opposed to 

findings via indirect assessments. When presenting results for specific outcome variables, the 

summary should include results by studied subgroups as well as for the full data collection, since 

nonresponse bias for each key survey item can vary by subgroup.
Best Practices for Nonresponse Bias Reporting June 2023 31

Assessments of post-data collection mitigation strategies based on the nonresponse bias 

analysis may vary by the key survey items. The summary should emphasize results from 

implemented strategies that have effects on nonresponse bias. Analyses and mitigation activities 

are often done in cycles which has the advantage of showing the effect that each mitigation 

strategy has on reducing observed bias. The nonresponse analysis completed after the last cycle 

of mitigation will be of most relevance to data users in their determination of whether the data 

are fit for their purposes and if any residual bias could impact their interpretation of the results of 

their own analyses.

Guideline 6.2. Discuss the implications and potential causes of contradictory results

Best Practice 3 advocates employing a variety of methods to assess nonresponse bias to 

obtain a more complete evaluation of bias. When more than one analysis method is applied to the 

same outcome variable, similarities and differences in the indicators of bias should be discussed, 

emphasizing the (cumulative) abundance of evidence for or against the presence of nonresponse 

bias. Contradictory results for the same outcome variable could be due to a variety of factors, 

including differences in the strength of the relationships between different sources of auxiliary 

data and survey estimates, auxiliary data shown to be of varying quality, or inappropriate 

assumptions in the assessment method, among others. Evaluate the relative strengths and 

weaknesses of the methods used with respect to the studied outcome variable. If possible, the 

summary should emphasize results obtained with the stronger methods, without completely 

discounting the others.
Best Practices for Nonresponse Bias Reporting June 2023 32

Guideline 6.3. Discuss potential implications as they relate to stated objectives of the data 

collection

The result of the nonresponse bias analysis and the impact of any mitigation strategies 

conducted should be discussed in terms of the core objectives of the data collection. The results 

should be discussed in relation to the key items in terms of whether the level and type of residual 

bias affect the usefulness of the data for the main objectives of the collection. Overwhelming 

evidence of nonresponse bias in a key item in one or more subgroups can affect the ability to use 

the data to meet the core data collection objectives. To the extent possible, provide guidance to 

the potential data user regarding strengths and limitations of the data for the objectives for which 

the data were collected.

Best Practice 7: Discuss recommendations for data collection methods and adjustment 

strategies to mitigate nonresponse bias in future waves or iterations of longitudinal and 

cross-sectional data collection

A thorough nonresponse bias analysis that follows the previous guidelines for reporting 

should provide useful insight into the likely sources of nonresponse bias. In recurring surveys, 

whether cross-sectional or longitudinal, this information can be used to improve future data 

collection. These improvements may either reduce response bias during data collection or 

improve the ability to reduce bias after data collection. Recommendations for how the results of 

the current evaluation may be used to mitigate nonresponse in the future should be offered and 

discussed.
Best Practices for Nonresponse Bias Reporting June 2023 33

Guideline 7.1. Discuss recommendations for modifications to sample design, questionnaires, 

or data collection strategies that may reduce the bias in collected data

The evaluation of nonresponse bias and mitigation strategies may identify potential 

sources of nonresponse bias. These insights can be used to suggest modifications to sampling, 

such as respecifying strata or oversampling groups with lower response propensities, or to future 

data collection strategies by including adaptive or responsive design. They may inform 

modifications to the questionnaire, such as relocating or removing items that result in high rates 

of breakoffs. The results of experiments embedded in previous waves may suggest optimal 

follow-up strategies or incentive structures. Based on the results described in Best Practice 5, the 

report should provide recommendations for further reducing nonresponse bias in collected data. 

Those recommendations may include specific ideas for research to better understand and address 

the impact of data collection strategies on nonresponse bias. Recommendations may also include 

design changes for sampling stages, waves, or key subgroups to increase sample sizes and 

enhance the power of post adjustment methods.

Guideline 7.2. Discuss recommendations that may improve post-data collection adjustment 

strategies

The report should offer and discuss recommendations for improving nonresponse 

mitigation and adjustment strategies following future data collections, given the results of the 

post-data collection adjustment strategies – such as statistical imputation, poststratification, 

raking, adjustment cell weighting and weight trimming – that were used. Recommendations may 

also include questionnaire modifications to better align constructs with existing benchmarks or 

collect new data that may permit more complex or refined weighting adjustments and 

imputations. If the availability of resources limited the scope of the present nonresponse bias 
Best Practices for Nonresponse Bias Reporting June 2023 34

analysis or post-data collection adjustment strategies, recommendations could also include a 

description of the desirable approaches that could not be evaluated. Recommendations should be 

discussed in terms of the tradeoffs between variance and nonresponse bias. The report should 

provide recommendations, if any, for the use of decision criteria for managing these tradeoffs. 

The recommendations may also include specific ideas for research to better understand and 

address post data collection adjustment strategies to mitigate nonresponse bias.
Best Practices for Nonresponse Bias Reporting June 2023 35 

REFERENCES

American Association for Public Opinion Research (AAPOR). (2023). Standard definitions: 

Final dispositions of case codes and outcome rates for surveys, 10th edition.

https://aapor.org/wp-content/uploads/2023/05/Standards-Definitions-10th-edition.pdf

Andridge, R., & Thompson, K. J. (2015). Using the fraction of missing information to identify 

auxiliary variables for imputation procedures via proxy pattern‐mixture models. International 

Statistical Review, 83(3), 472-492.

Barge, S., & Gehlbach, H. (2012). Using the theory of satisficing to evaluate the quality of 

survey data. Research in Higher Education, 53(2), 182-200.

Biemer, P. P. (2010). Total survey error: design, implementation, and evaluation. Public Opinion 

Quarterly, 74(5), 817–848.

Brick, J. M. (2013). Unit nonresponse and weighting adjustments: A critical review. Journal of 

Official Statistics, 29(3), 329-353.

Earp, M., Mitchell, M., McCarthy, J., & Kreuter, F. (2014). Modeling nonresponse in 

establishment surveys: Using an ensemble tree model to create nonresponse propensity scores 

and detect potential bias in an agricultural survey. Journal of Official Statistics, 30(4), 701-719.

Earp, M., Toth, D., Phipps, P., & Oslund, C. (2016). Identifying establishment characteristics of 

potential nonresponse bias. Bureau of Labor Statistics: Washington, DC. 

https://www.bls.gov/osmr/research-papers/2016/pdf/st160020.pdf 

Earp, M., Toth, D., Phipps, P., & Oslund, C. (2018). Assessing nonresponse in a longitudinal 

establishment survey using regression trees. Journal of Official Statistics (JOS), 34(2).

Federal Committee on Statistical Methodology (FCSM). (2020). A framework for data quality. 

FCSM 20-04. Federal Committee on Statistical Methodology: Washington, DC. 

https://www.fcsm.gov/assets/files/docs/FCSM.20.04_A_Framework_for_Data_Quality.pdf

Groves, R. M. (2006). Nonresponse rates and nonresponse bias in household surveys. Public 

Opinion Quarterly, 70(5), 646-675.

Groves, R. M. (2011). Three eras of survey research. Public Opinion Quarterly, 75(5), 861-871.

Groves, R. M., & Peytcheva, E. (2008). The impact of nonresponse rates on nonresponse bias: a 

meta-analysis. Public Opinion Quarterly, 72(2), 167-189.

Heerwegh, D., Abts, K., & Loosveldt, G. (2007, January). Minimizing survey refusal and 

noncontact rates: do our efforts pay off?. In Survey Research Methods (Vol. 1, No. 1, pp. 3-10).

Hendra, R., & Hill, A. (2019). Rethinking response rates: new evidence of little relationship 

between survey response rates and nonresponse bias. Evaluation Review, 43(5), 307-330.

Iwig, W., Berning, M., Marck, P., & Prell, M. (2013). Data quality assessment tool for 

administrative data. Federal Committee on Statistical Methodology: Washington, D.C. 

https://www.fcsm.gov/assets/files/docs/DataQualityAssessmentTool.pdf

Johnson, T. P., & Smith, T. W. (2017). Big data and survey research: Supplement or substitute?. 

In: Thakuriah, P., Tilahun, N., Zellner, M. (eds), Seeing cities through big data. Springer 

Geography. Springer, Cham.
Best Practices for Nonresponse Bias Reporting June 2023 36

Kreuter, F. (Ed.). (2013). Improving surveys with paradata: Analytic uses of process information 

(Vol. 581). John Wiley & Sons.

Kreuter, F., Olson, K., Wagner, J., Yan, T., Ezzati‐Rice, T. M., Casas‐Cordero, C., Lemay, M., 

Peytchev, A. & Raghunathan, T. E. (2010). Using proxy measures and other correlates of survey 

outcomes to adjust for non‐response: examples from multiple surveys. Journal of the Royal 

Statistical Society: Series A (Statistics in Society), 173(2), 389-407.

Krueger, B. S., & West, B. T. (2014). Assessing the potential of paradata and other auxiliary data 

for nonresponse adjustments. Public Opinion Quarterly, 78(4), 795-831.

Lepkowski, J. M., Mosher, W. D., Groves, R. M., West, B. T., Wagner, J., & Gu, H. (2013). 

Responsive design, weighting, and variance estimation in the 2006-2010 National Survey of 

Family Growth. National Survey for Health Statistics. Vital and Health Statistics, 2(158), 1-52.

Little, R. J., & Vartivarian, S. (2005). Does weighting for nonresponse increase the variance of 

survey means?. Survey Methodology, 31(2), 161-168.

Lohr, S. L. (2004). Small area and longitudinal estimation using information from multiple 

surveys. Statistical Policy, 77.

Lundquist, P., & Särndal, C-E. (2013). Aspects of responsive design with applications to the 

Swedish Living Conditions Survey. Journal of Official Statistics, 29(4), 557-582. 

Maitland, A., Casas-Cordero, C., & Kreuter, F. (2009, August). An evaluation of nonresponse 

bias using paradata from a health survey. In Proceedings of the Section on Survey Research 

Methods of the American Statistical Association (pp. 370-78).

Massey, D. S., & Tourangeau, R. (2013). Where do we go from here? Nonresponse and social 

measurement. The ANNALS of the American Academy of Political and Social Science, 645(1), 

222-236.

McConville, K. S., & Toth, D. (2019). Automated selection of post‐strata using a model‐assisted 

regression tree estimator. Scandinavian Journal of Statistics, 46(2), 389-413.

Merkle, D. M., & Edelman, M. (2009). An experiment on improving response rates and its 

unintended impact on survey error. Survey Practice, 2(3), 2964.

Miller, P., Fakhouri, T. H., Earp, M., Piscopo, K. D, Frenk, S. M. Christopher, E., & Madans, J. 

(2020). A systematic review of nonresponse bias studies in federally sponsored surveys. FCSM 

20-02. Federal Committee on Statistical Methodology. March 2020.

https://www.fcsm.gov/assets/files/docs/A_Systematic_Review_of_Nonresponse_Bias_Studies_F

ederally_Sponsored_SurveysFCSM_20_02_032920.pdf

Montaquila, J. M., Brick, J. M., Williams, D., Kim, K., & Han, D. (2013). A study of two-phase 

mail survey data collection methods. Journal of Survey Statistics and Methodology, 1(1), 66-87. 

Office of Management and Budget. (2006). Standards and guidelines for statistical surveys.

https://www.whitehouse.gov/wp-content/uploads/2021/04/standards_stat_surveys.pdf

Olson, K. (2013). Paradata for nonresponse adjustment. The Annals of the American Academy 

of Political and Social Science, 645(1), 142-170.
Best Practices for Nonresponse Bias Reporting June 2023 37

Peytcheva, E., & Groves, R. M. (2009). Using variation in response rates of demographic 

subgroups as evidence of nonresponse bias in survey estimates. Journal of Official Statistics, 

25(2), 193-201.

Phipps, P., & Toth, D. (2012). Analyzing establishment nonresponse using an interpretable 

regression tree model with linked administrative data. The Annals of Applied Statistics, 772-794.

Prell, M., Chapman, C., Adeshiyan, S., Fixler, D., Garlin, T., Mirel, L., & Phipps, P. (2019). 

Transparent reporting for integrated data quality: Practices of seven federal statistical agencies. 

FCSM 19-01. Federal Committee on Statistical Methodology. September 2019.

https://www.fcsm.gov/assets/files/docs/Transparent_Reporting_FCSM_19_01_092719.pdf

Rao, J. N. K. (2003), Small area estimation, New York: Wiley.

Redman, T. C. (2001). Data quality: the field guide. Boston: Digital Press.

Särndal, C.-E., & Lundström, S. (2008). Assessing auxiliary vectors for control of nonresponse 

bias in the calibration estimator. Journal of Official Statistics, 24(2), 167-191.

Schouten, B., Bethlehem, J., Buellens, K., Kleven, Ø., Loosveldt, G., Luiten, A., Rutar, K., 

Shlomo, N., & Skinner, C. (2012). Evaluating, comparing, monitoring, and improving 

representativeness of survey response through R-indicators and partial R-indicators. International 

Survey Review, 80(3), 382–399.

Wagner, J. (2012). A comparison of alternative indicators for the risk of nonresponse bias. Public 

Opinion Quarterly, 76(3), 555-575.

Wagner, J., West, B. T., Kirgis, N., Lepkowski, J. M., Axinn, W. G., & Ndiaye, S. K. (2012). 

Use of paradata in a responsive design framework to manage a field data collection. Journal of 

Official Statistics, 28(4), 477-499. 

West, B. T. (2013). An examination of the quality and utility of interviewer observations in the 

National Survey of Family Growth. Journal of the Royal Statistical Society: Series A (Statistics 

in Society), 176(1), 211-225. 

West, B. T. & Kreuter, F. (2015). A practical technique for improving the accuracy of 

interviewer observations of respondent characteristics. Field Methods, 27(2), 144-162. 

West, B. T. & Little, R. J. A. (2013). Non-response adjustment of survey estimates based on 

auxiliary variables subject to error. Journal of the Royal Statistical Society: Series C (Applied 

Statistics), 62(2), 213-231.
Best Practices for Nonresponse Bias Reporting June 2023 38 

APPENDIX A: AUXILIARY DATA SOURCE TYPES

Frame Data

One of the most common types of auxiliary data is sample frame data. The richness of the 

frame data — that is, the extent and nature of the data on all eligible sample units — tends to 

vary across establishment and household surveys (Brick, 2013; Groves, 2006). Establishment 

surveys like the Jobs Openings and Labor Turnover and the Occupational Requirements Surveys 

rely on rich sampling frames like the Quarterly Census of Employment and Wages (QCEW) that 

provide auxiliary data on establishment size, age, sector, location, employment rate, and wage 

data (Phipps & Toth, 2012; Earp et al., 2016). Household surveys like the Current Population 

Survey and the Current Expenditure Survey rely on less rich and at times less current sampling 

frames like the Master Address File. Most agriculture surveys use the Census of Agriculture as a 

frame, which provides an extensive list of farms, containing a rich source of household and 

establishment-level variables for farm operations (Earp et al., 2014). 

Establishment surveys may have access to richer frames than household surveys in terms 

of auxiliary data and may also include more relatively current census data. For example, the 

QCEW collects establishment data every quarter and the Economic Census is conducted every 

five years; in contrast, the Decennial Census collects household data every ten years (Brick, 

2013). As the time between sample selection and data collection increases, the comparability of 

the frame variables and survey items decreases. Furthermore, sampling unit compositions and 

classification can change over time. For example, a company might grow or shrink due to 

acquisitions or divestitures; an establishment could change industries when a different business 

moves into the same location.
Best Practices for Nonresponse Bias Reporting June 2023 39 

Administrative Records

Administrative data differ from survey data, even when both collect similar variables. 

Survey data are collected for statistical purposes, allowing for instrument design tailored to 

specific outcomes. Administrative data are collected to facilitate the operation of programs and 

reflect the requirements of those programs. Consequently, there can be differences in the 

collection units, which in turn can affect comparisons. It should not be assumed that 

administrative data are not subject to measurement error. Rather survey and administrative data 

are subject to different sources of measurement error related to how the data are collected and 

who is providing the information. As with other sources of auxiliary data, these potential 

differences must be considered when evaluating whether differences are due to survey 

nonresponse bias.

Paradata

According to Kreuter (2013) and Olson (2013), paradata are data that are actively or 

passively collected during the data collection process, can be collected in face to face, telephone, 

and web surveys, and including everything from contact history information and interviewer 

observations to potential mail tracing and respondent keystroke entries. An example of paradata 

use in a nonresponse bias analysis is the evaluation by Maitland et al. in the National Health 

Interview Survey (Maitland, Casas-Cordero, & Kreuter, 2009) This evaluation used data about 

what types of efforts were made to contact the household and data on household cooperation 

(mostly consisting of reasons households gave for not wanting to participate) to improve 

previous models of non-response bias in the survey. 

Benchmarks
Best Practices for Nonresponse Bias Reporting June 2023 40

Benchmarks are aggregate estimates (totals) that provide independently obtained 

estimates of the survey total. Differences between benchmarks and (appropriately weighted) 

respondent totals are often provided as estimates of nonresponse bias. 

Gold standard benchmarks should be valid measures of the construct, be defined 

similarly, have comparable coverage, and have high reliability in terms of sampling and 

nonsampling errors. Any possible differences in measurement, coverage, and reliability between 

the survey totals and corresponding benchmarks should be discussed in the nonresponse bias 

analysis before providing comparisons.
Best Practices for Nonresponse Bias Reporting June 2023 41 

APPENDIX B: AUXILIARY DATA QUALITY METRICS

Missingness

Considerations of missingness in auxiliary data might be different depending on whether 

the auxiliary data are survey, administrative, paradata, or other types of data. For example, if 

paradata or contact history instrument (CHI) data are used, it is important to understand the

nature and extent of missing data. Missingness in the auxiliary data may be correlated with 

missingness in the survey data and may limit the accuracy of the findings. 

Measurement Error

Auxiliary data typically comes from another source of data source and may have slightly 

different measurement properties as a result. According to Biemer et al. (2013), there are three 

main sources of measurement error to consider: 1) measuring the wrong variable; 2) incorrect or 

biased measures; and 3) measurement variability due to the instrument. When assessing the 

appropriateness of auxiliary data, consider the appropriateness of the way the variable is 

conceptualized, any known measurement biases, and any known measurement variability. 

Coverage

In the survey design literature, auxiliary data coverage describes overlap between the 

target population and the auxiliary data source. Known differences of coverage in auxiliary data 

should be noted when conducting and reporting nonresponse bias analyses. 

On occasion, an excellent source of auxiliary data will overlap with only part of the target 

population. In this case, the nonresponse bias analysis is valid for the overlap segment, but the 

analyses should not be extrapolated to the entire survey. For example, the target population could 

be the entire United States, but the auxiliary data could contain data collected solely in the 
Best Practices for Nonresponse Bias Reporting June 2023 42 

Midwest. In this case, the nonresponse bias analysis would be valid for the Midwest (assuming 

the survey data are representative of the Midwest), but not for the Northeast, South, or West 

census regions.

Timeliness

For nonresponse bias analyses, the terms “timeliness” and “age” are interchangeable and 

refer to the difference between the survey’s reference period for data collection and the reference 

period for the auxiliary data’s reference period. Both should be provided in the nonresponse bias 

analysis report. 

The age of the auxiliary data alone may not be a concern if some assessment has been 

done to ensure that data are still up to date in terms of the survey data collection period. 

However, older data may indicate a higher likelihood of divergence from the survey data. It is 

important for the analyst to know when the data were last collected and updated or checked for 

accuracy and to match the reference periods for data collection especially in cases where 

population fluctuations are known to happen at given intervals (e.g., households moving during 

the summer, or numbers of children enrolled in a school at the end of each school term).