Survey Quality Indicator Measures: 

Response Rates and Alternatives 

2012 Federal Committee on Statistical 

Methodology Research Conference 

January 11 2012 January 11, 2012 

Washington, DC 

Donsig Jang  Sixia Chen (Iowa State University) 

Fl L (N ti l S i F d ti ) Flora Lan (National Science Foundation) 


 

 

Background 

 Survey goal is to provide high-quality data 

 Response rate was rate was considered an indicator of considered an indicator of 

survey quality 

– Data collection goal: High response rate 

 But the threshold for a high response rate has 

changed due to decreasing response rates (90% 

to 70% or even lower) from the early 1990s to to 70%, or even lower) from the early 1990s to 

mid-2000s 

 Non-response bias response bias b y( )  (1 )( y y )

r nr

Non  (

2
 

Background (cont’d.) 

 Response rates alone are not good indicators 

of nonresponse bias (Groves and bias (Groves and Peytcheva Peytcheva 

2008) 

 Alternatives 

– Multiple thresholds of response rates by key 

domains 

– R-indicator (Schouten et al. 2009) measuring 

representativeness 

• Measure of response propensity rate variation among 

respondents

• L d eaing to focus on less representing subpopulations 

during a late stage of data collection 

3
 

Objectives 

 Empirical evaluation of response rates and 

R-indicators using real survey data 

– Ob th l ti hi b t th Observe the relationship between the two measures 

– Understand the relationship between each measure 

(response rate or R-measure) and potential 

nonresponse nonresponse bias 

 Decision for data collection closeout 

– Based on response rate, R on response rate, R -indicator, both or other indicator, both or other 

alternative indicators? 

4
 

Methods 

 Examine R-indicator and response rate trends 

over the data the data collection period collection period 

 Calculate key survey and frame variable

estimates on a weekly basis durin g data 

collection 

 Calculate upper-bound estimates for bias and 

root mean square errors for weekly estimates

during data collection 

5
 

Data: National Survey of Recent College 

Graduates (NSRCG) 

 Sponsored by the National Science Foundation

(NSF) and conducted every two or three years and conducted every two or three years 

since 1974 

 Targets recent graduates with bachelor’s or 

master’s degrees in science, engineering, or 

health 

6
 

 

NSRCG (cont’d.) 

 2008: AY06, AY07 

 Two-stage sample design: school sample (first sample design: school sample (first 

stage) and graduate sample (second stage) 

– Sample sizes: 300 schools and 18,000 graduates 

– For more information, visit 

www.nsf.gov/statistics/srvyrecentgrads 

 Information collected on demographics Information collected on demographics, 

education, employment, etc. 

 Mi d d M il/W b ith Mixed mode: Mail/Web with CATI f ll o ow-up 

 Final response rates 

– 71.4 (unweighted), 69.7 (weighted) 

7
Response Rates (Weighted) 

8
Response Rates (Unweighted) 

9
 

 

R-Indicator 

 Measures similarity or dissimilarity of response 

propensities propensities 

 A measure independent of specific outcome 

variables (similar to response rate) variables (similar to response rate) 

1

N

2 R( )   1 2 (i ) Rˆ

( )   1 2 w ( ˆ)

i i i N1i1 Nˆ

1

1i

ˆ

2

N N1 R

where ρ is an individual response propensity iis an individual response propensity 

10
R-Indicator (cont’d.) 

11
 

Tracking Estimates for Frame Variables 

Percentages Percentages of Minority Minority group

(Hispanic, Black, American Indians)

12
 

Tracking Estimates for Survey Variables

(Hispanic, Black, American Indians)

Unemployment rate estimates for Minority group 

(Hispanic, Black, American Indians)

13
 

Upper Bounds for Bias and Root Mean 

Square Error 

 Upper bounds of bias and root mean square 

errors can be estimated using survey errors can be estimated using survey 

response data: 

ˆ

(1 Rˆ( )) ( ) S y 2 R( )  ) ( ) S y

ˆ ˆ  ˆ ˆ ˆ ˆ

U B y( )  RMSE y U( ) B y( )v y( ) 2ˆ U

where v y

ˆ is a variance estimator of y ( )

ˆ

and

S y

ˆ ˆ ˆ ˆ

( )  nv y ( ) / deff ( ) y

14
 

Upper Bound Bias 

Upper Bound Bias of Unemployment Unemployment Rate

Estimates

15
Upper Bound Root Mean Square Error 

Upper Bound Root Mean Square Error for 

Unemployment Rate Estimates

16
 

 

Results 

 Response rate curves are monotonically

increasing, but in increments of only 1% 

across all key domains (gender and across all key domains (gender and 

race/ethnicity) during the last three weeks 

(weeks 29–31) 

 R-indicator curves are U-shaped, with the 

lowest values between week 13 (RR = 44%) 

and week 22 (58%) overall and for key domains and week 22 (58%) overall and for key domains 

– After week 22, R-indicator values for most domains 

steadily increase but not much 

17
 

 

 

 

Results (cont’d.) 

 The percentage of minority graduates among 

respondents is less than that of the full sample 

at th d f d t ll ti i di ti th t the end of data collection, indicating the 

importance of weighting adjustment 

 Survey estimates for “unemployment rate” estimates for “unemployment rate” of 

Minority group seem steady after week 25 

 The upper The upper bounds of potential of potential bias indicators bias indicators 

(bias and RMSE) for “unemployment rate”

estimates are steadily decreasing, although

th t f d th the rates of decrease over the l t as f k ew weeks 

are minimal 

18
 

 

 

Discussion 

 The last three weeks of 2008 NSRCG data 

collection added, at most, 1% point to the 

response rate response rate ―supporting the data collection supporting the data collection 

closeout decision at week 31 

 Other measures may have supported Other measures may have supported the 

decision made 

– Though R-indicator showing a steady upward trend, 

the slope was very small the slope was very small 

– Key survey estimates stabilized after 20+ weeks 

19
 

 

Discussion (cont’d.) 

 Oh h t er measures may have supported t he 

closeout of the data collection (cont’d.) 

– Bias and RMSE upper bound measures showed 

consistent results with response rates, 

R-indicators, and survey-estimate tracking 

 Importance of tracking various measures of tracking various measures 

during data collection―response rates, Rindicators, frame variables, key survey 

esti tma es, bi i di t bias indicator measures 

 The decision to close out data collection can 

be based on be based on quality measures and quality measures and other 

practical considerations: budget and datadissemination schedule 

20
For More Information 

 Please contact: 

– Donsig Jang 

• djang@mathematica-mpr.com

– Sixia Chen 

• sixia@iastate.edu

– Flora Lan 

• flan@nsf.gov

21

Mathematica® is a registered trademark of Mathematica Policy Research
 

 

 

References 

 Groves, R.M., and E. Peytcheva. “The Impact of 

Nonresponse Rates on Nonresponse Bias.” 

Public Opinion Opinion Quarterly Quarterly, vol. 72 pp , 2008, pp. 1–23.

 Schouten, B., F. Cobben, and J. Bethlehem. 

“Indicators for the Representativeness Indicators for the Representativeness 

of Survey Response.” Survey Methodology, 

vol. 35, 2009, pp. 101–113. 

 Sarndal, C.E. “Three Factors to Signal NonResponse Bias with Applications to Categorical 

Auxiliary Variables Auxiliary Variables. ” International International Statistical Statistical

Review, vol. 79, 2011, pp. 233–254. 

22