The U.S. Census Bureau Mail Return Rate Challenge:

Crowdsourcing to Develop a Hard-to-Count Score

Chandra Erdman and Nancy Bates

U.S. Census Bureau

Federal Committee on Statistical Methodology 

Washington, DC 

November 5, 2013 

Disclaimer: The views expressed on statistical issues are those of the authors only.
The U.S. Census Bureau Return Rate Challenge

“All you need is data and a question. Our data scientists will

provide the answer.”

– Kaggle.com

Our research question: Which statistical model best predicts 2010

Census mail return rates (block-group level)?

Our dataset: 2012 Census Planning Database (PDB)

Product: Updated model-based Hard-to-Count Score

Erdman & Bates (2013) Census Return Rate Challenge 2 / 17
Census Crowdsource Challenge

2009 America COMPETES Act

Contest ran August 31 - November 1, 2012

244 teams and individual competitors

Unanticipated challenges:

Non US citizens

Use of auxiliary datasets

Erdman & Bates (2013) Census Return Rate Challenge 3 / 17
Winning model and HTC score

Software developer from Maryland awarded top monetary prize

(MSE=2.60)

Used random forests and gradient boosting

Model included 342 variables – many from sources external to

Census PDB

How to apply Challenge results toward new model-based HTC score?

Erdman & Bates (2013) Census Return Rate Challenge 4 / 17
HTC-Related Studies

Bruce et al. (2001); Bruce and Robinson (2003)

Original HTC score

Guterbock et al. (2006)

Community attachment theory

Erdman et al. (2013)

Interviewer performance stratification

Erdman & Bates (2013) Census Return Rate Challenge 5 / 17
Model Selection Criteria

1 Restrict to PDB predictors

2 Small number of predictor variables

3 High predictive value (adjusted R2)

4 Low mean square error

5 Model works for both tracts and block-groups

Additional consideration: to include or exclude race/ethnicity

composition as predictors?

Erdman & Bates (2013) Census Return Rate Challenge 6 / 17
Winning Model Predictors

90 percent (308/342) of predictors from census

When ranked by relative influence, 24/25 top predictors from census

(Rank)

Relative Influence

0 10 20 30 40 50

1 2

3

4

(1) Renter

(2) Ages 18−24

(3) Female head of household, no husband

l

l

l

Erdman & Bates (2013) Census Return Rate Challenge 7 / 17 
Comparison of Predictors Across Studies

Table: Overlap Between Top 25 Predictors in Bame (2012) and Erdman et al.

(2013)

Predictor Bame Erdman Bruce Guterbock

Renter occupied units . . . .

Married family households . . .∗ .

Ages 65+ . . + .

Ages 18-24 . . − .

College graduates . . − .

Moved in 2005-2009 . . − .

Ages < 5 . . + .

Ages 5-17 . . + .

Vacant units . . .

Single unit structures . . .∗

Males . . −

Non-Hispanic White . . +

Persons per household . . −

Population Density . . −

Below poverty . . .

Hispanic . .

Non-Hispanic Black . .

Erdman & Bates (2013) Census Return Rate Challenge 8 / 17
Comparison of Predictors Across Studies (Cont.)

Table: Remaining Top 25 Predictors from Bame (2012)

Predictor Bame Erdman Bruce Guterbock

Not high school graduate

Different housing unit 1 year ago

Related child < 6

Ages 25-44

Median household income

Ages 45-64

Female head, no husband

Single person households

.

.

.

.

.

.

.

.

.

.

−

−

−

−

−

−

.

.

.

.

.

.

Erdman & Bates (2013) Census Return Rate Challenge 9 / 17
Comparison of Predictors Across Studies (Cont.)

Table: Remaining Variables from Bruce et al. (2001)

Predictor Bame Erdman Bruce Guterbock

Public assistance

Unemployed

Crowded units

Linguistically isolated households

No phone service

− −

.

.

.

.

.

.

Erdman & Bates (2013) Census Return Rate Challenge 10 / 17
Model Fit Statistics

Table: Comparison of Model Fit Statistics Across Studies and Geographies

Model

Block-group

R-squared MSE

Tract

R-squared MSE

Top 25 Bame (2012) including race

Erdman et al. (2013) including race

Top 25 Bame (2012) excluding race

Erdman et al. (2013) excluding race

Guterbock et al. (2006)

Bruce et al. (2001)

56.27 30.36

56.18 30.37

55.58 30.85

53.89 32.05

51.17 33.86

45.66 37.99

55.03

55.33

54.52

52.06

49.83

45.78

23.09

22.84

23.27

24.72

25.75

28.21

Erdman & Bates (2013) Census Return Rate Challenge 11 / 17
Deciles of Return Rates for Block-Groups in DC 

100%

90%

80%

70%

60%

50%

40%

30%

20%

10%

Fitted Actual

Erdman & Bates (2013) Census Return Rate Challenge 12 / 17 
Trinidad: 37% Ages 18-24;

59% Moved 5 years; 33%

Below poverty; 28% Vacant;

55% Black; 31% white; 87%

renters

Columbia Heights: 43% Hispanic;

36% Other Language; 92% 10+ multiunits; 64% non-family hhds; 85%

renters; 60% moved 5 years

Anacostia: 98% Black; 46% below

poverty; 89% single unit homes; 15%

non-family hhds; 21% moved 5 years;

93% renters

Three HTC Block-Groups in DC

Erdman & Bates (2013) Census Return Rate Challenge 13 / 17
Considerations

Independent variable is mail response; 2020 Census will

have an Internet response option

“Single Unattached Mobiles” (Bates and Mulry, 2011)

64.7 percent of American Community Survey self

response by Internet (Baumgardner, 2013)

In January, 2013, ACS began asking about Internet

connectivity

Erdman & Bates (2013) Census Return Rate Challenge 14 / 17
Summary

Challenge was successful

Winning model was complex but predictors in rank order of

influence proved useful

Accurate predictions with relatively few predictors

Simple HTC score: model fits

First score at this level of geography

Useful for planning and targeted advertising

Erdman & Bates (2013) Census Return Rate Challenge 15 / 17
References

Nancy Bates and Mary H. Mulry.

Using geographic segmentation to understand, predict, and plan for census and survey mail

nonresponse.

Journal of Official Statistics, 27(4):601–618, 2011.

S. Baumgardner.

Self-response check-in rate by mode and segmentation group – june 2013.

Email communication with the author, U.S. Bureau of the Census. September 17, 2013.

Antonio Bruce and J. Gregory Robinson.

Tract Level Planning Database with Census 2000 Data.

U.S. Government Printing Office, Washington, DC, 2007.

Antonio Bruce, J. Gregory Robinson, and Monique V. Sanders.

Hard-to-count scores and broad demographic groups associated with patterns of response

rates in census 2000.

In Proceedings of the Social Statistics Section. American Statistical Association, 2001.

Chandra Erdman, Tamara Adams, and Barbara C. O’Hare.

Development of interviewer performance standards for national surveys.

Draft Paper, 2013.

Thomas M. Guterbock, Ryan A. Hubbard, and Laura M. Hoilan.

Community attachment as a predictor of survey non-response.

Unpublished Paper, 2006.

Erdman & Bates (2013) Census Return Rate Challenge 16 / 17
Contact

chandra.erdman@census.gov 

nancy.a.bates@census.gov 

Erdman & Bates (2013) Census Return Rate Challenge 17 / 17