Mandatory Messaging in the 

American Community Survey 

Elizabeth Poehler, Dorothy Barth, and Todd Hughes 

U.S. Census Bureau 

Elizabeth.Poehler@census.gov 

2015 FCSM Research Conference 

1 

Any views expressed are those of the author(s) and not necessarily those of 

the U.S. Census Bureau.
American Community Survey 

(ACS) Background 

 Monthly survey sent to 3.5 million addresses per 

year to collect population and housing data 

 Mandatory survey 

 3-month multi-mode sequential data collection for 

each monthly panel 

 Month 1: Self-response (Internet and Mail) 

 Month 2: Telephone interviews with non-respondents 

(CATI) 

 Month 3: In-person interviews with a sample of nonrespondents (CAPI) 

2 
ACS Sample Background 

 Annual sample, split into 12 monthly panels of 

approximately 295,000 addresses 

 EaIh マoミth’s saマple is fuヴtheヴ suHdi┗ided 

into 24 equally representative groups of 

12,000 addresses 

 Allows testing to be conducted using production 

sample 

3 
Issue to Address 

 We are required by law to inform respondents 

that the survey is mandatory. 

 However, respondents and stakeholders of the 

ACS have expressed concerns about the 

prominent nature of mandatory messages on 

the mail materials 

 Conducted two tests to study the impact of 

reducing the prevalence of mandatory 

messages on the ACS. 

4 
Research Questions

 Test 1: What is the impact on response rates 

of removing the mandatory message from 

the envelopes for the ACS? 

 Test 2: What is the impact on response of 

modifying mandatory messages in the mail 

materials for the ACS? 

5 
Previous Research 

 1993 National Census Test 

 Coマpaヴed eミ┗elopes that iミIluded the phヴase さYouヴ 

Respoミse Is Reケuiヴed H┞ La┘ざ to eミ┗elopes ┘ithout 

the phrase 

 Difference in mail response rate of 9.2 percentage 

points (statistically significant) 

 2013 Messaging and Mail Package Assessment Research 

 Focus groups and one-on-one interviews 

 さReケuiヴed H┞ La┘ざ siミgle マost effeIti┗e マessage

 Most participants said mandatory messaging would 

motivate them to respond 

 Some bristled at the messaging 

6 
ACS Mail Strategy - May 

Mailout Description of Materials

Pre-Notice Letter 

 

Letter excludes mandatory language

Multi-Lingual Informational Brochure includes mandatory language

Initial Mailing Package Letter and FAQ Brochure include mandatory language

Internet Instruction Card excludes mandatory language 

First Reminder Postcard Excludes mandatory language 

Paper Questionnaire 

Package 

Letter, FAQ Brochure, and Instruction Guide include mandatory 

language.

Paper Questionnaire, Return Envelope, and Internet Instruction 

Card exclude mandatory language. 

Second Reminder 

Postcard Includes mandatory language 

Additional Postcard Includes mandatory language

7 
Test 1: Envelope Mandatory 

Messaging Test 

What is the impact on response rates of 

removing the mandatory message from the 

envelopes for the ACS? 

8 
Envelope Mandatory Messaging Test 

 Goal: Study impact of removing mandatory 

message from envelopes 

 Conducted in May 2015 

 Treatment 1: Control Materials with 

Mandatory Message on Envelopes 

 Treatment 2: Mandatory Message Removed 

from Envelopes 

 Sample Size: 24,000 addresses in each 

treatment 

9 
Mandatory Message on 

Control- Initial Envelopes 

10
Experimental - Initial Envelopes 

11
Mandatory Message on 

Control – Mail Package Envelopes 

12
Experimental – Mail Package Envelopes 

13
Envelope Mandatory Messaging 

Test Results 

Total Self-Response Return Rates (Internet & Mail combined)

Point in Data Collection Cycle

Control 

(Includes 

Mandatory)

Test 

(Removed 

Mandatory) 

Difference Significant? 

Before First Reminder Postcard 3.0 (0.2) 2.6 (0.1) 0.4 (0.2) Yes 

Before Paper Questionnaire Package 22.6 (0.4) 18.7 (0.3) 3.9 (0.5) Yes 

Before CATI 42.5 (0.5) 37.1 (0.4) 5.4 (0.5) Yes 

• Standard errors in parentheses. 

• Return Rates exclude Undeliverable as Addressed (UAAs) but include out-ofscope address in the universe. 

• Comparisons of Internet response and Mail response were also significant at 

the alpha=0.1 level. 

14
Envelope Mandatory Messaging 

Test Results 

Final Response Rate

Control 

(Includes 

Mandatory) 

Test 

(Removed 

Mandatory) 

Difference Significant

? 

Final Overall Response 96.3* (0.3) 95.6* (0.3) 0.7 (0.3) Yes

Distribution of Final Response by Mode

Internet 35.7 (0.5) 32.7 (0.4) 3.0 (0.7) Yes 

Mail 23.8 (0.4) 21.3 (0.4) 2.5 (0.6) Yes 

CATI 3.8 (0.2) 4.7 (0.2) -0.9 (0.2) Yes 

CAPI 33.1 (0.6) 36.8 (0.6) -3.7 (0.8) Yes 

• *Rates may not sum to total due to rounding. 

• Standard error in parentheses. 

• Partial interviews and vacant units are included as responses. 

15
Envelope Mandatory Messaging 

Test Results 

 Because the self-response rate in the test 

treatment was lower, additional paper 

questionnaires were mailed to nonrespondents, 

and the CATI and CAPI workloads were higher 

than the control panel. 

 So, while the overall response rate drop of 0.7 

percentages point might not seem to be large, it 

is estimated that cost impact of eliminating 

mandatory messages from the envelopes would 

increase the cost of the ACS by $9.5 million, 

annually. 

16
Test 2: Additional 

Mandatory Messaging Test 

What is the impact on response of 

modifying mandatory messages in the mail 

materials for the ACS? 

17
ACS Mail Strategy - September 

Mailout Description of Materials

Pre-Notice Letter NOT SENT 

Initial Mailing Package 

Letter, Multi-Lingual Brochure, and FAQ Brochure include 

mandatory language

Internet Instruction Card excludes mandatory language 

First Reminder Letter Includes mandatory language 

Paper Questionnaire 

Package 

Letter, FAQ Brochure, and Instruction Guide include mandatory 

language.

Paper Questionnaire, Return Envelope, and Internet Instruction 

Card exclude mandatory language. 

Second Reminder 

Postcard Includes mandatory language 

Additional Postcard Includes mandatory language

18
Additional Mandatory Messaging Test 

 Goal: Study impact of removing or modifying 

mandatory message from broader set of mail 

materials 

 Conducted in September 2015

 5 Treatments tested softening or removing 

mandatory messaging and tested a revised 

design 

 Sample Size: 12,000 addresses in each 

treatment 

19
Additional Mandatory Messaging 

Test Treatments 

 Control 

 No change to materials 

 Softened Control 

 Mandatory removed from initial letter, mail package letter, 

postcards, and envelopes 

 Mandatory kept in FAQ brochure, reminder letter, 

instruction guide 

20
Additional Mandatory Messaging 

Test Treatments (Con’t) 

 Revised Design 

 Redesigミed eミ┗elopes, use of Hold letteヴiミg, highlight Ho┝es, さOpeミ 

Iママediatel┞ざ

 Strong mandatory language 

 Softened Revised Design 

 Revised design used 

 Mandatory Removed from postcards and envelopes 

 Mandatory Softened in letters (plain text) 

 Minimal Revised Design 

 Revised design used 

 Mandatory removed in all materials except initial letter

 Mandatory in initial letter on back of page, in small font 

21
Softened Control Envelope 

22
Redesigned Envelope 

23
Mandatory Message in Control Letters 

The Census Bureau chose your address, 

not you personally, as part of a randomly 

selected sample. You are required by 

U.S. law to respond to this survey. The 

Census Bureau is required by U.S. law to 

keep your answers confidential. The 

enclosed brochures answer frequently 

asked questions about the survey. 

24
Revised Design Letter 

25

The revised design features: 

• Bold text 

• Box around the internet URL 

• Bulleted lists 

• Census Bureau logo 

• Text that appeals to the 

ヴespoミdeミt’s seミse of 

community 
Minimal Revised Design Letter 

26

• This design includes the 

revised design features. 

• Mandatory messaging is on 

the back of the letter in small 

font (below). 
Additional Mandatory Messaging 

Test 

 Field work concluded in December. 

 Results indicated that reducing the frequency 

and visibility of mandatory messages reduces 

response rates. 

27
Conclusions 

 Removing the Mandatory Messaging from ACS 

envelopes reduces self-response rates, 

resulting in a reduction in overall response 

rates 

 Reducing the reducing the frequency and 

visibility of mandatory messages in the mail 

materials reduces response rates. 

 Revisions to the design of the mail materials 

had a positive impact on response. 

28
Report References 

 Preliminary Results of the Envelope 

Mandatory Messaging Test are available here 

(final results forthcoming): 

http://www.census.gov/library/workingpapers/2015/acs/2015_Barth_01.html

 Results of the Additional Mandatory 

Messaging Test are available here: 

http://www.census.gov/library/workingpapers/2016/acs/2016_Oliver_01.html

29