Efforts to Address Respondent 

Concerns in the American 

Community Survey

Todd Hughes

American Community Survey Office

U.S. Census Bureau

FCSM Research Conference

December 1-3, 2015

Washington, DC
ACS Basics 

• Ongoing monthly survey sent to 

3.5 million addresses per year to 

produce detailed population and 

housing estimates each year

• Designed to produce critical 

information on small areas and 

small population groups 

previously collected on the 

decennial long form

• Supports over 300 known 

Federal Government uses

2 
Components of Respondent Burden and 

Concerns 

• It takes an average of 40 minutes per household to respond to 

the 72 questions included on the survey

• Some questions can be perceived by respondents as sensitive, 

personal, or difficult to answer

• It is unclear to ACS respondents why the Census Bureau needs 

to collect information on some topics

• Response to the ACS is required by law, and the multiple 

contact attempts by mail, telephone and personal visit can be 

perceived by some respondents as harassment

3 
Research to Address Respondent 

Concerns

4 

Topics of Research Underway

• Using Administrative

Records

• Asking Less Frequently or of 

Fewer Respondents

• Testing Messaging in Mail 

Materials

• Promoting Awareness of Data 

Uses

• Modifying Survey Questions • Reducing Follow-Up Attempts

See http://www.census.gov/programs-surveys/acs/operations-andadministration/2015-16-survey-enhancements.html
Using Administrative Records

We only want to ask households once for information

already reported to the government, potentially allowing 

us to remove some questions from the ACS.

• Identifying administrative and commercial data sources

• Evaluating coverage and quality of other data sources

• Preliminary results being released on a flow basis by topic 

from September 2015 through October 2016

• Implications to topic-specific estimates

• Preliminary results will be released on a flow basis by 

topic from March 2016 through March 2017

5 
Testing Messaging in Mail Materials and 

Promoting Awareness of Data Uses

 Field Testing Revisions to the Mail Materials 

 Tests conducted in Spring 2015 to revise mail methods and timing

 Envelope Mandatory Messaging Test

 Testing the removal of mandatory messages from the outside of the ACS 

envelopes in May 2015 (preliminary report released Sept. 4, 2015)

 Mandatory Messaging Test

 Testing more comprehensive changes throughout the mail packages in 

September 2015 to soften references to mandatory and explain data 

uses (preliminary report planned for December 2015)

 Why We Ask Brochure Test

 Testing a graphic-based insert with the paper questionnaire in 

November 2015 to provide information on why we ask the questions

6 
Results of the Envelope Mandatory 

Messaging Test

 Split-ballot experiment embedded within May 2015 production, where 

24,000 cases eliminated the phrase “YOUR RESPONSE IS REQUIRED BY 

LAW” from the initial mail package envelope and the paper questionnaire 

package envelope 

 Eliminating this phrase does significantly lower the self-response return 

rate by 5.4 percentage points (this rate is before the start of computerassisted telephone interviewing and it pushes additional cases into the 

more expensive follow-up modes)

 We estimate the preliminary cost impact of eliminating mandatory 

messages from the envelopes in the manner we tested to be an increase 

in the annual costs of the survey by roughly $9.5 million

 See full report at: 

http://www.census.gov/content/dam/Census/library/workingpapers/2015/acs/2015_Barth_01.pdf

7 
Evaluating Modifications 

to Survey Questions

We are researching the possibility wording questions 

differently to reduce some of the difficulty for 

respondents in completing the survey.

• 2016 Content Test – field test of changes to many questions 

that were proposed by Federal agencies in 2013

• 2015-2016 Lab Testing—work with Federal agencies to 

develop changes to make questions less difficult, and 

evaluate those changes in a lab setting

8 
2016 Content Test Topics

9 

• Telephone Service 

• Computer and Internet

• Relationship 

• Hispanic Origin and Race

• Health Insurance

• Journey to Work: 

 Commute to Work 

• Number of Weeks 

Worked

• Class of Worker

• Industry and Occupation

• Retirement Income

• Health Insurance

Premiums and Subsidies

• Journey to Work: Time 

Leave for Work
Evaluating Methods to Ask Questions Less 

Frequently or of Fewer Respondents 

The current design of the ACS asks all of the survey 

questions from all sampled households every year. 

Alternative survey designs might reduce that burden.

• Feasibility report issued September 30, 2015 identified some 

opportunities to include some questions periodically, rather 

than asking every question every year, or to ask some 

questions of only a subset of the ACS sample.

• The report also evaluated methods like “matrix sampling” 

and a hybrid approach using administrative records to reduce 

the burden on respondents.

10
ACS 

Questionnaire

Topics

A E

B F

C G 

D H 

ACS 

Questionnaire

Topics

A _ 

B F

_ G

D H 

ACS 

Questionnaire

Topics

A E

B F

_ G

D H 

ACS 

Questionnaire

Topics

A _

B F

C G 

D H 

Year 1

Form Version 1

In this example, Topic C is needed only every three years, while Topic E is needed every other 

year. One version of the questionnaire is used for all households in sample each year.

Illustration of Option 1: Periodic Inclusion

Used in full sample Used in full sample Used in full sample Used in full sample

Year 2 

Form Version 1

Year 3 

Form Version 1

Year 4 

Form Version 1
ACS 

Questionnaire

Topics

A E

B F

C G 

D H 

ACS 

Questionnaire

Topics

A E 

_ F

C G

D H 

ACS 

Questionnaire

Topics

A E

_ F

C G 

_ H 

Form Version 1

In this example, Topic B is needed only at the state level, while Topic D is needed at the county level.

Illustration of Option 2: Subsampling

Form Version 2 Form Version 3

Only the portion of the full sample that is needed to produce estimates at the necessary geographic 

level receives the corresponding form version. Since Topic B requires less sample than Topic D to 

produce only state level estimates, form version 1 is used for a smaller set of households.

1 

1

2

2

2 2

2

3

3
ACS 

Questionnaire

Topics

A _

B _ 

C _ 

D _

ACS 

Questionnaire

Topics

A E 

B F

_ _

_ _

ACS 

Questionnaire

Topics

A _

B _

_ G 

_ H 

Form Version 1

In this example, topics are assigned to form versions in a partially overlapping manner.

Illustration of Option 3: Matrix Sampling

Form Version 2 Form Version 3

Either only a portion of the full sample receives each form version and other statistical tools help 

to mitigate the impact of the missing data, or the total sample is increased to achieve the 

reliability needed for each topic.

1 

2

1

2

3 3 

1

2

3

1 

1

1

1

1 1 

2

2 2

2 2 3 3 3 

2 2 2 3 3 3

2 3 3 3

1 

1

1

OR
ACS 

Questionnaire

Topics

A E

B F

C _ 

D H 

ACS 

Questionnaire

Topics

A E

B F

C G 

D H

Form Version 1

In this example, Topic G has an alternative data source with good quality and coverage 

for most geographic areas that can be used directly in place of collecting the data on 

the questionnaire in those areas.

Illustration of Option 4: Administrative Records Hybrid

Form Version 2

Areas with good coverage for the alternative data source receive form version 1, while 

areas without good coverage for the alternative data source receive form version 2.

1 

1

1

1

2 2

1

2 

2

Alternative 

Data 

Source for 

Topic G

+ 
Assessing Each Option

A Census Bureau team assessed each option according to a set of 

factors that demonstrate the feasibility and impact of the method. 

These assessments were based on the professional judgment of the 

team members, and not on empirical criteria. These factors are: 

1. Operational and processing complexity 

2. Impact on the accuracy of the data 

3. Impact on data availability for small geographies and groups 

4. Estimated reduction in respondent burden 

5. Impact on richness of the data products 

6. Assessment of additional costs and resources required 

The report is available at: http://www.census.gov/content/dam/Census/programssurveys/acs/operations-and-administration/2015-16-surveyenhancements/Reducing_Burden_ACS_Feasibility_Assessment.pdf

15
The Team’s Recommendations

 Pursue Periodic Inclusion: The team recommends only periodically 

including any questions where the frequency and geographic needs for the 

data can be supported through asking some questions less frequently than 

every year. 

 Pursue options for incorporating Administrative Records. Using 

administrative records either as a substitute for survey data collection for 

some topics included in the ACS, or via a hybrid approach with partial 

survey data collection, could significantly reduce respondent burden. 

When compared with subsampling and matrix sampling, using 

administrative records also seems to involve fewer potential undesirable 

impacts. 

 Seek additional input on efficient possibilities for Matrix Sampling or 

topical subsampling. These options present potentially costly impacts on 

survey operations and the accuracy and richness of survey estimates. 

Therefore, the Census Bureau is seeking input that may help to develop 

research into efficient and effective designs for matrix sampling.

16
Reducing Follow-Up Contact Attempts

 In 2013 changes were implemented to reduce the 

calls made in the CATI operation

 In August 2015 a pilot test was conducted to reduce

the number of contacts made in CAPI

 The pilot employed stopping rules that also 

considered the previous contact attempts made 

by mail and CATI

 Targeting national deployment in May 2016

17
Projects Addressing Respondent Concerns 

and Reducing Respondent Burden

Fiscal Year 

2015 Fiscal Year 2016

Projects Q2 Q3 Q4 Q1 Q2 Q3 Q4 Comments 

Envelope Mandatory Messaging Test Possible implementation in 2016

Additional Testing of Mandatory Messaging Possible implementation in 2016

Reducing Personal Visit Contact Attempts Conduct pilot in August 2015, and implement in 

Spring 2016

Matrix Sampling: Operational and Statistical 

Assessment

Initial assessment complete in September 2015

Field Test of Question Wording Changes 

Recommended by Federal Agencies

To assess question changes proposed for 2017 

and 2019

Testing Changes to Question Wording to Reduce 

Difficulty and Sensitivity

Qualitative testing ongoing starting in 2015

Matching Coverage and Quality Assessments of 

Administrative Records

Coverage and quality assessments completed by 

September 2016

External Expert Consultations Providing input on reducing respondent 

concerns in general, and specific projects above

18
Thank you!

Todd.R.Hughes@census.gov 

Any views expressed are those of the author and not 

necessarily those of the U.S. Census Bureau. 

19