Respondent-Centered 

Establishment Survey Design 

Principles: An Overview

Melissa A. Cidade

October 25, 2023

Annual Conference of the Federal Committee on Statistical Methodology (FCSM)

College Park, MD

1

Any opinions and conclusions expressed herein are those of the author(s) and do not reflect the views of the 

U.S. Census Bureau. The Census Bureau has reviewed this data product to ensure appropriate access, use, and 

disclosure avoidance protection of the confidential source data (Project No. P-7529180, Disclosure Review Board 

(DRB) approval number: CBDRB-FY23-ESMD010-038).
What is respondent-centered design?

• Wilson and Dickinson (2022): put the respondent in the driver’s seat

• Smyth (2016): cognition, usability, and a holistic design model

• Willimack and Snijkers (2013: 39): “Businesses are different!”

2
Why do we want to be respondent-centered?

• Minimize measurement error

• Reduce non-response bias

• Reduce response burden

• Inform policy decisions

3
Interagency Collaboration

• Energy 

Information 

Administration, 

Department of 

Energy

Sarah Grady, Heather 

Ridolfo, and 

Benjamin Messer

• National 

Agricultural 

Statistics Service, 

Department of 

Agriculture

Kenny Herrell

• Census Bureau, 

Department of 

Commerce

Temika Holland, 

Kristin Stettler, and 

Melissa Cidade

• National Center 

for Education 

Statistics, 

Department of 

Education

Maura Spiegelman

4
Four Guiding Principles:

Gain as much information 

as possible about the 

response unit and the 

respondent prior to fielding 

the survey, and use that 

information to inform 

contact strategies;

Contact knowledgably

Consider the unique 

response processes in place 

for establishment surveys, 

particularly respondents’ 

authority, capacity, and 

motivations to respond;

Question carefully

Layout instruments to be 

intuitive with minimal 

support to minimize 

response errors; and

Design intuitively

Tailor data collection to 

respondents’ needs, 

including questionnaire 

delegation/sharing, 

response mode selection, 

edit and content checks, 

and the use of previously 

reported data, 

administrative records, and 

third party data.

Collect adaptively

5
Principle 1: Contact knowledgably

What it means:

• Gain as much information as 

possible about the response unit and respondent prior to fielding 

the survey, and use that 

information to inform survey 

design

Why it matters:

• Davis and Pihama (2009): staff turnover predicts response status

• Gravem et al. (2011): mismatches between survey questions and data 

availability were a leading cause of 

perceived response burden

• Snijkers et al. (2023): respondent communications predicts response 

status

6
Principle 1: Contact knowledgably

Best Practices:

• Consider the design implications 

of your response unit

• Tailor your communication 

strategies and messages to 

respondents

Research Methodologies:

• Record Keeping Studies: how 

companies keep their data

• Contact Tracing Studies: 

Identifying best contact

7
Principle 2: Question carefully

What it means:

• Hybrid Response Model: Different 

cognitive response process for 

establishment surveys (Willimack

and Nichols 2010)

• Business Survey Response Model: Different social response process 

for establishment surveys 

(Willimack and Snijkers 2013)

Why it matters:

• Janik and Kohaut (2011): 

establishment response process is 

predictive of response status

• Tuttle, Morrison, and Willimack

(2010): iterative, respondent- centered design produces a better 

survey

8
Principle 2: Question carefully

Best Practices:

• Keep instructions brief and use 

sparingly

• Tailor wording where appropriate

• Address privacy concerns

Research Methodologies:

• Cognitive Testing: identifies 

question issues

• Early-Stage Scoping: combines 

cognitive testing and exploratory 

interviewing

• Pilot Testing: uses qualitative and quantitative methods to test 

questions

9
Principle 3: Design intuitively

What it means:

• Usability Performance Metrics 

(Geisen and Romano Bergstrom 

2017; ISO/IEC TR 9126-4: 2004):

• Effectiveness (Accuracy)

• Efficiency

• Satisfaction

Why it matters:

• Romano Bergstrom, Erdman, and 

Lakhe (2016): placement of 

buttons impacts instrument 

performance

• Nichols, Olmsted-Hawala, Holland, 

and Anderson Riemer (2020): 

usability testing encompasses 

many methodologies

10
Principle 3: Design intuitively

Best Practices:

• Instrument flexibility and 

adaptability is a must

• Use design features to cue 

response

Research Methodologies:

• Paradata analyses – data 

generated by the instrument

• Usability testing – task-oriented 

interviews

• Debriefing interviews –

retrospective reporting

11
Principle 4: Collect adaptively

What it means:

• All of the activities included in 

the actual collection of survey 

data, including (but not limited 

to):

• Contact and collection mode(s)

• Contact and collection materials

• New and emerging collection 

techniques

Why it matters:

• Rodhouse and Ott 

(2022): Previously reported data 

lowered perceived burden

• Snijkers et al. (2023): 

respondent communications 

predict response status

12
Principle 4: Collect adaptively

Best Practices:

• Incentives are more complicated for 

establishment surveys

• Consider multiple methods of 

response within modes – spreadsheets, system-to-system data reporting, third party data

Research Methodologies:

• Classic experimental design – changing one factor of a research plan 

to track its impact on performance

• Response Analysis Survey – a survey sent after completion to estimate 

burden

• Small-scale technology pilot – working closely with technical staff to 

integrate new technologies

13
Annual Integrated Economic 

Survey (AIES)

14
NAS Panel of Experts and the ABSS

• Panel of Experts

• Factors:

• Budget pressures

• Declining response rates

• Increasing timeliness and granularity

• Increasing competition

• ABSS to include:

• Integrated and harmonized content

• Revamped sampling strategy

• Coordinated collection strategy

• Inclusion of alternate response methods

• Single dissemination platform

• Multi-sector and subnational estimates

15
16 For more information on AIES, visit: https://www.census.gov/programs-surveys/aies.html
Application 

of the 

Principles

Respondent-centered Response Options and the AIES

Rebecca Hutchinson

Iterative Instrument Usability Testing and the AIES

Rebecca Keegan

Combining Content: Considerations for Impactful Change

Heidi St.Onge

17
Combining Content: 

Considerations for Impactful 

Change

Heidi M. St.Onge

October 25, 2023

Annual Conference of the Federal Committee on Statistical Methodology (FCSM)

College Park, MD

18

Any opinions and conclusions expressed herein are those of the author(s) and do not reflect the views of the 

U.S. Census Bureau. The Census Bureau has reviewed this data product to ensure appropriate access, use, and 

disclosure avoidance protection of the confidential source data (Project No. P-7529180, Disclosure Review Board 

(DRB) approval number: CBDRB-FY23-ESMD010-039).
19

Contact Knowledgably

Best practices:

• Consider the design implications 

of your response unit

• Tailor your communication 

strategies and messages to 

respondents

• Design the survey and materials 

based on prior knowledge of the 

respondent and/or response 

unit.

What it means:
What did we consider?

• Structure

• What do we know about 

business?

• How can we leverage?

• Records

• Types?

• Storage?

• Access?

• Changes?

What we did:

Two-phased study on response 

units:

• Round 1: In-depth interviews 

focused on how kept records

• Round 2: Novel card-sort 

methodology on data 

accessibility

20

Contact Knowledgably N = 59
Findings:

• Record Keeping Study: 

• Industry classification is 

challenging for respondents

• Businesses varied in their 

operating units

• Consolidated financial records act 

as an “anchor” for other data.

• Accessibility Study: 

• Respondents struggled with their 

NAICS classification.

• Company-level data are the most 

accessible.

• State-level data were not 

accessible to respondents. 

21

Contact Knowledgably
How did we implement?

• Company-level data is collected separately 

from other “levels”

• Do not collect data at State level

• Implemented means to collect at multiple 

levels based on company records.

• NAICS/Industry was given additional 

attention

22

Contact Knowledgably
Question Carefully

What it means:

• Establishment surveys are 

completed by humans, they are 

using some of the same 

cognitive response processes, 

and records.

23

Best Practices:

• Keep instructions brief and use 

sparingly

• Tailor wording where 

appropriate
What did we consider?

• Pulling together all the 

appropriate people

• Stakeholders

• SMEs

• Decision making in a large 

group

• 1000+ items

24

Integration of Seven Annual Economic Surveys 

ACES—Annual Capital 

Expenditures Survey

ARTS—Annual 

Retail Trade Survey

AWTS—Annual 

Wholesale Trade 

Survey

ASM—Annual Survey 

of Manufactures

SAS—Service 

Annual Survey

COS—Report 

of Organization

M3UFO—

Manufactures' 

Unfilled Orders 

Survey

ACES

ARTS

AWTS

ASM

COS

M3UFO

SAS

AIES

Question Carefully
REVENUE EXAMPLE

25

*Agents, Brokers, Representatives and Electronic Markets

** Added establishments

RETAIL

What were the 

total sales of 

merchandise and 

other operating 

receipts for this 

firm in 2021?

WHOLESALE

What were this 

firm’s TOTAL sales 

and operating 

revenue in 2021?*

What were this 

firm’s sales and 

other operating 

receipts in 2021?

SERVICES

What were the 

revenues for this 

firm in 2021?

MANUFACTURING

What was the total 

value of products 

shipped and other 

receipts for this 

establishment?

What were the 

sales, shipments, 

receipts, or 

revenue?**

AIES

What were the 

total sales, 

shipments, 

receipts, or 

revenue in 202X?

Question Carefully
What we did:

• Phase 1 of Pilot

• Putting in one 

place to get 

reactions from Rs

26

Question Carefully
Findings:

• Complexity of certain constructs 

inhibited harmonization

• Content sometimes was 

ambiguous

• Duplicative content – higher 

burden

• Duplication of pre-listed 

information

Design Implications:

• Certain instructions/questions 

were kept separate.

• Updating language for multiple- industries

• Auto-sum functionality

• Included a step to “verify 

locations” in instrument

27

Question Carefully
Iterative Instrument Usability 

Testing and the AIES

Rebecca Keegan

Annual Conference of the Federal Committee on Statistical Methodology 

(FCSM)

October 25, 2023

College Park, MD

Any opinions and conclusions expressed herein are those of the author(s) and do not reflect the views of the 

U.S. Census Bureau. The Census Bureau has reviewed this data product to ensure appropriate access, use, and 

disclosure avoidance protection of the confidential source data (Project No. P-7529180, Disclosure Review Board 

(DRB) approval number: CBDRB-FY23-ESMD009-001).
Four Guiding Principles:

Gain as much information as 

possible about the response 

unit and the respondent 

prior to fielding the survey, 

and use that information to 

inform contact strategies;

Contact knowledgably

Consider the unique 

response processes in place 

for establishment surveys, 

particularly respondents’ 

authority, capacity, and 

motivations to respond;

Question carefully

Layout instruments to be 

intuitive with minimal 

support to minimize 

response errors; and

Design intuitively

Tailor data collection to 

respondents’ needs, 

including questionnaire 

delegation/sharing, response 

mode selection, edit and 

content checks, and the use 

of previously reported data, 

administrative records, and 

third party data.

Collect adaptively

29
Principle 3: Design intuitively

Best Practices:

• Instrument flexibility is a must

• Use design features to cue 

response

Research Methodologies:

• Usability testing – task-oriented 

interviews

• Debriefing interviews –

retrospective reporting

• Paradata analyses – data 

generated by the instrument

30
31

Fall, 2019 

Fall, 2021; Winter, 2022

Fall, 2022 

Winter, 2022

Winter, 2023

Fall, 2023 

Usability Testing 

Spring 2021

Record Keeping Study

Data Accessibility Study

AIES Phase I Pilot 

Survey Structure Study

Usability- Vignette Testing

AIES Phase II Pilot 

AIES Phase III Pilot (aka The Dress Rehearsal )

Fall, 2023

 
What did we consider?

• Structure

oEstablishment versus company 

data organization

• Records

oDoes the structure map to their 

records?

• Response Mode 

oSpreadsheet or traditional pageby-page

What we did:

• Cognitive Testing

o2 Rounds

o N = 39

• Participants viewed: 

• Early mockups of the instrument

• A consolidated list of the question 

topics within each module 

32

Design Intuitively

N = 39

Survey Structure 
33
Findings:

• Structure

o Separating company totals from 

establishment data made sense

• Record Keeping

o Want survey question previews 

• Response Mode 

oSpreadsheet better for detailed 

data 

oPage-by-page better for data that 

is easy to obtain

34

Design Intuitively

Survey Structure 
How did we implement?

• Structure 

o Company totals collected separately from establishment data

• Record Keeping

o Survey question previews made available

• Flexible Response Mode 

o Spreadsheet for detailed data 

o Page-by-page company level 

35

Design Intuitively 

Survey Structure 
36

Fall, 2019 

Fall, 2021; Winter, 2022

Fall, 2022 

Winter, 2022

Winter, 2023

Fall, 2023 

Usability Testing 

Spring 2021

Record Keeping Study

Data Accessibility Study

AIES Phase I Pilot 

Survey Structure Study

Usability-Vignette Testing

AIES Phase II Pilot 

AIES Phase III Pilot

Fall, 2023

 
37

Fall, 2019 

Fall, 2021; Winter, 2022

Fall, 2022 

Winter, 2022

Winter, 2023

Fall, 2023 

Usability Testing 

Spring 2021

Record Keeping Study

Data Accessibility Study

AIES Phase I Pilot 

Survey Structure Study

Usability-Vignette Testing

AIES Phase II Pilot 

AIES Phase III Pilot

Fall, 2023

 
Design intuitively: Usability Testing

• Method of assessing how functional a website is. 

• Respondents given tasks designed to ensure they interact with key features

• Researchers draw conclusions about the website’s:

o Layout

o Navigation

o Functionality

• Using three measures of evaluation:

o Effectiveness (Accuracy)

o Efficiency

o Satisfaction

38

Usability Testing:
What did we consider?

• Spreadsheet Design

oConcept: Answering by 

Establishment or Industry

oNavigation

• Features:

oAuto-summing; Company 

totals reference

39

Design Intuitively

• Low Fidelity Usability Testing 

oN = 10

• Assigned respondents 1 of 3 proxy 

companies & a ledger

• Task topics:

o Navigation; Auto-summing

o Concept: Answering by 

Establishment or Industry

What we did:

Usability- Vignettes 
Assigned a Company: 

“There is a section of the survey 

that asks about the value of each 

product that you sell at the Census 

Cat Company. Please indicate the 

value of your canned cat food.”

40

Task: 

Census Cat Company

• Complex Company

• 12 establishments

• 5 industries

• 2 manufacturing

• 3 nonmanufacturing 

(R, W, S)

Ledger:

*all data fictional

Usability-Vignette Testing 
41

Design Intuitively

• Spreadsheet Design

o Concept: Answering by Establishment or Industry 

 Well liked

 Difficult to understand

o Amount of content overwhelming

o Familiar functionality for navigation and manipulation desired 

 E.g, ctrl+f, copy/paste, filter and sort, hide and freeze.

• Features helpful, but not intuitive

o Auto-summing; Company totals reference

What we found:

Usability- Vignettes 
42

Design Intuitively

• Spreadsheet design:

o Incorporated establishment or industry choice into Pilot II for testing

o Content parsed out

• Company totals separated from establishment data 

o Familiar functionality incorporated

• E.g, ctrl+f, copy/paste, filter and sort, hide and freeze

• Features:

oAuto-summing & company totals reference given more attention in 

Pilot II and production instrument

How did we implement?

Usability- Vignettes 
43

Fall, 2019 

Fall, 2021; Winter, 2022

Fall, 2022 

Winter, 2022

Winter, 2023

Fall, 2023 

Usability Testing 

Spring 2021

Record Keeping Study

Data Accessibility Study

AIES Phase I Pilot 

Survey Structure Study

Usability-Vignette Testing

AIES Phase II Pilot 

AIES Phase III Pilot

Fall, 2023

 
44

, 

Fall, 2021; Winter, 2022

Fall, 2022 

Winter, 2022

Winter, 2023

Fall, 2023 

Usability Testing 

Spring 2021

Data Accessibility Study

AIES Phase I Pilot 

Survey Structure Study

Usability - Vignette Testing

AIES Phase II Pilot 

AIES Phase III Pilot

Fall, 2023

, 

Fall, 2021; Winter, 2022

Fall, 2022 

Winter, 2022

Winter, 2023

Fall, 2023 

Spring 2021

Fall, 2023 
What will we consider?

• End-to-end usability testing of 

instrument prototype

• Topics: Navigation, layout, 

edits, satisfaction

What we will do:

• Respondents given tasks 

designed to understand how 

they interact with the web 

instrument

o N ≈ 30

o Interviews happening, now!

45

N = 59

Design Intuitively

Usability
Respondent-centered Response 

Options and the AIES

Rebecca J. Hutchinson

October 25, 2023

Annual Conference of the Federal Committee on Statistical Methodology (FCSM)

College Park, MD

46

Any opinions and conclusions expressed herein are those of the author(s) and do not reflect the views of the 

U.S. Census Bureau. The Census Bureau has reviewed this data product to ensure appropriate access, use, and 

disclosure avoidance protection of the confidential source data (Project No. P-7529180, Disclosure Review Board 

(DRB) approval number: CBDRB-FY24-ESMD011- 001).
Four Guiding Principles:

Gain as much information 

as possible about the 

response unit and the 

respondent prior to fielding 

the survey, and use that 

information to inform 

contact strategies;

Contact knowledgably

Consider the unique 

response processes in place 

for establishment surveys, 

particularly respondents’ 

authority, capacity, and 

motivations to respond;

Question carefully

Layout instruments to be 

intuitive with minimal 

support to minimize 

response errors; and

Design intuitively

Tailor data collection to 

respondents’ needs, 

including questionnaire 

delegation/sharing, 

response mode selection, 

edit and content checks, 

and the use of previously 

reported data, 

administrative records, and 

third party data.

Collect adaptively

47
Collect Adaptively

48

What it means

• Use new and emerging response 

options to make the collection 

experience more automated and less 

burdensome

• Supplement or replace traditional 

survey collection

What did we consider?

• What are non-traditional methods 

of response that might be useful?

• Do these data fulfill the traditional 

AIES collection instrument ask?
Alternative Data Sources

49

Benefits

• More timely and granular data

• Reduced respondent burden

• Mitigated survey nonresponse

Challenges

• Cost

• Processing

• Storage

• Unstructured Data
Administrative Data

50

Obtain data received by other government agencies for nonstatistical purposes or through special agreements or publicly 

available feeds

• Benefits

• Repurposes data already submitted for other use

• Long history of use at Census Bureau 

• Constraints

• Can be available on a lag

• Can be difficult to align data to survey items

COMPANY

GOVERNMENT 

AGENCY

CENSUS
Third-Party Data

51

Purchase data from a vendor using acquisitions process

• Build upon success of using point-of-sale retail data in our retail and 

construction indicator programs 

• Benefits

• More granular

• More timely

• Constraints

• Data items can be limited

• Limited or no availability in some sectors

COMPANY

VENDOR

CENSUS
Direct Company Feed

52

Companies push financial feeds rich in data to Census either 

directly or via a third-party platform

• Benefits

• Makes use of existing company feeds or files

• Can pass high volume of data along to Census

• Constraints

• Need to align items in feeds to survey items

• Currently limited to a small number of companies

• Requires data cleaning

COMPANY

CENSUS

3P 

PLATFORM
How do we QA other 

data sources?

4 C’s

OF

QUALITY REVIEW

Third Party Data
Thank you!

54

Contact 

Knowledgably

Question 

Carefully

Design Intuitively Collect Adaptively

Melissa Cidade

Melissa.Cidade@Census.gov

Heidi St.Onge

Heidi.M.St.Onge@Census.gov

Rebecca Keegan

Rebecca.Keegan@Census.gov

Rebecca Hutchinson

Rebecca.J.Hutchinson@Census.gov