Using Iterative Quick Turnaround 

Cognitive Interviewing to Integrate 

Quality Into the 

2022 U.S. Economic Census

Kristin Stettler

Melissa Cidade

United States Census Bureau

Any opinions and conclusions expressed herein are those of the authors and do not reflect the views of the U.S.Census Bureau. The Census 

Bureau has reviewed this data product for unauthorized disclosure of confidential information and has approved the disclosure avoidance

applied. (Approval ID: CBDRB-FY22-ESMD002-033)
Cognitive Interviewing

• Qualitative methodology used to test and improve survey 

questions by asking respondents open-ended probes

• Ensures that respondent’s answers to survey questions 

meet the original intent

• Goals of these cognitive interviews:

• Determine whether respondents can understand questions and 

report correctly

• Identify what, if any, changes are necessary to the proposed wording

• Provide recommendations to address cognitive issues identified 

during testing

2
Overview of Cognitive Research

for EC22

3

By the Numbers:

• 230 cognitive interviews

• 19 topics

• 65 questions

• 3 rounds of interviewing

• 6 interviewers

• 4 months (August –

November 2021)
Outline of Presentation

• Methodology

• Components of high volume cognitive testing

• Iterative design

• Recruitment

• Test instrument

• Interviewing protocol

• In-field support

• Progress tracking

• Analysis

• Examples

• Lessons learned

4
Methodology

• Respondents recruited over email and self-scheduled themselves

• Generally, contacted respondents from 2017 Economic Census

• Some targeted recruitment for specific industries

• Most participants were in accounting, finance, management, etc. roles

• Respondents came from both single- and multi-unit businesses, large and small

• Approximately 30-minute interviews over phone or virtual meeting software

• Intro/consent form/background information

• Respondents viewed a select set of questions relevant to their industry and some 

cross-sector questions (assigned randomly) via online data collection tool

• Interviewers viewed identical questions but were provided space to ask probing 

questions and take notes on responses

5
Iterative Design

• User comments and verbalizations noted during cognitive 

interview sessions

• Analyzed comments across all participants

• Met with stakeholders after each round

• Revised questions tested in iterative rounds of cognitive 

interviews, as appropriate

• Some topics/questions removed from consideration

• Remaining questions included as tested or revised based on 

findings

6
Recruitment

• Very specific targets

• Size

• Location

• Kinds of activities

7

Solution:

Training to pull 

cases directly 

from the 

Business Register.
Recruitment (cont)

• Self-scheduler –

respondents pick the 

interview day and 

time 

8

Recruitment (cont)
Test Instrument

9
Interviewer Protocol

10

• Respondent Screen • Interviewer Screen
In-field Support

11

• Protocol updates
In-field Support (cont)

• Specialization:

• Dedicated project lead

• Multiple interviewers

• Dedicated programmer and data analyst

• Dedicated chat – communication between interviewers and support 

staff

• Real-time troubleshooting – could answer issues in real- or near-realtime

12
Progress Tracking

13
Progress Tracking (cont)

14

N

GENERAL MANUFACTURING, QUESTION 1 Sum 3.00

M1 PATH Sum 9.00

R1 PATH, QUESTION 1 Sum 11.00

R1 PATH, QUESTION 2 Sum 9.00

R1 PATH, QUESTION 4 Sum 9.00

R1 PATH, QUESTION 5 Sum 2.00

R1 PATH, QUESTION 6 Sum 1.00

S1 PATH, QUESTION 1 Sum 4.00

S2 PATH, QUESTION 1 Sum 5.00

S3 PATH, QUESTION 1 Sum 10.00

S4 PATH, QUESTION 1 Sum 12.00

S4 PATH, QUESTION 2 Sum NONE

S5 PATH, QUESTION 1 Sum 1.00

S6 PATH, QUESTION 1 Sum NONE

S7 PATH, QUESTION 1 Sum 22.00

S7 PATH, QUESTION 2 Sum 16.00

GW PATH, QUESTION 1 Sum 10.00

GW PATH, QUESTION 3 Sum 1.00

GW PATH, QUESTION 6 Sum 10.00

W1 PATH, QUESTION 1 Sum 5.00

W1 PATH, QUESTION 3 Sum NONE

W1 PATH, QUESTION 6 Sum 5.00

W3 PATH, QUESTION 1 Sum 11.00

IA PATH Sum 7.00

Note: contains fictional data.

Critical Item Response Tracking

Progress Tracking (cont)
Analysis

15
Analysis (cont)

16
Example 1: Human Capital

 Goal was to capture information about formal training employees receive

Types of formal training

Expenditures for formal training

Hours of formal training

 Cognitive interviews found that training records are inconsistent, the concepts 

were not easily understood, the questions were burdensome, and they caused 

privacy concerns

 Questions will not appear on the EC22

17
Example 2: Business Technologies

 Goal was to provide a new detail regarding which types of firms 

and subsectors are adopting business technologies

In 2022, did this establishment use [business technology] in its own operations?

How did this establishment acquire the [business technology] used in its own 

operations? 

Types of technologies: Touchscreens/kiosks, Additive manufacturing (3d printing), Radio- frequency identification (RFID), Industrial robots, Service robots, Automated guided vehicles 

(AGVs), Automated storage and retrieval systems, and Augmented reality

 Questions were asked of all industries, but many respondents felt 

they weren’t relevant to them

 Difficult to create definitions that all respondents understood, if 

they weren’t already familiar with the technology

 Revised questions will be on EC22 -- Reduced the number of 

questions from eight to six, re-ordered and clarified instructions

18
Example 3: Remanufacturing

• Goal was to provide new detail on remanufacturing

What percent of the $,000.00 of Sales, Shipments, Receipts, or Revenue reported in Item 5 

was from remanufactured goods produced at this establishment?

 Concept of “remanufacturing” and provided definition did not resonate 

with respondents in Round 1

 For Round 2, drafted a new version of a current question about products 

to evaluate whether respondents could report “new” vs. 

“remanufactured”

 Many respondents in Round 2 did not notice the breakout of “new” vs. 

“remanufactured” and some respondents still found definitions 

confusing

 Clarifications and revisions to the wording/instructions were 

recommended and accepted

 Selected products will be broken out into “New” and “Rebuilt or 

remanufactured” and will be collected in the products section on EC22

19
Example 4: Retail Health Clinic

• Goal was to provide new detail regarding Retail Health Clinics

A retail health clinic is an in‐store clinic with a health care professional who provides 

medical care (e.g., health screenings, treatment of minor injuries and illnesses, or 

management of chronic medical conditions).

Exclude:

- Patient care services delivered only by pharmacists such as dispensing vaccines and 

medications.

• Questions about Retail Health Clinics were tested for both 

Services and Retail industries

• This term did not resonate with respondents in either industry

• Most did not understand what types of establishments this 

definition referred to

• Questions will not be asked on EC22

20
Lessons Learned

• Multiple rounds of cognitive interviews (iterative design)

• Allowed for revisions to improve question wording

• Determined which questions should NOT be included on EC22

19 topics

 12 included on EC22

 7 not included on EC22

65 questions

 37 included on EC22

 28 not included on EC22

21
Lessons Learned (cont)

• Use of innovative technology supported completion of 230 

cognitive interviews in a completely virtual environment (high 

volume testing)

• Statistical analysis software

• Online survey platform

• Chat functionality

• Qualitative data handling software

• Allowed us to keep project on target but also maintain flexibility

• Able to identify poorly performing questions and determine if they 

could be further refined or should be removed from testing

• Able to quickly change wording and pathing based on feedback 

from respondents and stakeholders

22
Thank you!

Kristin Stettler

Kristin.J.Stettler@census.gov

Melissa Cidade

Melissa.Cidade@census.gov

23