Multi-phase Pretesting during a 

Survey Redesign 

Mary C. Davis

Questionnaire Development and Evaluation Group Leader 

Center for Survey Measurement, U.S. Census Bureau

Presented at 2018 CE Survey Methods Symposium

July 17, 2018

Disclaimer: Any views expressed are those of the authors and not necessarily those of the U.S. Census Bureau.

1
Qualitative Testing For Questionnaire 

Development and Evaluation

• Expert Review

• Focus Groups

• Technical Review Panels

• Cognitive Interviewing

2
What Are Cognitive Interviews?

•One-on-one interviews conducted with respondents in the survey 

target population

• Identifies “nonsampling error”-- problems in the response process

• Examines the mental processes of respondents as they complete the 

survey

• Identifies potential problems and their behavioral/cognitive causes

3
National Teacher and Principal Survey (NTPS)

• Redesign of the School and Staffing Survey conducted by NCES

• Provide more timely data (every 2 years instead of every 4 years)

• Provide greater flexibility for adding content of interest to researchers and 

policymakers

• First fielded in 2015/16 school year

• Mixed-mode survey

• 3 survey questionnaires – School, principal, and teacher

4
NTPS Testing Cycles 

• For 2015/16

• 2 rounds of cog testing by contractor 

• Single round of testing focused on unconfirmed recommendations

• For 2017/18 

• TRP for new modules – experts in field convened to determine statistics of interest 

and consult on proper terminology

• 4 rounds of iterative testing plus confirmation round for timing length of survey 

• For 2019/20 – currently in process

• Review of edit and imputation rates from 15/16 to identify problematic questions

• Review of internet paradata to examine web questions that need revision

• 4 iterative rounds of testing: 2 cognitive only and 2 joint cog and usability testing

5
Example 1: Hours Delivering Instruction Question

6
Phase 1 Modifications to address over-reporting 

• Better reflect as subset of contract hours

7
Original version 

Version 1

No Exclusion statement

Version 2

• Exclusion list moved to first bullet 

• Added 4 activities to exclusion list 

• Dropped “monitoring students”

Version 3

• Exclusion list added within question 

text

• Same list of activities as Version 2

8
Phase 2 Modifications to address over-reporting 

• V1. Listed more activities to exclude as not ‘delivering instruction'

• “Of the hours you are CONTRACTED to work, excluding time spent on planning, lunch, 

break/recess, proctoring exams, arrival/dismissal of students, and otherwise NOT delivering 

instruction, how many hours during a typical full week do you DELIVER INSTRUCTION to 

students at THIS school?”

• V2. Report hours for activities instead of hours delivering instruction

• “Of the hours you are CONTRACTED to work, how many hours during a typical full week do 

you spend on planning, lunch, break/recess, proctoring exams, arrival/dismissal of students, 

and otherwise NOT delivering instruction to students at THIS school?”

Result: Inconclusive findings; further testing needed

9
Phase 3 Modifications to address over-reporting 

• Removal of one example not done in a “typical week”

• “Of the hours you are CONTRACTED to work, excluding time spent on 

planning, lunch, break/recess, proctoring exams, arrival/dismissal of students, 

and otherwise NOT delivering instruction, how many hours during a typical 

full week do you DELIVER INSTRUCTION to students at THIS school?”

• Re-order question so definition appears first 

• “Excluding time spent on planning, lunch, break/recess, arrival and dismissal 

of students, and otherwise NOT delivering instruction, how many of your 

weekly CONTRACT hours reported in 6-1 do you spend DELIVERING 

INSTRUCTION to students in THIS school?”

10
Example 2: Classroom Organization Question

11

Phase 1 – A problem is revealed
Phase 2 – Less is More 

12
Closing Thoughts

• Cognitive interviewing is good at identifying potential problems in 

survey questionnaires

• Cognitive interviewing allows for iterative testing to explore solutions 

with low cost

• Multiple rounds are best with the aim of your last round being a confirmation 

round of all changes 

• Early rounds can have fewer respondents because big problems are found 

quickly

• “Normally we run out of time before running out of problems.” 

(Willis, 2017 DC-AAPOR course)

13
Questions?

Mary.C.Davis@census.gov

14