National Center for Health Statistics

A Cognitive Interview Evaluation of the Proposed Combined Race and 

Ethnicity Question for Statistical Policy Directive 15, Standards for 

Maintaining, Collecting, and Presenting Federal Data on Race and 

Ethnicity: Findings from the National Center for Health Statistics

Presenter: Stephanie Willson, Collaborating Center for Questionnaire Design and 

Evaluation Research (CCQDER)

Federal Committee on Statistical Methodology, Session F-5

October 25, 2023
Arriving at Results Required a Total CCQDER Team Effort

 Timeline = February – June 2023 (5 months!)

– Clearance

– Interviews

– Analysis

– Report to ITWG

 Operations team

– Multiple recruiters

– Technical support (Q-Notes)

 Interviewers

– Nine CCQDER interviewers

– Two Research Support Services (RSS) interviewers
Methodology
 150 Cognitive Interviews

– 100 in English

– 50 in Spanish

 Question evaluated in the context of other topics

– Gender identity

– COVID-19

– Cancer screening

 Question evaluated in 2 modes (half received each version)

– Self administered

– Interviewer administered
Self Administered

Version:

What is your race or ethnicity? 

Select all that apply AND enter 

additional details in the spaces 

below. Note, you may report more 

than one group.
Interviewer Administered

Version: 

What is your race or ethnicity? 

Select all that apply. Are 

you...[read main categories]

Follow Up: Please provide more 

detail. Are you...[read 

subcategories]
Respondent Demographics by Language

*Numbers do not add to denominators because respondents could choose more than one category. 

English (n=100) Spanish (n=50) Total (n=150)

Race/Ethnicity*

 White 37 21 58

 Black 12 2 14

 Asian 20 0 20

 American Indian or Alaska Native 2 1 3

 Middle Eastern or North African 29 0 29

 Hispanic 9 50 59

Gender

 Female 55 14 69

 Male 43 11 54

 Transgender, non-binary, or another gender 2 25 27

Age

 18-29 28 8 36

 30-49 51 30 81

 50-64 15 12 27

 65 and over 6 0 6
Research Questions
1. What is the response process for the combined question?

2. How do respondents interpret the question?

3. Are there any subgroup patterns? 

4. Are there any differences by mode?

5. Is the question wording (stem & instructions) clear? (Are 

respondents confused by the merging of race & ethnicity?)
Findings
1. Response Process for the Combined Question

 Reflexive answer

– Little or no thought/easy to answer

– Same answer given in any situation or on any form

 Answers require some thought

– Race is a multidimensional concept (people can think of themselves 

different ways)

– Thought is given to how to answer depending on question intent
2. Interpretations of Question Intent

A. Question is asking for a personal assessment of race and ethnicity

B. Question is asking about genealogy
A. Question Intent = Personal Assessment 

 Cultural: feelings of connectedness to a group arising from shared 

experiences, language, customs, and/or place of birth

 Social: the way others in society view and define a person

 Administrative: how a person answers in an official capacity

 Ancestral: based on genealogy
B. Question Intent = Genealogy

 Question structure (subcategories) prompted this interpretation

 Especially prominent among White respondents (prompted over-reporting)

– Tendency to report findings from DNA tests, irrespective of personal 

identity

 Also prominent among Hispanic respondents (prompted under-reporting)

– Subcategories defining ‘White’ seen as European-only ancestry 
3. Group Patterns

a) MENA

b) Hispanic

c) Afro-Latino/a

d) Black/African American
a) MENA

 Respondents easily chose this category 

 The term resonated with respondents

 Category may be missed by those unaccustomed to seeing it

 Some continued to also mark ‘White’ 

– Administratively accustomed to choosing ‘White’

– Socially accustomed to choosing ‘White’ based on others’ perceptions

 Most chose only ‘MENA’

– Personal, cultural identity

 May be some reluctance to report MENA, given perceived prejudice 

towards the group
b) Hispanic

 Category was easily chosen and seen as most appropriate

 Some respondents continued to also mark ‘White’

– Administratively accustomed to choosing ‘White’ 

 Most respondents chose only ‘Hispanic’

– Personal, cultural identity

– White subcategories defined the main category as European (and, 

therefore, not related to those with Hispanic backgrounds)
c) Afro-Latino/a

 Small number (6) due to time constraints & recruitment difficulty

 Recruitment based on 

– Those who marked both ‘Black’ and ‘Hispanic’ on screener

– NOT necessarily those who self-identify as Afro-Latino/a

 Question captured 2 respondents as intended (mark ‘Black’ and ‘Hispanic’)

– Answers based on ancestry

 Other 4 respondents chose only ‘Hispanic’

– Based on personal cultural identity

– Social answer based on perceived US custom to identify as 1 race
d) Black/African American

 Question intended to capture both descendants of enslaved people and 

recent immigrants 

 Subcategories can be confusing

– Respondents don’t necessarily intuit that the subcategories relate to 

recent immigrants and that

– The subcategory ‘African American’ is intended to apply to descendants 

of enslaved people

 Subcategories are potentially insensitive for Black Americans who are 

descendants of enslaved people

– People don’t know from what country their ancestors were taken
4. Mode Effects

 Self Administered

– Misunderstanding the relation of main category to subcategories

• Failure to check a main category

– Misunderstanding of the open field for unlisted subcategories

• Failure to add ethnicities not represented by a check box

– Difficulties could be managed in, for example, a web format 

 Interviewer Administered

– Cumbersome to read

– Need to repeat subcategories

– Difficulties could be managed with better wording to improve flow
5. Clarity of Wording & Instructions

 Question stem and instructions had little bearing on how respondents 

understood and answered the question

 Respondents paid little attention to the actual question and more attention 

to the categories with check boxes

 Categories defined the question intent