Race and Ethnicity Measurement: 

Effects of Response Format 

Randall K. Thomas, Frances M. Barlas, & Wendy Gross 

GfK Government and Academic Research 

Presented at the Federal Committee on Statistical Methodology (FCSM) 

Research Conference, Washington, DC. November 4-6, 2013 
2 

Research Question: 

Do different measures of race and 

ethnicity influence responses and 

population estimates? 
3 

Introduction 

In modern complex societies, race and ethnicity 

are often seen as influencing people's thoughts, 

behaviors, and outcomes. 

As a result, these concepts are often measured in 

surveys because we see them as useful 

explanatory mechanisms – even just running 

cross-tabs with race-ethnicity breaks for other 

variables is an attempt to understand if and how 

race-ethnicity influences, or is related to, the 

variable of interest. 
4 

Introduction 

As in most countries, in the United States 

conceptual and measurement problems of both 

race and ethnicity abound. A primary ethnic 

focus in the U.S. is the Hispanic-Latino culture. 

The Hispanic identity is considered a cultural 

orientation in some cases, while in other cases, it 

is based on country of origin. 

In most surveys to this point, Hispanic identity 

has been assessed separately from racial identity 

with its own question. 
5 

Introduction 

Just as the conceptual definition of Hispanic 

identity is varied and sometimes multidimensional, measurement of Hispanic identity 

has also been operationalized in different ways in 

large-scale federal surveys. 
Introduction 

The U.S. Census is primarily a self-administered 

paper-pencil survey. For measuring Hispanic 

identity, the Census provides 5 responses, 4 of 

which indicate 'Yes’:

Is [Person 1] of Hispanic, Latino, or Spanish origin? 

 q No, not of Hispanic, Latino, or Spanish origin 

 q Yes, Mexican, Mexican Am., Chicano 

 q Yes, Puerto Rican 

 q Yes, Cuban 

 q Yes, another Hispanic, Latino, or Spanish origin 

6 
7 

Introduction 

The National Health Interview Survey (NHIS) is 

primarily a face-to-face interview, and provides a 

simple dichotomous yes or no response to 

measure Hispanic ethnic identity: 

Do you consider yourself to be Hispanic or Latino? 

Ο Yes 

Ο No

For those who respond ‘Yes’ they are then asked 

‘Where did your ancestors come from?’
8 

Introduction 

In addition to different conceptualizations and 

operationalizations of ethnicity, there have been a 

number of different ways of conceptualizing and 

measuring race. 

In the Census, the wording of the race question 

(“What is [Person 1’s] race?”) may encourage 

respondents to select a single response using a 

singular referent, while the NHIS wording (“What 

race or races do you consider yourself to be?”) 

may encourage selection of more than one race. 
9 

Introduction 

Another challenge for measurement of race is 

that many people who identify as Hispanic seem 

to find it difficult to select a race in the race 

identity item, with a fairly high proportion 

selecting ‘Other race.’ 

As a result of the difficulties that Hispanics have 

had in answering the race question, and in 

preparation for the 2020 Census, there has been 

some experimentation to create a more general 

single item that places the Hispanic response in 

the field of racial categories. 
10

Introduction 

Responses, especially for multi-element choices 

(as occurs for both Hispanic and race 

measurement), can significantly differ based on 

the nature of the response format. 

In web-based questionnaires Thomas and Klein 

(2006) and Smyth, Dillman, Christian, & Stern 

(2006) identified response format effects for event 

assessment. 
11

Introduction 

In self-administered surveys, the general finding 

has been that Yes-No Grids had higher levels of 

endorsement than Multiple Response Formats 

across topic areas. 

While the Census has traditionally used a multiple 

response format for race measurement with 

respondents being able to select one or more 

racial categories, another option to consider 

using is a Yes-No Grid (categories in rows, ‘Yes’ 

and ‘No’ in response columns). 
12

Objectives 

In Experiment 1, we compared the 2010 Census 

race-ethnicity format with that developed by NHIS 

to discern differences in outcome. 

Experiment 2 built on Experiment 1, but included 

2 additional conditions – 1) a combination 

measure that placed the ‘Hispanic’ category 

within a Multiple Response Format and 2) a 

combination measure that placed the ‘Hispanic’ 

category within a Yes-No Grid format. 
13

Method 

Experiment 1 Respondents: 

As part of a larger project conducted in January and 

February 2013 for the Advertising Research 

Foundation (FOQ2), 57,104 respondents completed 

an online questionnaire that had the experimental 

manipulation embedded within it. Respondents came 

from 17 different sample providers, with at least 3,000 

respondents provided by each provider. All 

respondents came from non-probability opt-in 

sources: 50,852 were from opt-in panels and 6,252 

were from river samples. 
14

Method 

Experiment 2 Respondents: 

This experiment fielded in April and May 2013 with 

3,249 respondents from GfK’s KnowledgePanel®. The 

KnowledgePanel is the largest U.S. probability-based 

online panel; it uses ABS recruitment to be 

representative of U.S. adults and includes: adults 

with no Internet access (22% of adults) for whom they 

are provided a web-enabled device and free ISP; cell 

phone-only households (30% of adults and growing); 

Spanish-language, and; extensive profile data 

maintained on each member with demographics, 

attitudes, behaviors, health, media usage, etc. 
15

Method 

In Experiment 1, respondents were randomly 

assigned to either Condition 1 or 2. In Experiment 

2, respondents were randomly assigned to 

Conditions 1, 2, 3, or 4. 

Experiments 1 & 2 – Condition 1 – Census Version 

 Are you of Hispanic, Latino, or Spanish origin? 

q No, not of Hispanic, Latino, or Spanish origin 

q Yes, Cuban 

q Yes, Mexican, Mexican American, Chicano 

q Yes, Puerto Rican 

q Yes, another Hispanic, Latino, or Spanish origin 
16

Method 

Experiments 1 & 2 – Condition 2 presented the 

dichotomous NHIS Version of the Hispanic 

identity question. 

Do you consider yourself to be Hispanic or Latino? 

 O No 

 O Yes 
17

Method 

Experiments 1 & 2 – Conditions 1 & 2 

Census Item Stem: What is your race? Please select ALL that apply 

to you. 

NHIS Item Stem: What race or races do you consider yourself to be? 

Please select ALL that apply to you. 

q White or European American 

q Black or African American 

q American Indian or Alaska Native 

q Asian or Asian American (e.g. Chinese, Japanese, Korean, 

Filipino, Indian, Pakistani, etc.) 

q Native Hawaiian or Pacific Islander 

q Another race 
18

Method 

Experiment 2 - Condition 3 was a Multiple Response 

Format (MRF) which combined the Hispanic identity item: 

 What is your race or origin? Please select ALL that apply to you. 

q White or European American 

q Black or African American 

q Hispanic, Latino, or Spanish origin 

q American Indian or Alaska Native 

q Asian or Asian American (e.g. Chinese, Japanese, Korean, 

Filipino, Indian, Pakistani, etc.) 

q Native Hawaiian or Pacific Islander 

q Another race 
19

Method 

Experiment 2 - Condition 4 was a Yes-No Grid (YNG) which 

requested a response for each element and included 

Hispanic identity. 

Is your race or origin...? Please select a response for each item. 
20

Results – Time to Complete 

Experiment 1 Task Duration. Condition 1 – 

Census Version took 1.1 second longer than 

Condition 2 - NHIS version to complete (17.5 

versus 16.4 seconds, p<.001). 

Experiment 2 Task Duration. Condition 3 – 

Combined Hispanic in Multiple Response took the 

least amount of time to complete (10.6 seconds), 

significantly less than all other conditions. 

Condition 1 took the longest time (21.9 seconds), 

Conditions 2 and 4 were not significantly different 

(18.6 vs. 19.3 seconds). 
21

Results – Time to Complete 
Results 

 

22

Hispanic Ethnicity. In both Experiments, Conditions 1 

and 2 did not differ significantly in the number of 

people classifying themselves as Hispanic, though 

there was a marginal effect (with Condition 2 slightly 

higher than Condition 1 – 11.7% versus 11.2% in 

Experiment 1; 8.2% versus 6.4% in Experiment 2). 

In Experiment 2, the proportion selecting ‘Hispanic’ in 

Conditions 3 and 4 were not significantly different 

than Conditions 1 and 2, though Condition 4 had the 

highest level of endorsement (8.6%). 
23

Results 

Experiment 2 
24

Results 

Black – African American Classification. In both 

Experiments, Conditions 1 and 2 did not differ 

significantly in the number of people classifying 

themselves as Black-African American. 

In Experiment 2, the proportion selecting ‘BlackAfrican American’ in Conditions 3 and 4 were not 

significantly different than Conditions 1 and 2 

(though Condition 4 was the lowest value at 7.3%). 
25

Results 

Experiment 2 
26

Results 

White – European American Classification. In 

Experiment 1, there was no significant difference 

between Conditions 1 and 2 for ‘White – European 

American.’ 

The primary effect of the manipulations occurred in 

Experiment 2 for classification as ‘White – European 

American’. Far fewer people classified themselves as 

‘White’ when Hispanic was combined with race 

measurement for both Condition 3 - MRF and 

Condition 4 - YNG. 
27

Results 

Experiment 2 
28

Results 

Experiment 1 – The number of people selecting more than 

1 race was about 1% higher (p<.01) for Condition 2. 

Though significant, it was a small effect. 
29

Results 

Experiment 2 – The number of people selecting more than 1 

race was much less in Condition 3 than Conditions 1 & 2. 

Condition 4 was also significantly lower than Condition 2. 
30

Discussion 

While the current Census and NHIS versions of raceethnicity measurement were generally comparable, 

they were associated with some small differences 

that seemed consistent across both experiments with 

two different populations. 

Across both experiments, the biggest difference 

seems that the NHIS measure (Condition 2) takes 

slightly less time to complete and the item stem for 

the race measure for NHIS may encourage slightly 

greater selection of more than one race. 
31

Discussion 

In examining the new measures, Condition 3 (Multiple 

Response Format) did seem to show the greatest 

difference from the current measures in time to 

complete, taking the least amount of time. 

Both Conditions 3 & 4 significantly reduced the 

proportion of people classifying themselves as ‘White 

– European American’. 
32

Discussion 

However, in our analyses of the 4 category single 

race classification (White – Non-Hispanic, Black – 

Non-Hispanic, Hispanic, Other) that is often used for 

weighting and reports, we found no differences 

between experimental conditions. It appears that 

most who classify themselves as ‘Hispanic’ do not 

also select ‘White’ in the combined measures. 
33

Discussion 

These studies point to some significant differences in 

the measurement of race-ethnicity. Additional largerscale work will be needed to confirm the effects with 

greater precision. 

And as always in this area, clearer conceptualization 

and a better understanding of how people selfclassify with regard to race and ethnicity will be 

needed. 
Thank You!

Contact: 

Randall K. Thomas 

randall.thomas@gfk.com