1 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Quantitative Testing Results from a New Race

and Ethnicity Question from the

Bureau of Labor Statistics

Tywanquila Walker, Rebecca L. Morrison,

Robin Kaplan, Victoria R. Narine, & Erica Yu

U.S. Bureau of Labor Statistics

October 25, 2023

FCSM Research & Policy Conference
2 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Background 

 The U.S. Bureau of Labor Statistics conducted two quantitative 

studies with nonprobability panels to examine the potential 

effects of combining the race and ethnicity questions proposed 

in Federal Register Notice 88 FR 5375. 

Office of Management and Budget. (2023). Initial Proposals for Updating OMB’s Race and Ethnicity Statistical 

Standards. Federal Register. https://www.federalregister.gov/documents/2023/01/27/2023-01635/initialproposals-for-updating-ombs-race-and-ethnicity-statistical-standards
3 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Research Questions

Do changes to question stem or instructions affect

œResponse distribution?

œParticipant understanding?

œParticipant response?
4 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Participants

 Recruited from two nonprobability panels as part of online 

screener questionnaires for other surveys

 Participants (N = 2364) 

œ Study 1, n = 1363 - jobs and work schedules, currently employed

œ Study 2, n = 1001 - health conditions and work, had health condition 

that limited the kind or amount of paid work they could do

 Data for studies combined, results are reported together

Study 1 was fielded May 3-8, 2023. Study 2 was fielded May 24-26, 2023.
5 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Design – Stems and Instructions

 2x2 between-subjects design, random assignment to treatment 

groups

 Questions stems

œ “What is your race and/or ethnicity?”

œ “What is your race or ethnicity?”

 Brief or detailed instructions 

œ “Select all that apply.”

œ “Select all that apply AND enter additional details in the spaces below. 

Note, you may report more than one group.”
6 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Design – Four Treatment Groups

 Question Stems and Instructions

œ race and/or ethnicity; select all

œ race and/or ethnicity; select all AND enter additional details

œ race or ethnicity; select all

œ race or ethnicity; select all AND enter additional details

 Random assignment to treatment groups

 E.g., What is your race or ethnicity? Select all that apply.
7 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Survey Questions

 Open-ended question to describe race or ethnicity

 Closed-ended, combined race and ethnicity questions

 Reflection on race and ethnicity question

 Origins and identity questions
8 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Survey Questions – Open-Ended

 Open-ended question to describe race or ethnicity

(race and/or ethnicity)
9 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Survey Questions – Closed-Ended

 Closed-ended, combined race and ethnicity questions

œSeven broad categories

œDetailed categories appeared under broad categories 

œDetailed categories collected on same screen with an unfolding design
10 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Survey Questions – Categories

Broad Category Detailed Category
11 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Survey Questions – Reflection

 Participants were asked how well the race and ethnicity 

question reflected how they see themselves.
12 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Survey Questions – Origins

 Pair 1 – origin central to identity

 Pair 2 – familiar with origins

 Pair 3 – connection to origin
13 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Results
14 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Item Nonresponse – Broad Category

Item Nonresponse for Race and Ethnicity by Treatment Group

Treatment Group Nonresponse (%) Response (%) n

race and/or ethnicity; select all 2 (0.34) 593 (99.66) 595

race and/or ethnicity; select all AND enter additional details 2 (0.33) 598 (99.67) 600

race or ethnicity; select all 1 (0.17) 582 (99.83) 583

race or ethnicity; select all AND enter additional details 1 (0.17) 585 (99.83) 586

Total 6 (0.25) 2358 (99.75) 2364

 Low item nonresponse (n = 6) for broad category question

*All six participants answered the open-ended question “How would you describe your race or ethnicity (race and/or ethnicity)?” 

White (n = 3), Multi-racial (n = 1), Mixed (n = 1), White American with a Native American grandmother (n = 1)
15 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Broad Categories Selected

 Most participants selected at least 

one broad category.

œ0.25% did not select a broad 

category, but answered openended race or ethnicity question

œ92.6% – one broad category

œ6% – two broad categories

œParticipant who selected all 7 

categories wrote “prefer not to 

disclose” in response to openended question

Number of Broad Categories Selected

Number Selected Count Percent

0 6 0.25

1 2190 92.64

2 142 6.01

3 20 0.85

4 3 0.13

5 2 0.08

6 0 0.00

7 1 0.04

Total 2364 100
16 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Broad Categories Selected by Treatment Group

 Average number of broad categories not significantly different

between treatment groups, F(3, 2360) = 1.84, p = .138

Average Number of Broad Categories Selected by Treatment Group

Treatment Group Mean Minimum Maximum

race and/or ethnicity; select all 1.11 0 5

race and/or ethnicity; select all AND enter additional details 1.07 0 3

race or ethnicity; select all 1.07 0 7

race or ethnicity; select all AND enter additional details 1.09 0 4

Total 1.08 0 7
17 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Detailed Categories Selected

 Most participants selected a 

detailed category

œ13.45% – did not select a 

detailed category

œ86.55% – selected one or 

more detailed categories

Number of Detailed Categories Selected

Number Selected Count Percent

0 318 13.45

1 1203 50.89

2 500 21.15

3 223 9.43

4 86 3.64

5 21 0.89

6 9 0.38

7 2 0.08

8 2 0.08

Total 2364 100
18 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Detailed Categories Selected by Treatment Group

 Average number of detailed categories not significantly different 

between treatment groups, F(3, 2360) = 1.44, p = .231

Average Number of Detailed Categories Selected by Treatment Group

Treatment Group Mean Minimum Maximum

race and/or ethnicity; select all 1.44 0 8

race and/or ethnicity; select all AND enter additional details 1.40 0 6

race or ethnicity; select all 1.41 0 7

race or ethnicity; select all AND enter additional details 1.52 0 7

Total 1.44 0 8
19 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Detailed Categories Write-Ins

 Most participants did not 

write details.

œ74% – did not provide a 

write-in response

œ26% wrote something in at 

least one text box

œOne participant wrote 

“unknown” in all seven 

boxes

Number of Write-Ins

Number of Write-Ins Count Percent

0 1749 73.98

1 581 24.58

2 32 1.35

3 1 0.04

4 0 0.00

5 0 0.00

6 0 0.00

7 1 0.04

Total 2364 100
20 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Detailed Write-Ins by Treatment Group

 Average number of write-ins not significantly different between 

treatment groups, F(3, 2360) = 0.44, p = .728

Average Number of Write-ins by Treatment Group

Treatment Group Mean Minimum Maximum

race and/or ethnicity; select all 0.29 0 2

race and/or ethnicity; select AND enter additional details 0.27 0 2

race or ethnicity, select all 0.29 0 7

race or ethnicity, select all AND enter additional details 0.27 0 3

Total 0.28 0 7
21 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Detailed Categories Write-Ins

 Participants used write-in boxes to provide additional 

categories, explain their ancestry, or provide commentary about 

why they selected a particular category.

œ “Unknown” or “unsure” because they or a parent were adopted 

œ “7 generations all US born.”

œ “Mother is from the US, father is from the UK.”

œ “I am mixed like most people. Would have to get a DNA test to know 

for sure what exactly.”
22 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Reflection of Self in Race or Ethnicity Question

 “Now thinking about the last 

question, how well did it 

reflect how you see your 

race and/or ethnicity?”

œ75% – “very well”

œ16.9% – “somewhat well”

How Well Question Reflected Race or Ethnicity

Response Count Percent

Did Not Respond 1 0.04

Not well at all 58 2.45

Not too well 130 5.50

Somewhat well 399 16.88

Very well 1776 75.13

Total 2364 100
23 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Reflection of Self in Race or Ethnicity Question

72.78

84.46 83.33 85.37

78.69

17.94

12.44 14.58 12.2

18.03

6.44

2.79 2.59 1.39 2.44 3.28 0.52 0.69 0 0

0

10

20

30

40

50

60

70

80

90

100

White

(n = 1756)

Black or African

American

(n = 193)

Asian

(n = 144)

Hispanic or Latino

(n = 82)

White, Hispanic or

Latino

(n = 61)

Percent

Very Well

Somewhat Well

Not Too Well

Not Well At All

Thinking about the last question, how well did it reflect how you see your race and/or ethnicity?”
24 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Origins – Broad Categories

 Average number of broad categories not significantly different 

between the two statements

 Pair 1 - F(1, 2362) = 0.09, p = .767

œ My origin is not central to my identity (M = 1.09, SD = 0.31, n = 1443)

œ My origin is central to my identity (M = 1.08, SD = 0.39, n = 921)

 Pair 2 - F(1, 2360) = 0.04, p = .837

œ I am not too familiar with my origins (M = 1.08, SD = 0.38, n = 911)

œ I am very familiar with my origins (M = 1.09, SD = 0.35, n = 1451)

 Pair 3 - F(1, 2361) = 0.8, p = .372

œ I do not feel a strong connection with the cultural origin of my family (M = 1.08, SD = 0.37, n = 1209)

œ I feel a strong connection with the cultural origin of my family (M = 1.09, SD = 0.35, n = 1154)
25 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Origins – Detailed Categories

 Average number of detailed categories significantly different

between the two statements in Pair 1 and Pair 3

 *Pair 1 - F(1, 2362) = 36.5, p < .001

œ My origin is not central to my identity (M = 1.55, SD = 1.14, n = 1443)

œ My origin is central to my identity (M = 1.27, SD = 0.95, n = 921)

 Pair 2 - F(1, 2360) = 2.23, p = .136

œ I am not too familiar with my origins (M = 1.48, SD = 1.10, n = 911)

œ I am very familiar with my origins (M = 1.41, SD = 1.07, n = 1451)

 *Pair 3 - F(1, 2361) = 17.81, p < .001

œ I do not feel a strong connection with the cultural origin of my family (M = 1.53, SD = 1.12, n = 1209)

œ I feel a strong connection with the cultural origin of my family (M = 1.34, SD = 1.03, n = 1154)
26 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Origins – Detailed Categories

 Average number of detailed categories significantly different

between the two statements in Pair 1 and Pair 3

 *Pair 1 - F(1, 2362) = 36.5, p < .001

œ My origin is not central to my identity (M = 1.55, SD = 1.14, n = 1443)

œ My origin is central to my identity (M = 1.27, SD = 0.95, n = 921)

 Pair 2 - F(1, 2360) = 2.23, p = .136

œ I am not too familiar with my origins (M = 1.48, SD = 1.10, n = 911)

œ I am very familiar with my origins (M = 1.41, SD = 1.07, n = 1451)

 *Pair 3 - F(1, 2361) = 17.81, p < .001

œ I do not feel a strong connection with the cultural origin of my family (M = 1.53, SD = 1.12, n = 1209)

œ I feel a strong connection with the cultural origin of my family (M = 1.34, SD = 1.03, n = 1154)
27 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Conclusions

 Participants in this study

œ Able to select broad and detailed categories that reflected their race 

or ethnicity

œ Willing to answer the race and ethnicity questions, but did not make 

additional effort to provide written responses

œ Indicated questions reflected their race or ethnicity “very well” or 

“somewhat well”
28 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Conclusions

No significant differences by treatment group for broad 

categories, detailed categories, or write-ins

Changes to question stem or instructions did not affect

œResponse distribution

œParticipant understanding

œParticipant response
Contact Information

29 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Office of Survey Methods Research (OSMR) - www.bls.gov/osmr

Tywanquila Walker

walker.tywanquila@bls.gov