Results from Qualitative Testing of New Race 

and Ethnicity Question(s) from the USDA’s 

National Agricultural Statistics Service

FCSM, College Park, MD

October 2023

Kathy Ott and Struther Van Horn

USDA’s National Agricultural Statistics Service

Acknowledgements: Participants, RFO staff, field enumerators, advisory committees
USDA’s National Agricultural Statistics Service

• Publish official agricultural statistics for the United States

• Conduct hundreds of surveys every year to collect data 

about agricultural topics including commodity 

production, farm practices, economics, labor, chemical 

use, etc.

• Collect data predominantly from farms and ranches

– Considered establishments, but for some questions/concepts, 

more like households

2
Race/Ethnicity Questions

• Demographic questions, including race and ethnicity, are 

asked on a limited number of surveys, including the 

Census of Agriculture, to measure and serve the farm and 

ranch population

• OMB Interagency Technical Working Group (ITWG) on 

Race and Ethnicity tasked with content and testing of 

proposed question(s)

3
Initial Proposed Examples

4

Combined Question with 

Minimum Categories

Combined Question with 

Minimum and Detailed Categories
NASS Research Questions

• Which question stem better allows respondents to 

understand the question?

– “What is your race or ethnicity?”

– “What is your race and/or ethnicity?”

• Does simplifying the instructions impact how respondents 

understand and answer the question?

– “Select all that apply”

– “Select all that apply AND enter additional details in the space 

below. Note, you may report more than one group”

• How does the combined question affect understanding and 

self-identification? 5
NASS Cognitive Testing

• Conducted 33 cognitive interviews in March – May 2023

– 10 mimicked self-administered paper data collection, using Zoom 

with screen sharing

– 23 mimicked telephone data collection, conducted over the phone

• Race and ethnicity questions embedded with other 

demographic questions including number of people involved in 

the operation, name, age, sex, and some farmer characteristics

6
Cognitive Interview Participants

• Recruited farmers, ranchers, stake-holders, and survey 

enumerators in several states

• Recruited people of all races and ethnicities that we have data 

for

• Attempted to recruit people we suspected may identify as 

Middle Eastern or North African

7
Self-reported Race/Ethnicity of Cognitive 

Interview Participants

Minimum Race/Ethnicity Category Count1/

White 17

Hispanic or Latino 8

Black or African American 8

Asian 4

American Indian or Alaska Native 4

Middle Eastern or North African 2

Native Hawaiian or Pacific Islander 2

Total 45

1/ Respondents could report more than one race/ethnicity 8
7

NASS Race/Ethnicity 

Question(s) - Paper

9
7

NASS Race/Ethnicity 

Question(s) - Paper

“or” vs. “and/or”

10
7

NASS Race/Ethnicity 

Question(s) - Paper

“or” vs. “and/or”

Long instructions vs. 

“Select all that apply”

11
12

NASS Race/Ethnicity Question 1 - Telephone
13

NASS Race/Ethnicity Question 1 - Telephone

“or” vs. “and/or”
NASS Race/Ethnicity Question 2 - Telephone

14
Cognitive Interview Versions

Description Count

Detailed Question, Long Instructions, Telephone 11

Detailed Question, Short Instructions, Telephone 12

Detailed Question, Long Instructions, Paper 6

Detailed Question, Short Instructions, Paper 4

Total 33

15
Research Question #1

• Which question stem better allows respondents to understand the 

question?

– “What is your race or ethnicity?”

– “What is your race and/or ethnicity?”

• Many respondents (all paper, some telephone) were asked about 

their preference for “or” vs “and/or”

• Main finding: For those respondents who were asked their 

preference, all but one indicated that they would prefer the use of 

“and/or”

– Several respondents indicated that just using “or” can impact how they may 

understand and respond to the question

16
Selected Comments on “or” vs. “and/or”

• “When you look at me or my husband, you may think we are 

African American, when you say ‘and/or,’ you are asking for things 

beyond the obvious, what may not be known” 

• “I prefer ‘and/or’, it would cue me in to think more about a 

detailed answer” 

• “If it was ‘and/or’ it would have changed how I felt about the 

question, the ‘or’ made it seem like people don’t care about me” 

17
Research Question #2

• Does simplifying the instructions impact how respondents 

understand and answer the question?

– “Select all that apply”

– “Select all that apply AND enter additional details in the space below. Note, 

you may report more than one group” 

• Respondents on paper and telephone were exposed to 

either the longer or shorter instructions, then asked about 

their preference

• Main findings: 

– Survey enumerators generally preferred the shorter instructions

– All other participants preferred the longer instructions

18
Selected Respondent Comments on Instructions

• In reference to preference for the longer instructions: 

– “That sounds like it wants me to be as inclusive as I can be” 

– “...is a little clearer, you can tell that you really can check more than 

one race and ethnicity” 

19
Select All That Apply

• Participants were asked if they knew they could select all that 

apply

• All were exposed to at least “Select all that apply” 

• Main finding: While many respondents did select more than 

one or said they knew they could, there were at least eight 

respondents who indicated they did not know

– The more detailed instructions did not seem to reduce the number of 

respondents who did not realize they could report more than one

20
Research Question #3

• How does the combined question affect understanding and 

self-identification?

– On what basis do respondents answer the question? 

– What factors influence the race/ethnicity respondents report? 

– What level of detail did respondents prefer to use to self-identify? 

21
On What Basis Do They Answer?

• Ancestry or culture they identify with

– “What are you made of, what is your make up, who are your 

ancestors, what makes you who you are...”

• Parental lineage, genetic make-up, results from DNA testing

– “Because when my sister did the DNA testing, it showed more of 

African descent than Indian descent...”

• Process of elimination

– “Well, because I am not any of the others”

22
What Factors Influence Self-identification? 

• Who is asking?

– “In the livestock business, you can go in to apply for assistance, if you are 

American Indian, female, or minority, you don’t have to pay – but if I am 

White, I have to pay. In my records with USDA, I am American Indian” 

• This multi-racial respondent indicated that they do not always report that they are 

American Indian 

• Why are they asking?

– “I really didn’t feel like I needed to put that in. If I knew the reason 

behind the survey or what the survey was being used for – I would select 

more than one” 

• Reported a minimum race and/or ethnicity but not detail. When asked why, they said 

didn’t see the importance of providing more detail such as “German” and “Irish” 23
Mode Specific Findings

• For telephone, respondents often interrupted the enumerator 

before they read all response options 

• For paper, respondents often did not check boxes for both the 

minimum reporting category and the detailed categories 

24
Proxy Reporting Findings

• Nineteen participants had a second person involved in the 

farm or ranch and were asked to report as a proxy

• Most could easily report as a proxy because of a familial 

relationship they had with the second person (e.g., spouse, 

sibling, parent, child)

25
Summary of Cognitive Interview Findings

• Respondents strongly preferred using “What is your race 

and/or ethnicity”

• Farmers and ranchers (but not enumerators) generally 

preferred the longer instructions

• Most respondents could report detailed race and/or ethnicity 

for themselves

• Most could report as a proxy for a second person with familial 

ties

26
Thank You

Full report available from the authors

USDA’s National Agricultural Statistics Service

Kathy.Ott@usda.gov

Struther.Vanhorn@usda.gov

27