National Center for Health Statistics

Results of a Mixed-Method Study to Evaluate 2-Step 

Gender Identity Measures

Valerie Ryan, PhD

Paul Scanlon, PhD

Kristen Miller, PhD

FCSM Research and Policy Conference

October 25, 2022

The findings and conclusions in this presentation are those of the authors and do not necessarily represent the official position of 

the National Center for Health Statistics, Centers for Disease Control and Prevention.
Outline

 Background

 Research and Development Survey (RANDS) 5 Experimental Study

 National Survey of Family Growth (NSFG) Cognitive Interviewing Study

 Summary and Related Work
Background

 We’ve been testing 2-step gender questions to see how they perform

– Sex assigned at birth (SAAB) and gender identity (GI)

 Quantitative work

– RANDS 5 comparing 2 versions of the GI question and question order

 Qualitative work

– NSFG cognitive interviews

– Government Employee Management System cognitive interviews

– General Population Health Survey cognitive interviews
RANDS 5 Experimental Study
RANDS 5 Background

 RANDS1 program: ongoing series of surveys by the Division of Research 

and Methodology at NCHS using quantitative and mixed methods to 

assess measurement error, validity, and bias

 RANDS 5 administered by NORC at the University of Chicago, AmeriSpeak

survey panel; January & March 2022

 2 x 2 experiment testing question order (sex assigned at birth first vs. 

gender identity first) and two question versions (Pew vs. Williams 

Institute)

1. https://www.cdc.gov/nchs/rands/index.htm
The Questions
Findings

 Of 6,896 completes:

– 66 non-matches; 6,806 matches; 24 missing

 We don’t see statistically significant differences when comparing weighted 

results by gender identity question (Pew vs. Williams Institute) and order 

of question (sex assigned at birth first vs. gender identity first)

 Confirmation question: 45 people skipped; 31 said ‘no’

– Only 14 who said ‘no’ changed their answers

 When asked, ‘Why did you answer “Don’t know” for what sex you were 

assigned at birth, on your original birth certificate?’ 10/23 respondents 

said it was because they did not want to answer the question
Misclassification

 10 confirmation question

 9 protest write-ins

Note:2 Rs said ‘no’ to the confirmation question and changed their answer; 1 was a protest write-in and the other changed from a female match to a 

male match so those 2 are not included in the confirmation question misclassification here, though they were in the table on the previous slide.

*Denominator is number of people in non-match category if data were misclassified, n = 83.
64.7

35.3

71

29

Excellent, Very Good, and Good Fair or Poor

Self-Rated Health: Comparing Correct and Not Corrected 

Weighted Percent Estimates for Non-Match Respondents; 

RANDS 5, 2022

Correct Not Corrected
53.6

44.5

62.2

36.4

Excellent, Very Good, and Good Fair or Poor

Self-Rated Mental Health: Comparing Correct and Not 

Corrected Weighted Percent Estimates for Non-Match 

Respondents; RANDS 5, 2022

Correct Not Corrected
NSFG Cognitive Interviewing Study
NSFG Cognitive Interviewing Study

 What sex were you assigned at birth, on your original birth certificate? 

– Male 

– Female 

 How do you describe yourself? You can select all that apply. 

– Male 

– Female 

– Transgender 

– Something Else

 n = 31; 24 gender non-minority, 7 gender minority
NSFG Cognitive Interviewing Study

The primary focus of the study was two-fold:

1. Do respondents see a difference between the 2 questions?

2. What are the ways respondents defined or conceptualized those 

constructs when formulating answers? 

End Goal: Investigate construct validity & provide insight into potential 

response error
Seeing Difference Between Sex and Gender

1. Difference between sex assigned at birth and gender identity

– No Difference

– Vague Understanding of Difference

– Clear Understanding of Difference

2. Constructs captured

– Sex Assigned at Birth = Biology, Gender = Biology

– Sex Assigned at Birth = Biology, Gender = Social

– Sex Assigned at Birth = Social, Gender = Social

 Potential response error

– Seeing the questions as repetitive

– Confusion over terms
Summary & Related Work

 2-step questions are not performing well

– Differences in how Rs interpret SAAB and GI questions

– Without a confirmation question or write-in gender option, 22.9% of people 

classified as gender diverse would be false positives (cisgender people 

misclassifying themselves)

• This leads to differences in estimates

– Many people are skipping SAAB and/or the confirmation question

 Our group is working on a single gender identity question

– See Kristen Miller’s presentation on Thursday at 10:30 am

• Development of an X Gender Marker Definition for the US Passport: Methodology 

and Implications for Asking about Gender
Our papers

 Miller, K., Willson, S., & Ryan, V. (2021). An Initial Cognitive Evaluation of a 2-Step 

Gender Identity Measure. National Center for Health Statistics - QDRL. Hyattsville, 

MD https://wwwn.cdc.gov/QBank/Report.aspx?1219

 Willson, S., Miller, K. (2022). Cognitive Interview Evaluation of X Gender Marker 

Definitions for the U.S. Passport Application Form. National Center for Health 

Statistics - QDRL. Hyattsville, MD https://wwwn.cdc.gov/qbank/report.aspx?1225

 RANDS 5 Experimental Study paper coming soon!
For more information, contact CDC

1-800-CDC-INFO (232-4636)

TTY: 1-888-232-6348 www.cdc.gov

The findings and conclusions in this report are those of the authors and do not necessarily represent the 

official position of the Centers for Disease Control and Prevention.

My contact information: VRyan2@cdc.gov

Q-Bank: providing access to survey question evaluation 

reports, question design and performance 

https://wwwn.cdc.gov/qbank/

Q-Notes: designed to facilitate the management and analysis 

of cognitive interviews 

https://www.cdc.gov/nchs/ccqder/products/qnotes.htm