Just to Confirm: Evaluating the Reliability and Validity of 

Survey Questions on Sex and Gender

Elise Christopher, PhD

Study Director, High School 

Longitudinal Studies

National Center for Education Statistics

David Richards, PhD

Study Director, Beginning 

Postsecondary Students Longitudinal 

Study

National Center for Education Statistics

Maura Spiegelman, PhD

Study Director, National Teacher and 

Principal Survey

National Center for Education Statistics

2023 FCSM Research and Policy Conference

October 25, 2023

Harper Haynes

Research Education Analyst

RTI International
National Center for 

Education Statistics 2

This presentation is intended to promote ideas. The views expressed are part of ongoing 

research and do not necessarily reflect the position of the U.S. Department of Education.
National Center for 

Education Statistics

Agenda for today’s talk

3

• Motivation for the research and overview of research questions

• SOGI data available from NCES, data used for this research, and item wording

•Research questions in detail

•Results

• Discussion

• Next steps
National Center for 

Education Statistics 4

Motivation for the research and overview of research 

questions
National Center for 

Education Statistics

Sex and Gender: Confirmation questions

5

• False positive rates are a concern for sexual and gender minority populations

– A relatively small amount of measurement error can have a large impact on population estimates for 

relatively small subgroups within populations (for example, if cisgender respondents are mistakenly 

categorized as transgender)

– Response errors on sex-assigned-at-birth and gender items lead cisgender respondents to be falsely 

categorized as gender minorities, potentially inflating population estimates or yielding inaccurate conclusions 

about transgender individuals (Miller et al. 2022)

• Transgender respondents may feel singled out or “othered” by a follow-up question 

asking them to confirm their sex if different from their gender

– Asking the same confirmation of all respondents, including those who report the same answer for their sex 

and gender, does not necessarily reduce that stigma for transgender individuals, since they would not know 

that all respondents were asked to confirm their answers. 
National Center for 

Education Statistics

Single confirmation method

6

Double confirmation method

Sex assigned at birth

Gender identity

Confirmation

Sex and gender 

do not match

Next question

Sex and 

gender match

Sex assigned at birth

Confirmation

Gender identity

Confirmation
National Center for 

Education Statistics

Research questions

7

1. How do respondents answer two separate confirmation questions, one for sex assigned 

at birth, and one for gender? Do they confirm or change their responses?

2. Do respondents find the two-step gender questions difficult to answer? 

3. How do respondents answer sex and gender questions upon reinterview?
National Center for 

Education Statistics 8

SOGI data available from NCES, data used for this research, 

and item wording
National Center for 

Education Statistics

Background

9

• Since 2016, several NCES studies have included sexual orientation and gender 

identity (SOGI) items on surveys of adults

– High School Longitudinal Study of 2009 (HSLS:09) Second Follow-up (2016)

– Baccalaureate and Beyond (B&B:08/18, B&B:16/17, B&B:16/20)

– Beginning Postsecondary Students Longitudinal Study (BPS:20/22, BPS:20/25 - planned)

– National Postsecondary Student Aid Study (NPSAS:20, NPSAS:24 - planned)

• Cross-sectional study of students at all levels of postsecondary education

• Conducted every 3–4 years since 1987

• Data collected primarily through web surveys with phone interviews as a secondary method

• Main study preceded by field test
National Center for 

Education Statistics

Study members included in this analysis

10

Study name Eligibility Collection year Cases

NPSAS:20 Fullscale

Postsecondary 

student in 2019-20 

school year

2020 n=96,510

NPSAS:24 Field 

Test

Postsecondary 

student in 2022-23 

school year

2023 n=3,230
National Center for 

Education Statistics 11

Sex assigned at birth

What sex were you assigned at birth (what the doctor put on 

your birth certificate)?

1=Male

2=Female

Gender Identity

What is your gender? Your gender is how you feel inside 

and can be the same or different than your biological or 

birth sex. 

 Male

 Female

 Transgender, male-to-female

 Transgender, female-to-male

 Genderqueer or gender nonconforming (please 

describe)

 You are not sure (please describe)

Gender Identity questions in NPSAS:20 Full Scale Collection
National Center for 

Education Statistics 12

Sex assigned at birth

What sex were you assigned at birth, on your original birth 

certificate?

1=Male

2=Female

Just to confirm, you were assigned [{if sex = 1} male {else 

if sex = 2} female] at birth, on your original birth 

certificate. Is this correct? 

1 = Yes 

0 = No

Gender Identity

What is your gender? Your gender is how you feel inside 

and can be the same or different than your biological or 

birth sex.*

 Man

 Woman

 Transgender

 [{If race = American Indian or Alaska Native} 

Two-Spirit] 

 Nonbinary

 A gender identity not listed. (Please describe)

Just to confirm, you identify as the following:

[list genders selected]

Is this correct?

1 = Yes 

0 = No

*Respondents could choose more than one gender option

Gender Identity questions in NPSAS:24 Field Test Reinterview
National Center for 

Education Statistics 13

Research questions in detail
National Center for 

Education Statistics 14

Sex assigned at birth

Confirmation

Gender identity

Confirmation

Sex assigned at birth

Gender identity

Next question

reliability

reliability

NPSAS:24 field test 

Survey (n = 3,230)

NPSAS:24 field test 

reinterview (n = 870)
National Center for 

Education Statistics

Research Question 1:

How do respondents answer two separate confirmation questions, one 

for sex assigned at birth, and one for gender? Do they confirm or 

change their responses?

15

• How often do respondents confirm their answers to the Sex at Birth and 

Gender Identity items?
National Center for 

Education Statistics

Research Question 1:

How do respondents answer two separate confirmation questions, one 

for sex assigned at birth, and one for gender? Do they confirm or 

change their responses?

16

• OK, but what about burden?
National Center for 

Education Statistics

Research Question 2:

Do respondents find the two-step gender questions difficult to answer? 

17

• Probe with questions at the end of the survey about how difficult it was to 

answer
National Center for 

Education Statistics

Research Question 3:

How do respondents answer sex and gender questions upon reinterview?

18

• How often do respondents answer exactly the same in the field test survey 

and in the re-interview?
National Center for 

Education Statistics 19

Results
National Center for 

Education Statistics

Question 1: Confirmation questions

20

Confirmation question asked after each question in the reinterview for the twostep gender measure:

• 99.9% confirmed their sex at birth

• 100% confirmed their gender identity

• One respondent left the Sex at Birth confirmation question missing, leading to a 99.9% 

confirmation rate. Otherwise, all respondents confirmed their previous selection.
National Center for 

Education Statistics

Question 1: Confirmation questions

21

• Low response time for SOGI items

Study

NPSAS:20 NPSAS:24 FT main survey

Mean response time (sec) Mean response time (sec)

Sex assigned at birth 6.5 5.6

Gender identity 6.9 5.7

Sexual orientation 7.4 5.9

Income 21.3 20.1

Disability 6.9 3.3
National Center for 

Education Statistics

Question 1: Confirmation questions

22

• Generally, reduction in item nonresponse for new item format vs. old format

Study

NPSAS:20 NPSAS:24 FT main 

survey

Item nonresponse (%) Item nonresponse (%)

Sex assigned at birth 0.29 0.34

Gender identity 0.29 0.00

Sexual orientation 2.22 1.00

Income 0.35 0.28

Disability 0.29 0.28
National Center for 

Education Statistics

Question 1: Confirmation questions

23

• Low breakoff rates 

Study

NPSAS:20 NPSAS:24 FT 

main survey

Breakoff rate (%) Breakoff rate (%)

Sex assigned at birth 0.06 0.09

Gender identity 0.00 0.09

Sexual orientation 0.06 0.03

Income 0.79 1.09

Disability 0.07 0.00
Gender Identity

24

What is your gender? Your gender is how you feel inside and can be the 

same or different from your biological or birth sex.

0.2% 0.3%

1.0%

0.2%

3.9%

1.4%

3.3%

0.5% 0.6%

Transgender

(M2F)

Transgender

(F2M)

Transgender Genderqueer Nonbinary A different identity Missing

NPSAS:20 NPSAS:24 Field Test

* * * * *

Not in survey
National Center for 

Education Statistics

Question 1: Confirmation questions

25

• Low rate of changing initial responses in the field test and the reinterview

Survey

NPSAS:24 Field Test NPSAS:24 Reinterview

Backup rate (%) Backup rate (%)

Sex assigned at birth 1.7 0.6

Gender identity 2.3 0.3

Sexual orientation 2.9 0.6

Income 4.1 1.9

Disability 1.9 1.6
National Center for 

Education Statistics

Question 2: Debriefing 

26

On a scale from 1 to 5, with 1 being “Very difficult” and 5 being “Very easy”, how 

difficult or easy was it to answer the previous question about your gender?

Discomfort with 

revealing personal 

information.

Because you guys are 

fussy about asking 

about gender. There’s 

only two... # 1% 3% 3%

91%

Very difficult Somewhat

difficult

Neither

difficult nor

easy

Somewhat

easy

Very easy

Struggling with 

gender identity.

# Rounds to zero Note: Detail may not sum to 100 percent due to rounding. 
National Center for 

Education Statistics

Question 3: Re-interview results

27

• High agreement between field test survey and re-interview

Percent 

Agreement

Cramer’s V

Sex assigned at birth 100.0 1.00

Gender identity: Male 99.4 0.98

Gender identity: Female 99.3 0.98

Gender identity: Transgender 99.8 0.92

Gender identity: Two-spirit 100.0 1.00

Gender identity: Nonbinary 98.7 0.83

Gender identity: A different identity 99.5 0.33
National Center for 

Education Statistics 28

Discussion
National Center for 

Education Statistics

Discussion

29

• Do we need confirmation questions?

• No – reinterview rates convincingly argue that most people do not change their answer 

between original survey and re-interview, indicating low risk of false positives

• A reinterview may be a better option

• If you are still concerned about false positives for your population... 

• Double-confirmation does not single out those who endorse non-aligning sex and gender 

categories as responding potentially in error

• Examine write-ins and missingness rates over time

• Generally accepted that terminology and identities may fluctuate over time within a 

population, so continually evaluating item performance may help keep missingness rates 

low
National Center for 

Education Statistics

Next steps

30

• What analyses do we have planned next?

• NPSAS:24 full-scale collection planned to occur spring 2024
National Center for 

Education Statistics

Thank you!!!

31

David Richards, PhD

Study Director, Beginning Postsecondary 

Students Longitudinal Study (BPS)

National Center for Education Statistics 

Institute of Education Sciences

David.Richards@ed.gov

https://nces.ed.gov/surveys/bps

Elise Christopher, PhD

Study Director, High School 

Longitudinal Studies

National Center for Education Statistics 

Institute of Education Sciences

Elise.Christopher@ed.gov

https://nces.ed.gov/surveys/hsls09

Maura Spiegelman, PhD

Study Director, National Teacher and 

Principal Survey

National Center for Education Statistics 

Institute of Education Sciences

Maura.Spiegelman@ed.gov

https://nces.ed.gov/surveys/ntps

Harper Haynes

Research Education Analyst

Center for Education Data Integration

RTI International

hhaynes@rti.org

Special thanks to Dr. Tracy Hunt-White, Study Director for NPSAS