Assessing Open-Ended Self-Reports of Sexual Orientation 

and Gender Identity: Is There Room For Improvement?

David Richards, PhD

Study Director, Beginning Postsecondary 

Students Longitudinal Study

National Center for Education Statistics

Elise Christopher, PhD

Study Director, High School 

Longitudinal Studies

National Center for Education Statistics

Maura Spiegelman, PhD

Study Director, National Teacher and 

Principal Survey

National Center for Education Statistics

2022 FCSM Research and Policy Conference

October 25, 2022
2

This presentation is intended to promote ideas. The views expressed are part of ongoing 

research and do not necessarily reflect the position of the U.S. Department of Education.
Agenda for talk

3

• Background and research questions

• Study introductions

• Sexual orientation and gender identity items used

• Methodology for present research

• Results

• Discussion
Background

4

• Around 2012, NCES identified a need to introduce measures of sexual orientation and 

gender identity (SOGI) wherever possible to national education studies

– A lack of data on educational and labor force outcomes for LGBTQ+ populations

– Since 2016, several NCES studies have included SOGI items asked of non-minors

• Use and utility of open-ended response options in questions about SOGI have varied 

across federal surveys

– About 16% of write-in responses to questions on sexual orientation in the 2020 Census Barriers, Attitudes, and Motivators 

Study (CBAMS) were an unlisted identity (Bates et al. 2019).

– For the 2020 CBAMS Mindsets Panel Web Survey and 2020 Census Tracking Survey, about 32% to 37% of write-in 

responses were an unlisted identity. For the 2020 Census Opinion Survey, only 7% provided an unlisted identity. Unlisted 

identities were more often reported by younger, female, and White respondents (Feurer et al. 2022).

• Presence of an open-ended response option and treatment of respondents for analysis 

can impact findings (West and McCabe 2021)
Research Questions

5

Among surveys of postsecondary students and recent graduates:

1. How often did respondents use an open-ended response option when categories were 

provided?

2. What percentage of open-ended responses were meaningful?

3. What meaningful write-in responses were provided?

4. What percentage of open-ended responses were protest responses?
6

Study introductions
National Postsecondary Student Aid Study (NPSAS)

7

• The National Postsecondary Student Aid Study:

– Cross-sectional study of students at all levels of postsecondary education

– Focused on college financing and financial aid

– Conducted every 3–4 years since 1987

– Most recent NPSAS covers academic year 2019-20

– NPSAS years act as alternate base years for Beginning Postsecondary Students 

(BPS) and Baccalaureate and Beyond (B&B) longitudinal studies

– Data collected primarily through web surveys with phone interviews as a 

secondary method
Baccalaureate and Beyond Longitudinal Study (B&B)

8

• Focuses on life after completing a bachelor’s degree, including labor market outcomes, 

further education, and family formation

— Alternating NPSAS collections serve as the base-year collection

— Follow-up collections 1 year, 4 years, and 10 years following graduation 

— Data collected through mix of web and phone interviews

• Nationally representative sample of students graduating with baccalaureate degrees

• Special emphasis on new teachers, but in 2008 also oversampled STEM majors
Study members included in this analysis

9

Study name Eligibility Collection year Cases

B&B:08/18 Earned a BA in 

2008

2018 n=14,670

B&B:16/20 Earned a BA in 

2016

2020 n=17,160

NPSAS:20 Postsecondary 

student in 2020

2020 n=100,410
10

Sexual orientation and gender identity items used
Sexual Orientation (SO) and Gender identity (GI) questions in NCES 

postsecondary surveys

11

• Initial focus group and expert recommendations suggested that gender identity measure should 

be “two-step”

– Sex at birth asked first, then current gender with more than two options 

• Reviewed extant measures – not many federal surveys asked GI at time of development, but for 

SO consulted YRBS, NHIS, NSFG

• Needed to balance ability for sample members to report identities with minimization of 

measurement error and burden 

• SOGI questions were added to the demographics section of the surveys
SOGI questions in B&B and NPSAS

12

Gender Identity

What sex were you assigned at birth (what the doctor 

put on your birth certificate)?

• 1=Male

• 2=Female

What is your gender? Your gender is how you feel 

inside and can be the same or different than your 

biological or birth sex. 1=Yes, 0=No

• Male

• Female

• Transgender, male-to-female

• Transgender, female-to-male

• Genderqueer or gender nonconforming (please describe)

• You are not sure (please describe)

Sexual Orientation

Do you think of yourself as...

• 1=Lesbian or gay, that is, homosexual

• 2=Straight, that is, heterosexual

• 3=Bisexual

• 4=A different identity (please describe)

• 5=Don’t know (please describe)
13

Methodology for present research
Research Question 1:

How often did respondents use open-ended response options?

14

Use of open-ended responses was coded and percentages were calculated for each of the 

following responses:

• Gender

- Genderqueer or gender nonconforming (please describe)

- You are not sure (please describe)

• Sexual Orientation

- A different identity (please describe)

- Don’t know (please describe)

Separate percentages for the above four items were calculated for three studies 

(B&B:08/18, B&B:16/20, NPSAS:20).
Research Question 2:

What percentage of open-ended responses were meaningful?

15

We developed a rubric for coding whether open-ended responses were 

meaningful (e.g., substantive, relevant, intelligible, cooperative, novel).

Non-meaningful responses include those categorized as the following:

– PROTEST (IDEOLOGICAL) – Protest of the question, social commentary. 

– PROTEST (METHODOLOGICAL) – Protest of the way we’re asking, phrasing, terminology.

– DECLINE – Prefers not to answer.

– UNSURE – Does not know answer.

– REDUNDANT – Specified an option already provided.

– UNINTELLIGIBLE – Unclear what the answer means.
Research Question 2:

What percentage of open-ended responses were meaningful?

16

• Three researchers independently coded the open-ended responses.

• To determine final coding, independent codes for the researchers were reviewed for 

agreement. A response was determined to be meaningful if at least two researchers 

coded it so.

• Researchers computed the percentage of meaningful responses out of open-ended 

responses for each question.

•2/3 = some agreement, majority present

•1/3 = no agreement, no majority
Research Question 3:

What meaningful write-in responses were provided?

17

• The responses that were identified as meaningful were then coded into new categories 

not originally provided as response options. 

• Separate categories were created for gender identity and sexual orientation.

• The frequency of meaningful and novel open-ended responses was then measured 

within gender identity and sexual orientation.
Research Question 4:

What percentage of open-ended responses were protest responses?

18

• As noted previously, researchers coded non-meaningful responses as

– PROTEST (IDEOLOGICAL) – Protest of the question, social commentary.

– PROTEST (METHODOLOGICAL) – Protest of the way we’re asking, phrasing, terminology.

– DECLINE – Prefer not to answer.

– UNSURE – Does not know answer.

– REDUNDANT – Specified an option already provided.

– UNINTELLIGIBLE – Unclear what the answer means.

• As with meaningfulness, a response was determined to be an ideological protest if at 

least of the researchers coded it as such.

• Researchers computed the percentage of cases that responded with an ideological 

protest.
19

Results
How often did respondents use open-ended response options?

20

• About 1% to 2% of respondents provided an 

open-ended response instead of selecting an 

offered category for their sexual orientation, 

and about 0.3% to 1% for their gender 

identity

• When respondents provided an open-ended 

response, this was typically because they 

had a response that was different than those 

offered (an “other” response), rather than 

because they "didn't know"

• Open-ended responses were provided more 

frequently for surveys with a more recent 

sample. (Though this was not tested with 

inferential statistical analysis.)
What percentage of open-ended responses were meaningful?

21

• Most write-in responses for "other" 

sexual orientations and gender 

identities were considered 

meaningful

• Relatively fewer explanations 

where respondents "didn't know" 

their sexual orientation or gender 

identity were coded as meaningful

• The meaningful rates for "don't 

know" respondents varied by 

study. (Though this was not tested 

with inferential statistical 

analysis.)
What meaningful write-in responses were provided?

22

• For sexual orientation, text answers were most frequently categorized as pansexual (23% to 

32%), asexual (17% to 23%), or queer (8% to 20%).

• For gender identity, text answers were most frequently categorized as nonbinary (14% to 

41%).

• Some respondents reported terms that were sub-constructs within the larger sexual orientation 

construct.

– Sexual behavior: "polyamorous"

– Romantic orientation: "aromantic," "panromantic"

• Some respondents reported terms that were sub-constructs within the larger gender identity 

construct.

– Gender expression/presentation: "present as femme“

– Gendered pronouns: "they/them pronouns," "use she/they pronouns"
What percentage of write-in responses were protest responses?

23

• A minority of write-in responses for 

"other" sexual orientations and gender 

identities were considered ideological 

protest responses (e.g., "there are only 2 

genders"), though the proportion varied 

by study

– 1% to 6% for sexual orientation –

these are of the 0.8% to 2.1% of 

respondents who wrote in responses

– 3% to 43% for gender identity – these 

are of the 0.3% to 1.0% of 

respondents who wrote in responses
24

Discussion
Discussion

25

• Additional terminology to consider: asexual, pansexual, nonbinary, queer

• Protest responses – were they an issue?

– Relatively tiny when considering how many wrote protest responses compared to total 

number of respondents

• More testing should investigate whether adding new terms confuses people who are not 

writing in terms

– Cognitive testing is always a good idea!

• More testing should investigate the sub-constructs that people think of when they are 

considering whether to use the write-in box
Thank you!!!

26

David Richards, PhD

Study Director, Beginning 

Postsecondary Students Longitudinal 

Study (BPS)

National Center for Education 

Statistics, Institute of Education 

Sciences

David.Richards@ed.gov

https://nces.ed.gov/surveys/bps

Elise Christopher, PhD

Study Director, High School 

Longitudinal Studies

National Center for Education 

Statistics, Institute of Education 

Sciences

Elise.Christopher@ed.gov

https://nces.ed.gov/surveys/hsls09

Maura Spiegelman, PhD

Study Director, National Teacher 

and Principal Survey

National Center for Education 

Statistics, Institute of Education 

Sciences

Maura.Spiegelman@ed.gov

https://nces.ed.gov/surveys/ntps