National Center for Science and Engineering Statistics

Social, Behavioral and Economic Sciences

National Science Foundation

Measuring Sexual Identity and Gender Identity (SIGI) 

Among College Graduates: Results from a Methodological 

Experiment Using a Non-Probability Sample

Rebecca L. Morrison, Flora Lan, Jesse Chandler,* and Karen Hamrick+

November 3, 2021

FCSM Research & Policy Conference

* Mathematica

+ NCSES, Retired
National Center for Science and Engineering Statistics https://ncses.nsf.gov

Disclaimer

This presentation provides results of exploratory research 

sponsored by the National Center for Science and Engineering 

Statistics (NCSES) within the National Science Foundation 

(NSF). This information is being shared to inform interested 

parties of ongoing activities and to encourage further 

discussion. Any views expressed are those of the authors and 

not necessarily those of NCSES or NSF. 

2
Background

3
National Center for Science and Engineering Statistics https://ncses.nsf.gov

NCSES, NSF’s federal statistical agency

Mission: Policy-neutral policy-relevant statistical data on the U.S. 

science and engineering enterprise

Federal clearinghouse for data that provides key insights on the 

American economy

• Science and engineering education

• Science and engineering workforce

• Research and development (R&D)

• U.S. competitiveness in science and engineering

4
National Center for Science and Engineering Statistics https://ncses.nsf.gov

What we know about collecting SIGI data

Data users want NCSES to collect this information

Language is ever-changing

• Implications for question wording, time series trends

Generational differences

• Younger people more likely to report being non-binary

Best practices are evolving

• FCSM interagency group is doing a lot of work here

• Ask questions, collect data in a way that is relevant to the research

5
National Center for Science and Engineering Statistics https://ncses.nsf.gov

Research questions

Methodological experiments

• Q1: Does the order of the response options in a sexual identity 

question affect response?

• Q2: Does the question order of the 2-step gender identity series 

affect response?

Other descriptive findings

• Q3: Are respondents willing to provide their gender pronouns?

• Q4: Are people comfortable with disclosing their sexual identity, 

gender identity, and pronouns to various people and entities? 

6
Methods and Results

7
National Center for Science and Engineering Statistics https://ncses.nsf.gov

MTurk-recruited convenience sample 

• Web-only survey

• Field period: Spring 2021

• Incentive: $2

• Respondents: 2,811

• 2,680 college graduates 

with IP addresses in the 

United States

8
National Center for Science and Engineering Statistics https://ncses.nsf.gov

Instrument

Sexual Identity: Varied order of response options

Gender Identity: Varied question order

Personal Pronoun Question

Probes:

• Comprehension of SIGI 

and pronoun questions

• Comfort with SIGI and 

pronoun questions

• Willingness to disclose SIGI 

and pronouns to others 

9
National Center for Science and Engineering Statistics https://ncses.nsf.gov

Probe questions

Sexual Identity Gender Identity Pronouns

I understood what the question was asking 

about me

X X

I understood all of the answer choices X X

I was able to accurately report my _____ identity X X

How easy or difficult was it to answer the 

question

X X

I am certain of my _______ identity X X

The answer choices for the question were 

complete

X X X

I would feel comfortable telling...

• Friend

• Family member

• Doctor

• Police officer

• Federal government

X X X

10
National Center for Science and Engineering Statistics https://ncses.nsf.gov

Q1: Does the order of the response options in a 

sexual identity question affect response?

Gay or Lesbian First (n = 1,345)

Which of the following best represents 

how you think of yourself?

o Gay or lesbian

o Bisexual

o Straight; that is, not gay, lesbian, or 

bisexual

o Something else

Straight First (n = 1,333)

Which of the following best represents 

how you think of yourself?

o Straight; that is, not gay, lesbian, or 

bisexual

o Bisexual

o Gay or lesbian

o Something else

11
National Center for Science and Engineering Statistics https://ncses.nsf.gov

Q1 results – question response

Sexual Identity Gay or Lesbian First Straight First

Gay or lesbian 4.6% 2.9%

Bisexual 7.7% 7.4%

Straight 86.4% 87.7%

Something else 1.3% 2.0%

• Significantly more participants selected “gay or lesbian” when it was listed first 

(Fisher’s exact test)

• < 2% participants identified as “something else.” Most identified as asexual, 

pansexual, or queer

12
National Center for Science and Engineering Statistics https://ncses.nsf.gov

Q1 results – probe questions

When “gay or lesbian” was 

listed first, sexual minority 

participants reported

• Less able to accurately 

report their sexual identity

• More difficulty in 

answering the question

Straight 

participants

Straight listed 

first

Straight 

participants

Gay or 

lesbian 

listed first

Sexual 

minority 

participants

Straight 

listed first

Sexual 

minority 

participants

Gay or 

lesbian 

listed first

I understood all of the answer 

choices 1.05 1.04 1.04 1.09

I was able to accurately report 

my sexual identity 1.05 1.03 1.2 1.28

How easy or difficult was it to 

answer the question 1.06 1.04 1.28 1.42

I am certain of my sexual identity 1.07 1.09 1.35 1.31

The answer choices for the 

question were complete 1.3 1.33 1.79 1.89

13

Mean scores of web probe questions (unadjusted), by sexual identity

(Higher score = less favorable evaluation of the sexual identity question)
National Center for Science and Engineering Statistics https://ncses.nsf.gov

Q2: Does the order of the 2-step gender identity 

series affect response?

Gender Identity First (n = 1,334)

Do you describe yourself as male, 

female, or transgender?

o Male

o Female

o Transgender

o Do not identify as female, male or 

transgender

What sex were you assigned at birth, on 

your original birth certificate?

o Male

o Female

Sex at Birth First (n = 1,345)

What sex were you assigned at birth, on 

your original birth certificate?

o Male

o Female 

Do you describe yourself as male, 

female, or transgender?

o Male

o Female

o Transgender

o Do not identify as female, male or 

transgender

14
National Center for Science and Engineering Statistics https://ncses.nsf.gov

Probe questions

Statistically significant interaction 

between gender minority status 

and question order

Open-ended comments

Add “nonbinary,” “genderfluid,” or a 

write-in option

Of the closing comments submitted 

by 381 participants...

• 5 suggested the gender identity 

question was unnecessary given 

the survey topic

• 18 seemed hostile to the 

concept

15

Q2 results – probe questions

Sex 

assigned at 

birth first

Cisgender

Sex 

assigned at 

birth first

Gender 

minority

Gender 

identity 

first

Cisgender

Gender 

identity 

first

Gender 

minority

I understood what the question was 

asking about me. 1.02 1.28 1.02 1.13 

I understood all of the answer choices.

1.06 1.33 1.05 1.27

I was able to accurately report my 

gender identity. 1.03 1.89 1.03 2.6

How easy or difficult was it to answer 

the question? 1.03 1.78 1.04 1.93

I am certain of my gender identity. 1.04 1.28 1.04 1.53 

The answer choices for the question 

were complete. 1.31 2.78 1.31 3.6 

Mean scores of web probe questions (unadjusted), by gender identity

(Higher score = less favorable evaluation of the gender identity question series)
National Center for Science and Engineering Statistics https://ncses.nsf.gov

Q3: Are respondents willing to report their gender 

pronouns?

Which set of gender pronouns do you use to refer to yourself?

o She/her

o He/him

o They/them

o Something else (specify)

16
National Center for Science and Engineering Statistics https://ncses.nsf.gov

Q3 results

Which set of gender pronouns do you use to refer to 

yourself?

Overall

She/her 54%

He/him 44%

They/them 1%

Something else (specify) 1%

• Zero item nonresponse

• Web-probing indicated problems with question comprehension

• Of the closing comments submitted by 381 participants, 31 raised concerns 

about the concept of nonbinary pronouns (about the same proportion of 

participants in our sample who reported using nonbinary pronouns)

17
National Center for Science and Engineering Statistics https://ncses.nsf.gov

Q4: Are people comfortable with disclosing their 

sexual identity, gender identity, and pronouns?

Probes asked about comfort disclosing sexual identity, gender 

identity, and pronoun usage with 5 different people/entities:

• Friend

• Close family member

• Doctor or other medical professional

• Police officer

• Federal government agency that produces official statistics (with 

additional language concerning confidentiality and use for statistical 

purposes only)

18
National Center for Science and Engineering Statistics https://ncses.nsf.gov

“I would feel comfortable telling [person/entity] that I 

am [sexual identity]”: Percent of participants selecting 

“strongly agree” or “somewhat agree”

Police officer

Federal government

Friend

Doctor

Family member

0% 20% 40% 60% 80% 100%

Straight

(n = 2,342)

Police officer

Family member

Federal Government

Doctor

Friend

0% 20% 40% 60% 80% 100%

Gay or Lesbian

(n = 105)

Police officer

Family member

Federal government

Doctor

Friend

0% 20% 40% 60% 80% 100%

Bisexual

(n = 203)

Police officer

Family member

Federal government

Doctor

Friend

0% 20% 40% 60% 80% 100%

Something Else

(n = 36)

19
Discussion, Implications, and 

Recommendations

20
National Center for Science and Engineering Statistics https://ncses.nsf.gov

Discussion

✅ MTurk recruiting was successful

✅ Web probing was useful and enlightening

❓ Straight First vs. Gay or Lesbian First?

❓ Sex at Birth First vs. Gender Identity First?

❓/ ✅ Pronouns

❓/ ✅ Disclosure to government

21
National Center for Science and Engineering Statistics https://ncses.nsf.gov

Recommendations

✅ Conduct more research on sexual-identity response 

option order and 2-question gender identity series

o Sexual identity question: Consider adding response options for 

asexual, pansexual, or queer

o Gender identity: Consider adding response options for 

nonbinary and genderfluid

⚠ Proceed with caution on collecting pronouns (perhaps this 

is important for surveys that pre-fill)

22
Contact information

Rebecca L. Morrison

rlmorris@nsf.gov

703.292.7794

23