Assessing measurement error in sex and gender identity 

measures in an NCES survey

Elise M Christopher

Study Director, HSLS:09

National Center for Education Statistics
2

This presentation is intended to promote ideas. The views expressed are part of ongoing 

research and do not necessarily reflect the position of the U.S. Department of Education.
3

Introduction
High School Longitudinal Study of 2009 (HSLS:09)

4

• Nationally representative study of 9th-graders in 2009, followed throughout high school and into 

postsecondary education and early careers with a STEM focus

• 23,000+ students in U.S. public and private high schools

• Base year: Fall 2009 (9th grade)

• First follow-up: Spring 2012 (most were 11th-graders)

• 2013 Update: summer-fall 2013 (most had just completed high school)

• High school transcripts collected in 2013

• Second follow-up: 2016 (most were 3 years post-high school)

• 2016 collection modes: Web survey, CATI, and CAPI for nonrespondents and high priority cases 

– Responsive design with monetary incentives; cases such as previous-round nonrespondents and dropouts were 

prioritized 
Gender identity (GI) questions in NCES surveys

5

• Initial focus group and expert recommendations suggested that gender identity measure should 

be “two-step”

– Sex at birth asked first, then current gender (with more than two options) 

• Reviewed extant measures – not many federal surveys asked GI at time of development

• Needed to balance ability for sample members to report identities with minimization of 

measurement error and burden 

• Given concerns about declining response rates, the GI questions were added at the end of the 

survey with the demographics questions 
Sex question in HSLS:09 Base year and First follow-up

6

Student

• What is your sex? 

• Male

• Female 

Parent

• What is [your 9th grader]'s sex? 

• Male

• Female
GI questions in HSLS:09 Second follow-up

7

Gender Identity

• What sex were you assigned at birth (what the doctor put on your birth certificate)? (select one)

• 1=Male, 2=Female

• What is your gender? Your gender is how you feel inside and can be the same or different than your 

biological or birth sex. (check all that apply)

• Male 

• Female 

• Transgender, male-to-female 

• Transgender, female-to-male 

• Genderqueer or gender nonconforming, or some other gender 

• You are not sure
Initial testing of GI questions HSLS:09 Second follow-up

8

• Field-tested items – April – July 2015, Sample size=800

– Number missing, possibly from skips or refusals, was less than 5 cases

– Number of breakoffs on these questions was 0 cases

– Several individuals indicated a transgender gender identity

– Apparent issues with these items: None

• Cognitive testing with gender identity items

– June – July 2015, number= 40 (12 LGBTQ)

– Participants liked having more options

– Participants Googled unfamiliar terms, but help text definitions were helpful

– Simpler question stems worked better

– Option for “something else” or “different identity” was appreciated

– Two-step measure was confirmed to be least confusing

– Apparent issues with items: None
9

Evaluation methods
GI questions in HSLS:09: Evaluation methods

10

• On the data file, variables are provided individually for each response option

– Data users can create their own composite variables from the sex and gender information 

• Responses on Second follow-up (2016) GI items were re-coded – Sex item response was compared to gender identity item

– Responses on the gender identity item were evaluated and cases placed in different gender identity categories

• Sex and gender variables from Base year (2009) and First follow-up (2012) 

compared to variables collected in Second follow-up (2016) – Derived versions of variables indicating student sex that incorporated information collected from parents and 

schools (e.g., school roster information) were compared to student-reported data

– Imputation information considered
GI questions in HSLS:09: Coding of response data

11

• Patterns of reported sex and gender information across base year, first follow-up, and secondfollow-up variables were compared 

– Used to make decisions about recoding into a new gender identity composite variable 

• High school variables (2009 and 2012)

– Variables using student survey data compared to derived versions of those variables with inputs 

from parent surveys and school roster information used for sampling

– Imputation flags (i.e., variables developed to help analysts identify when changes were made to an 

original variable) were noted

• Second follow-up (2016) sample member survey variables 

– Sex at birth (Male or Female)

– Gender identity (check all that apply) 

• Many respondents checked more than one option and each response option was included as a separate 

variable on the data file
GI questions in HSLS:09: Coding of response data

12

• Cases for which reported sex in high school did not match BOTH sex and gender 

identity reported in 2016 were investigated further

– “Investigation group”

– Disagreements between any of the below coding rules were flagged for further follow-up

Investigation group coding rules:

• Transgender = “Transgender, male-to-female” and “Transgender, female-to-male”

– OR cases whose birth sex and gender identity were different (e.g., female on Sex and male on 

Gender), even if Transgender options were not selected

• Another gender = “Genderqueer or gender nonconforming, or some other gender” was 

selected, even with any other selected options

• Unsure = “You are not sure” was selected, even with any other selected options
13

Results
Gender identity status of 2009 ninth-graders in 2016

14

• A very small percentage (0.3 percent) 

identified as Transgender

– In line with other population estimates 

(Meerwijk & Sevelius, 2017)

• A very small percentage (0.8 percent) 

identified as Another Gender identity

– The size of this group in the population is 

less well understood

• Overall, ~2 percent reported a gender identity 

in 2016 that did not match 2009 or 2012 reports

– This group was investigated further for 

possible measurement error 

(“investigation group”)
Gender identity status in 2016 – Investigation group

15

• Of investigation group:

• 45 percent coded as Another gender 

identity

• 28 percent coded as Unsure

• 18 percent coded as Transgender

• 9 percent coded as Possible Errors

• Next, each group will be described 

further....
Gender identity status in 2016 – Investigation group

16

• Another gender identity (45 percent 

of investigation group)

• Selected:

– Genderqueer or gender 

nonconforming, or some other 

gender

– (May have also selected M or F 

to current gender identity)

• Did NOT select:

– Either Trans option or Not sure
Gender identity status in 2016 – Investigation group

17

• Unsure (28 percent of 

investigation group)

• Selected:

– Not sure

– (May have also selected any 

other gender identity)

• Did NOT select:

– Every gender identity option 

available (see Possible errors slide)
Gender identity status in 2016 – Investigation group

18

• Transgender (18 percent of investigation 

group)

• Selected: – Transgender (M-to-F or F-to-M) – Sex at birth = F and GI = M or Sex 

at birth = M and GI = F – (May have selected GI of M or F 

and not Trans) – 40 percent did not 

select Trans 

• Did NOT select: – Not sure – Genderqueer, gender nonconforming 

or another gender identity 

• Some identities changed since high school, 

but some aligned with data collected in high 

school
Gender identity status in 2016 – Investigation group

19

• Possible errors (9 percent of 

investigation group)

• Sources of errors:

– Students did not respond, or left sex 

items blank in early rounds

– Either parent or school provided 

incorrect data in earlier rounds

– Data were missing from all sources 

and imputed incorrectly in earlier 

rounds

– Mischievous responders (Cimpian 

et al. 2018)
20

Discussion
Summary: GI item performance in HSLS:09

21

• Two-step measure likely decreases measurement error

– Individuals who in earlier rounds had provided gender information and not sex 

information could now provide both

– Individuals who did not indicate that they were transgender could be identified by 

comparing sex at birth and current gender identity

– Errors in the data for individuals whose sex information had previously been entered 

incorrectly, was missing, or had been imputed incorrectly could be corrected

– Real change over time does occur – gender identities are fluid – and so more research is 

needed to help tease apart differences in reporting between rounds
Discussion: Recommendations for survey designers

22

• Consider including an option for “Another gender identity”

– More individuals selected this option than transgender options – fine if combining these groups, but 

potentially not indicative of the same experiences

• Include an “unsure” option

– Gender identity is a fluid construct and some populations may be especially prone to uncertainty 

(e.g., youth). Important in a longitudinal study to allow for changes in identification across time

• Include help text to explain what is meant by answer choices

– Words and terms may be unfamiliar to some groups, including those still questioning their identity 

(e.g., “genderqueer”)

– May help filter out “don’t know” responses reflecting lack of understanding from “unsure” due to 

identity questioning 
Discussion: Recommendations for survey designers

23

• Consider including an open-ended answer option, resources permitting – May help clarify cases where there is lingering confusion about terms 

– May help to further identify mischievous responders or protest responses

– May help reduce skips (e.g., in cases of protest)

• Follow up with a confirmation question 

– In cases where sex at birth does not match the indicated gender identity, asking a follow-up question 

to confirm may resolve some reporting or data entry errors

• Consider reporting goals when designing gender items! Possible goals (include but are 

not limited to):

– Allowing respondents to feel seen by reporting on current gender

– Understanding the experiences of gender minority individuals

– Minimizing missing data

– Identifying sources of measurement error
References

24

• California Health Interview Survey (CHIS) 2019 SOGI Work Group Summary and Recommendations. 

http://healthpolicy.ucla.edu/chis/2019-

20wk/Documents/CHIS%202019%20SOGI%20Work%20Group%20Summary%20and%20Recommend

ations%2020180629.pdf

• Cimpian, J. R., Timmer, J. D., Birkett, M. A., Marro, R. L., Turner, B. C., & Phillips, G. L., 2nd (2018). 

Bias From Potentially Mischievous Responders on Large-Scale Estimates of Lesbian, Gay, Bisexual, or 

Questioning (LGBQ)-Heterosexual Youth Health Disparities. American journal of public health, 

108(S4), S258–S265. https://doi.org/10.2105/AJPH.2018.304407

• The GenIUSS Group. (2014). Best Practices for Asking Questions to Identify Transgender and Other 

Gender Minority Respondents on Population-Based Surveys. J.L. Herman (Ed.). Los Angeles, CA: The 

Williams Institute.

• Meerwijk, E.L., and Sevelius, J.M. (2017). Transgender population size in the United States: A metaregression of population-based probability samples. American Journal of Public Health, 107(2), e1–e8. 

Available: https://www.ncbi.nlm.nih.gov/pmc/articles/ PMC5227946.
Thank you!!!

25

Elise Christopher, PhD

Project Officer, High School Longitudinal Studies

National Center for Education Statistics, Institute of Education Sciences

elise.christopher@ed.gov

https://nces.ed.gov/surveys/hsls09