National Center for Health Statistics

Asking about Gender as a Single, Non-binary 

Construct Versus a Two-Step Question Approach

Valerie Ryan, PhD

Kristen Miller, PhD

Morgan Earp, PhD

Paul Scanlon, PhD

FCSM Conference

October 25, 2023

The findings and conclusions in this presentation are those of the authors and do not necessarily represent the official position of 

the National Center for Health Statistics, Centers for Disease Control and Prevention.
Outline

 Question design at NCHS

 Our gender identity projects and cognitive interview findings

 RANDS background and data

 RANDS findings
Question Design Approach at NCHS

 Empirical, data-driven question design decisions 

 Respondent-centered (as opposed to expert-centered) 

 Qualitative study of respondents’ interpretive processes

 Comparative analyses across different respondent groups

 Mixed-method validation with split-sample experiments
CCQDER/NCHS Gender Identity Projects

 Applied Projects: Develop/Test single item, non-binary gender question

– US State Dept: Passport (n = 100)

– US State Dept: Employee Management System (n = 44)

– NCHS: Health Surveys (n = 150)

• English

• Spanish (Research Support Services)

 Research Studies: Examine performance of 2-question design

– Cognitive interviewing studies 

– Split-sample experiments with embedded construct/error probes (RANDS)

 Studies available: www.cdc.gov/qbank/SOGI
Cognitive Interview Findings: Response Process 

Differences

Gender Minority Respondents

 Response process framed by opposing 

interests: Inclusion vs Privacy

• Inclusion (benefits) 

• Recognition and affirmation

• Data collection to understand and 

reduce disparities

• Privacy (risks)

• Exposure to rudeness and degradation

• Exposure to discrimination: job loss, 

housing, access to health care

• Personal safety; Physical harm

 Sophisticated understanding of concepts

 Fluid understanding of self; shifting 

identities

Non-minority Respondents

 No risk in reporting

 Gender is typically understood as inherent 

and fixed

 Gender and sex are typically understood 

as the same

 Some respondents conflate ‘sex,’ ‘gender,’ 

with ‘sexual orientation’
Final Wording: Non-binary Question

Are you: Select all that apply.

 Female

 Male

 Transgender, non-binary, or another gender

 To increase sense of inclusion and reduce non-response for gender minorities:

– Identity labels

– ‘Select all that apply’ 

 To reduce confusion and false-positive error for non-minorities

– Simple response items 

– Omission of words ‘sex’ and ‘gender’ in the question stem
RANDS Background and Data
RANDS Background

 RANDS1 program: ongoing series of surveys by the Division of Research 

and Methodology at NCHS using quantitative and mixed methods to 

assess measurement error, validity, and bias

 Administered by NORC at the University of Chicago, AmeriSpeak survey 

probability-based panel + opt-in Lucid and Community Marketing Insights 

non-probability panels

 RANDS 8, June - July, 2023 (n = 16,648)

– Gender minority oversample (n = 526)

– 1⁄2 sample two-step question, 1⁄2 sample non-binary question

1. https://www.cdc.gov/nchs/rands/index.htm
Two-step

What sex were you assigned at birth on your original birth 

certificate?

• Female

• Male

Your responses are very important and will be kept 

confidential. If you still prefer not to say, please indicate 

that below.

What sex were you assigned at birth, on your original birth 

certificate?

What is your current gender? Mark only one.

• Female

• Male 

• Transgender

• Two-Spirit [AI/AN only]

• I use a different term, please specify: [text]

Just to confirm, you were assigned {FILL} at birth and 

describe yourself as {FILL}. Is that correct?

Non-binary gender

Are you: Select all that apply.

 Female

 Male

 Transgender, non-binary, or another gender
Cis women Cis men Cis total Gender 

minority

Missing

Two-step, no 

confirmation, no 

nudge

4,305 (52%) 3,599 (43%) 7,904 (95%) 352 (4%) 24 (0.3%)

Two-step, with 

nudge, no 

confirmation

4,314 (52%) 3,611 (44%) 7,925 (96%) 354 (4%) 1 (0.01%)

Two-step, with 

confirmation 

and nudge

4,330 (52%) 3,618 (44%) 7,948 (96%) 330 (4%) 2 (0.02%)

Cis women Cis men Cis total Gender 

minority

Missing

Non-binary 

question

4,374 (52%) 3,597 (43%) 7,971 (95%) 379 (5%) 18 (0.2%)
Analyses

 Chi-squared tests of homogeneity with Rao-Scott second-order correction

– Self-rated health, life satisfaction, anxiety, depression by question version

• Two-step, no confirmation vs. non-binary

• Two-step, with confirmation vs. non-binary

– Low power, despite gender minority oversample

 Balanced AmeriSpeak and opt-in panels

– Opt-in panel aligned to AmeriSpeak using propensity score methods

• Sample weights adjusted and survey source treated as strata

– Not weighting to the population
Power

Null hypothesis True False

Rejected Type I Error (α) Correct

Not rejected Correct Type II Error (β)

power
RANDS Findings
Two-step, no confirmation vs. non-binary: 

χ2 = 4.10, df = 3.98, p = 0.39, power = 0.38

Two-step, with confirmation vs. non-binary: 

χ2 = 3.23, df = 3.98, p = 0.52, power = 0.30 

balanced frequencies

unbalanced frequencies
weighted frequencies

unweighted frequencies

balanced frequencies

unbalanced frequencies
Two-step, no confirmation vs. non-binary: 

χ2 = 3.17, df = 1, p = 0.08, power = 0.47 

Two-step, with confirmation vs. non-binary: 

χ2 = 2.05, df = 1, p = 0.15, power = 0.33 

balanced frequencies

unbalanced frequencies
weighted frequencies

unweighted frequencies

balanced frequencies

unbalanced frequencies
weighted frequencies

unweighted frequencies

Two-step, no confirmation vs. non-binary: 

χ2 = 0.18, df = 2.98, p = 0.98, power = 0.06

Two-step, with confirmation vs. non-binary: 

χ2 = 0.35, df = 2.98, p = 0.95, power = 0.07

Two-step, no confirmation vs. non-binary: 

χ2 = 0.82, df = 1, p = 0.36, power = 0.16

Two-step, with confirmation vs. non-binary: 

χ2 = 0.41, df = 1, p = 0.52, power = 0.11

balanced frequencies

unbalanced frequencies
Findings Summary

 Two versions produce similar percentages for cisgender and gender 

minority respondents 

 Differences between question version for self-rated health and anxiety, 

but power too low to determine statistically significant differences

– Two-step corrected is closer to the non-binary question

– Having false positives could inflate estimates of self-rated health and decrease 

estimates of anxiety

 Did not see differences between question version for life satisfaction and 

depression
For more information, contact CDC

1-800-CDC-INFO (232-4636)

TTY: 1-888-232-6348 www.cdc.gov

The findings and conclusions in this report are those of the authors and do not necessarily represent the 

official position of the Centers for Disease Control and Prevention.

My contact information: VRyan2@cdc.gov

RANDS: https://www.cdc.gov/nchs/rands/index.htm

Q-Bank: providing access to survey question evaluation reports, 

question design and performance https://wwwn.cdc.gov/qbank/

Q-Notes: designed to facilitate the management and analysis of 

cognitive interviews 

https://www.cdc.gov/nchs/ccqder/products/qnotes.htm