National Center for Health Statistics

Examining Health Equity Measurement and Representation through 

the Lens of Total Survey Error

Morgan S. Earp, Ph.D.

John R. Pleis, Ph.D.

Lauren Rossen, Ph.D.

Acknowledgements:

Kristen Miller, Ph.D.

Stephanie Willson, Ph.D.

Paul Scanlon, Ph.D.

Katherine Irimata, Ph.D.

Van Parsons, Ph.D.

Rebecca Hu, Ph.D.

Guangyu Zhang, Ph.D.

Yulei He, Ph.D.

2023 Federal Committee on Statistical Methodology, Hyattsville, MD

October 24, 2023
Overview

• In the face of emerging health challenges and rapid technological evolutions, NCHS 

strives to remain at the forefront of health policy guidance and research 

advancement. 

• One of the goals set by NCHS is to strengthen NCHS’ role in informing policies that 

promote health equity. 

• Currently NCHS is not only examining overall health equity but is also looking 

closely at the relationship between equity and the various sources of total survey 

error to ensure accuracy of health equity findings. 

• This presentation will include an overview of the different types of methods being 

used by NCHS to assess health equity measurement and representation through 

the lens of the total survey error paradigm leading up to dissemination. 
CDC’s CORE Commitment to Health Equity | Health Equity | CDC
National Center for Health Statistics’ Commitment to 

Health Equity 

 In the face of emerging health challenges and rapid technological evolutions, 

NCHS strives to remain at the forefront of health policy guidance and research 

advancement. 

– Strengthening NCHS’ role in informing policies that promote health equity.

– Using NCHS data, policymakers, researchers, and public health 

professionals will be able to identify health disparities and track progress 

toward greater health equity in the United States.

– NCHS’ health equity priorities:

• 1) Expand data collection and analyses

• 2) Innovate through methodological work

• 3) Assess cross-sector disparities using linked data
National Center for Health Statistics’ Commitment to 

Diversity, Equity, and Inclusion

 NCHS will meet current and future challenges by adopting innovative 

workforce models and establishing operational improvements. 

– Demonstrate NCHS’ commitment to building and supporting a diverse 

workforce and inclusive culture.

– Expanding on the success of CDC’s Diversity and Inclusion Executive Steering 

Committee (DIESC), NCHS will continue to cultivate diversity, equity, and 

inclusion (DEI) efforts.

– Including development of workforce metrics for hiring and promotions, 

emphasis on rewards and recognition, creation of mentorship programs for 

diverse hires, and increased reliance on enterprise training and education.
NCHS has always recognized the importance of measuring 

differences

History
NCHS Core Values
Total Survey Error 

Groves, R. M., & Lyberg, L. (2010). Total 

survey error: Past, present, and future. Public 

opinion quarterly, 74(5), 849-879.
Why Look at Equity Through a Total Survey Error 

Perspective?

 To accurately assess health inequities, we need to assess, understand, and correct for 

measurement inequities and representation inequities.

 Some survey error inequities may be completely independent or related to different types of 

biases

 Some survey errors may be related to the same types of demographic biases, and therefore 

compound the distortion throughout the survey process long before we produce estimates of 

health inequity

– If a construct was defined by and for a given population

– If a measure was developed by and for a given population

– If a population frame was developed by and for identifying a given population

– If a sampling strategy was developed by and for selecting a given population

– If a surveying strategy was developed by and for recruiting a given population

– Is it possible that there are inherent biases in the survey process that inadvertently but ultimately exclude other 

populations by design? Are there systemic biases that exist in our survey processes? If so, where do they exist?
• Construct Validity 

– Cognitive Interviews

– Item Response Theory

• Measurement Error

– Cognitive Interviews

– Linear Regression Trees

Measurement Representation

NCHS Examples of Examining Health Equity from a 

Total Error Perspective

• Coverage Error

– Combining data sources

• Sampling Error

– Sampling methodology

– Sample weights

• Nonresponse Error

– Imputation

– Adjustment weights
• National Health Interview Survey (NHIS)

• National Health & Nutrition Examination 

Survey (NHANES)

• National Survey of Family Growth (NSFG)

• National Ambulatory Medical Care Survey 

(NAMCS) 

• National Hospital Ambulatory Medical Care 

Survey (NHAMCS)

• National Hospital Care Survey (NHCS) 

Traditional NCHS Surveys Newer Web Panel Surveys

Quick Overview of Different NCHS Surveys Discussed 

in this Presentation

• Research and Development Survey (RANDS)

• Rapid Surveys System (RSS)
Measurement Equity

 Measurement equity has important implications for survey outcomes, 

such as health and healthcare.

 It is important to understand when measures are biased or may be subject 

to differential measurement error as this can distort (either exacerbating 

or concealing) health inequities.

 Biased measures can arise from differential construct validity or 

differential measurement error.
Construct Validity
Examining Construct Validity Using Cognitive 

Interviews

 The NCHS Collaborating Center for Questionnaire Design and Evaluation Research 

(CCQDER) uses cognitive interviewing methodology to make survey research more 

objective, so that our data is more comparable, more inclusive, and more 

equitable.

 Cognitive interviews are used to identify the constructs captured by individual 

questions by identifying the specific phenomena that account for a respondent’s 

answers

– What are respondents thinking about?

 Comparability studies are used to determine whether constructs are consistently 

captured across salient respondent groups?

– Do demographic subgroups think about the phenomena the same?
Comparative Study

1. Identify the various constructs captured by individual questions 

2. Determine whether they are consistently captured across groups of 

respondents

3. For identified differences, determine reason for differences 

– What about respondents’ experiences inform how they interpret or 

process a survey question? 

– Do the differences impact comparability?
Construct Validity

Visual Representation of Construct Schema 

Question: In general, would you say your health is excellent, very good, good, fair or poor?

Phenomena Considered by 

Respondents

Health Conditions

Pain or Discomfort Physical Limitations

Dependence on 

Medication

Health Habits

Healthy: Regular 

Exercise, Proper 

Nutrition

Unhealthy: 

Smoking, 

Overeating, Alcohol 

Consumption
Phenomena Considered by 

Respondents

Auditory Hearing

Hearing

Quiet Environment

Hearing 

Noisy Environment

Listening

Following Instruction

Unpleasant Tasks

Following Instruction

Pleasant Tasks

Construct Validity

Visual Representation of Construct Schema 

 

 Question: Does your child have difficulty hearing?
Comparative Study

1. Identify the various constructs captured by individual questions 

2. Determine whether they are consistently captured across groups of 

respondents

3. For identified differences, determine reason for differences 

– What about respondents’ experiences inform how they interpret or 

process a survey question? 

– Do the differences impact comparability?
Phenomena Considered 

by Respondents

Safety of Water 

Is the water at school safe?

Occurrence 

Does my child drink water 

during school?

Version 1: 

Does my child drink any 

kind of water during school?

Version 2:

Does my child drink only 

school water during school?

Child's Physical Ability 

Is my child physically able to 

access water at school?

Version 1:

With assistance, is my child 

able to access water? 

Version 2: 

Is my child able to access 

water without assistance?

Comparative Study

United States/English, Jamaica/English: Does your child use drinking water facilities at school?

India/Hindi: �ा (नाम) �ू ल म�की पानी पीनेकी सुिवधा को आसानी सेइ�ेमाल कर सकता/ सकती है?

Parents of children in India Parents of children in US and Jamaica

Parents of children with disabilities
Examining Construct Validity Using Item Response 

Theory (IRT)

 We’ve started using item response theory (IRT) to explore construct validity by 

comparing item agreeability and inter-item correlations across demographic subgroups 

on the National Health Interview Survey (NHIS) to better understand where differential 

item functioning occurs among NHIS questions

 Research Questions: 

– How does an individual’s standing across multiple latent dimensions of health (θ) 

impact their responses to individual NHIS items?

• Are some NHIS items easier or more difficult to agree with (b) given θ?

– Does the difficulty/agreeability of an item for a given θ vary across 

demographic subgroups?

• Do some items have a lesser/greater impact (a) on θ?

– Does the impact on an item on θ vary across demographic subgroups?
Examining Construct Validity Using Item Response 

Theory (IRT) - Example

 For example, in an exploratory analysis using only a single latent dimension 

of heath we noticed the following:

– Differential Item Difficulty (b):

• Females found it more difficult than males to agree with ever having smoked 100 

cigarettes or more in their life, given the same within group latent dimension 

standing

• College graduates with a bachelor degree or higher found it more difficult than noncollege graduates to agree with ever having smoked 100 cigarettes or more in their 

life, given the same within group latent dimension standing

– Differential Item Discrimination (a):

• Ever having smoked 100 cigarettes or more had a greater impact on within group 

latent dimension standing for females than for males
Measurement Error
Examining Measurement Error Using Cognitive 

Interviews

 For example, when studying interpretative response processes for gender 

identity, CCQDER discovered the following:

– Gender minorities and non-minorities interpret the question in different ways

• The information collected are riskier for gender minorities to report, and therefore 

may result in underreporting (false negatives) or nonresponse

• The sex, gender, and sexual orientation constructs are less well understood and 

conflated by non-minorities, resulting in overreporting (false positives)
Examining Measurement Error Using Linear 

Regression Trees

 We used conditional linear regression trees to assess measurement error by 

comparing self-reported versus lab measurements of chronic conditions collected 

on the National Health and Nutrition Examination Survey (NHANES)

 Research Questions:

– Do self-reporting errors (β1) vary across demographic subgroups?

– Are identified self-reporting errors (β1

*) significantly different from zero?
Representation Equity

 Representation equity has important implications for survey outcomes, 

such as health.

 It is important to understand when estimates are biased or may be subject 

to differential representation error as this can distort (either exacerbating 

or concealing) health inequities.

 Biased representation can arise from differential coverage error, 

differential sampling error, or differential nonresponse error
Coverage Error
Examining Coverage Error in Web Panel Data

 RANDS during COVID-19 Round 1 (June 9-July 6, 2020) and Round 2 

(August 3-20, 2020) were conducted using NORC’s AmeriSpeak Panel and 

an opt-in panel (Dynata)

 Dynata invites potential respondents through various sample sources 

including:

– Travel, entertainment, media and retail loyalty programs

– Mobile apps

– A broad range of websites, including school and community sites

27
Examining Coverage Error in Web Panel Data: Example

 Respondent characteristics 

varied between Dynata and 

AmeriSpeak respondents

 Opt-in respondents compared 

to AmeriSpeak were

– Younger

– Not married

– Had a lower education 

attainment

– A higher percentage were in the 

Northeast and a lower 

percentage were in the Midwest

28
Sampling Error
Examining & Addressing Sampling Error

 Smaller and more rare populations or domains can produce larger standard errors. 

 NCHS sometimes uses oversampling to increase statistical power and reduce 

sampling error for these smaller more rare populations. 

 For example,

– NCHS recently used oversampling in their Research and Development Survey (RANDS) 8 to produce a 

larger sample of gender minorities to better study health outcomes and produce more precise 

estimates among this population

– NCHS recently used oversampling in RANDS 9 to produce a larger sample of the Afro-Caribbean and 

Middle East, North Africa (MENA) population to better study health outcomes and produce more 

precise estimates among this population
Nonresponse Error
Examining & Addressing Unit Nonresponse Error

 The NCHS Collaborating Center for Statistical Research and Survey Design (CCSRSD) 

examines NCHS surveys for unit nonresponse bias and develops unit nonresponse 

bias adjustment methods to better understand and address unit response inequity

 For example, 

– NCHS recently examined nonresponse bias in the 2016 National Ambulatory Medical 

Care Survey (NAMCS) Supplement on Culturally and Linguistically Appropriate Services 

for Office-based Physicians, specifically focusing on gender, age, and metro status

– NCHS recently used machine learning-based approaches to improve nonresponse 

weights and response equity for the 2018 NAMCS across physicians’, specifically focusing 

on age, specialty, and compensation.
Examining & Addressing Item Nonresponse Error

 The NCHS Collaborating Center for Statistical Research and Survey Design (CCSRSD) examines 

NCHS surveys and CDC surveillance data for item nonresponse bias and develops item 

nonresponse imputation methods to address item response inequity

 For example, during the COVID-19 pandemic, incompleteness in surveillance data limited 

understanding of disparities. 

– CDC's case-based surveillance system contained case-level information on most COVID-19 cases in the United States. 

Case-level surveillance data were used to investigate COVID-19 disparities by race/ethnicity, sex, and age. 

– However, demographic information on race and ethnicity is missing for a substantial percentage of COVID-19 cases 

(e.g., 35.8% and 47.2% of cases analyzed were missing race and ethnicity information, respectively). 

– NCHS assisted with developing methods to impute missing race and ethnicity to derive more accurate incidence and 

incidence rate ratio (IRR) estimates for different racial and ethnic groups 

– Multiple imputation can provide more accurate incidence and IRR estimates to better monitor disparities in tandem 

with efforts to improve the collection of race and ethnicity information for pandemic surveillance.
Total Survey Error
Subgroup Estimates (95% CI)

27.3 (24.6-30)

24.8 (22.3-27.3)

48.9 (39.1-58.7)

32.6 (29.3-35.9)

12.3 (9.8-14.8)

Subgroup Data

27.3

24.8

Suppressed

32.6

Suppressed

Addressing Total Survey Error Using Model Based 

Estimates

 Estimates for small groups/small domains often suppressed due to concerns about 

reliability.

 Small domain estimation can be used to improve and generate estimates for small 

subgroups by 'borrowing strength' over time or across groups
Addressing Total Survey Error Using Small Domain 

Estimation

 Enhancements to an existing 

statistical tool and modeling 

approach (modified Kalman filter)

 Mixed effects models borrow 

strength over time and across 

groups

 Large improvements in precision

– Equivalent to up to a six-fold 

increase in sample size in some 

cases

Simulated quarterly trends in diagnosed diabetes 

by group from the National Health Interview 

Survey, 2019-2021
Addressing Total Survey Error Using Model Based 

Estimates

 Small area estimation can be used to 'borrow strength' across geographic 

areas to produce more reliable estimates for small groups/geographies

Disparities in infant mortality rates between non-Hispanic Black and White infants, 2017-2019

Only ~5% of counties have sufficient data to 

calculate rates for both groups (n>10)

Model-based estimates of relative Black-White 

disparities in infant mortality rates
Summary

 CDC and NCHS have a commitment to providing data and measurements to 

support health equity

 NCHS has been and continues to develop methods that can be used for assessing 

and improving estimation of small, under-represented populations both from a 

measurement equity and representation equity perspective

 NCHS uses a variety of qualitative and quantitative methods to examine and 

address a variety of total survey errors, with a focus on reducing total survey error 

inequities to improve measurement and representation equity, so we can better 

measure, understand, and address true health inequities
For more information, contact CDC

1-800-CDC-INFO (232-4636)

TTY: 1-888-232-6348 www.cdc.gov

The findings and conclusions in this report are those of the authors and do not necessarily represent the 

official position of the National Center for Health Statistics, Centers for Disease Control and Prevention.

Morgan S. Earp

mearp@cdc.gov