National Center for Health Statistics

Leveraging Commercial Online Survey Panels

for Timely and Actionable Health Statistics

Stephen Blumberg, Jennifer Parker, Paul Scanlon, Jonaki Bose, Amy Brown

CDC/National Center for Health Statistics

2022 FCSM Research and Policy Conference

October 26, 2022
Vision for a “Rapid Surveys” Program

 A new data system for NCHS, emphasizing timeliness and relevance, 

providing CDC partners with data on new, emerging, and priority health 

topics that are fit-for-use for decision making
CDC Moving Forward Initiative

 Dr. Walensky: “We must institutionalize new internal systems, processes, 

and policies to improve our accountability, collaboration, communication, 

and timeliness within CDC and with our customers.”

 Initiative objectives:

– Share scientific findings and data faster

– Translate science into practical, easy to understand policy

– Prioritize public health communications

– Promote results-based partnerships

– Develop a workforce prepared for future emergencies
Key Characteristics of “Rapid Surveys” Program

 Based on two commercial probability-based online survey panels

– AmeriSpeak panel (NORC) and KnowledgePanel (IPSOS Public Affairs)

 Quarterly data collection, with at least 4,000 surveys per quarter

 Topics will be solicited from CDC and other HHS programs

 Short data collection period and timely release of estimates

 Model-based sampling weights

 Calibration using data from NCHS core data systems

 Production of public use and restricted-access data files

 Regular transparent evaluations (scientific and statistical) of data quality
“Rapid Surveys” Program Goals

1. To provide CDC and other partners with relevant, timely data on new, emerging, 

and priority health topics that are fit-for-use for decision making

2. To carefully evaluate and communicate the fitness for use of health estimates from 

commercial online panels

3. To develop and carefully evaluate estimates for key health outcomes (that is, topics 

that are currently included on NHIS) using statistical models that make joint use of 

data from recent quarterly panel surveys with older data from NHIS
Commercial Probability-Based Online Survey Panels

 A sample of persons who have agreed to complete surveys via the Internet

 Members recruited from randomly selected addresses or telephone 

numbers

 Members receive some form of compensation for completing surveys

 Regularly complete 1-4 surveys per month (sometimes more)
Commercial Probability-Based Online Survey Panels

 Benefits

– Can field quickly

– Can target sample characteristics

– Can easily yield longitudinal data

– Can be used as a low-cost surveillance tool

– Can have visual elements

 Challenges

– Mode or interviewer effects may impact comparisons with NHIS

– Coverage issues: Are panelists similar to people not in an online panel?
RANDS – Research and Development Survey

 Purpose: Methodological survey program that primarily uses web-based survey 

panels for evaluating measurement and estimation questions, started by Division 

of Research and Methodology (DRM) in 2015

 Statistical research: DRM has used RANDS to explore the quality of commercial 

online panel data, and how panel data can be weighted and calibrated to produce 

higher-quality estimates. Research has led to a core set of National Health 

Interview Survey variables included on every round of RANDS for calibration 

purposes

 Measurement research: RANDS has served as a test-bed for question design 

research, allowing NCHS to use mixed-method approaches to question/naire

evaluation. Typically use a combination of experimental designs and embedded 

cognitive (or “web”) probes
RANDS during COVID-19

 RANDS during COVID-19 content: Work loss due to illness, telemedicine, 

and reduced access to care

 RANDS during COVID-19 dates: June 9-July 6, 2020 (Round 1), August 3-

20, 2020 (Round 2*), May 17-June 30, 2021 (Round 3)

 Sample: NORC AmeriSpeak panel, with both internet and phone panels

 Sample size: ~6,000 adults in rounds 1 and 2, ~ 5,200 round 3

 Completion rates (rounds 1, 2, 3): 78.5%, 69.1%, 69.5%

 Response rates: 23.5%, 20.3%, 11.8%

*Round 2 included longitudinal follow-up of Round 1 respondents
Comparison of RANDS with “Rapid Surveys”

RANDS Rapid Surveys

Purpose Research, including question design 

evaluations and experiments and statistical 

calibration and alignment methods

Produce rapid estimates for high priority 

and emerging health topics to meet the 

needs of NCHS and CDC/HHS partners

Timing Research projects vary in timelines.

Lower priority for release of estimates and 

files

High priority for rapid release of estimates. 

Higher priority for release of files for 

transparency

Frequency As needed, depends on funding Regular quarterly surveys 

Sample size Varies by need At least 4000 completes per round 

(6000 per quarter in 2023)

Funding Varies, typically CCQDER collaborative 

projects

American Rescue Plan (for FY23)
Comparison of RANDS with “Rapid Surveys”

RANDS Rapid Surveys

Division Lead DRM, PI CCQDER DHIS, in collaboration with DRM

Data collection Currently NORC’s AmeriSpeak panel Development: RANDS platform

Implementation: AmeriSpeak and 

KnowledgePanel (IPSOS)

Content Determined by DRM, including CCQDER 

collaborative projects/research and 

CCSRSD research needs

Determined by NCHS and CDC priorities for 

health information; questions developed in 

collaboration with CCQDER

Primary output Research papers, DRM memos, Q-Bank 

reports, presentations 

Released estimates (data query tool, 

tables)

Weights Depend on research purpose Model-based calibrated weights using 

NCHS established data systems
NCHS Board of Scientific Counselors

 Charged with answering the following questions:

– Given the current scientific knowledge, under what conditions would 

you recommend the use of panel surveys for emerging or 

supplemental topics where “gold-standard” survey data may or may 

not be available? 

– What additional research and evaluation is recommended, if any, to 

increase your confidence in the fitness-for-use of estimates from panel 

surveys for these purposes?
BSC Recommendations for Using Panel Surveys

 “Online panels are particularly desirable for timely estimates for new, emerging, or 

supplemental topics.”

– Avoid estimates that can be produced by NCHS’s current cross-sectional surveys.

– A risk to NCHS is the production of alternative and discrepant estimates. 

 “Online panels are well-suited for generating estimates of change over time.”

 “Avoid estimates related to the data collection methods or to willingness to participate.”

 “Avoid some subgroup/subdomain estimates.” 

 “Online panels can provide information on the performance of new or altered survey 

questions while retaining inference to the general population, to inform decisions about new 

content or changes to the household population surveys.”
BSC Recommendations for Additional Research

 Research on augmenting a panel with other samples or administrative data to address 

undercoverage and nonresponse

 Conduct an evaluation study to better inform the suitability of an online panel for particular 

types of estimates, the limitations of the panel approach, any needed changes to the design, 

and any improvements to the weighting and estimation

 Evaluate different postsurvey weighting adjustment methods

 Design auxiliary variables to aid weighting adjustments, rather than only rely on measures 

that are currently available

 Periodically evaluate online panel methodology

 Research how to communicate data quality for web panel data relative to established NCHS 

survey data
Summary

 Rapid Surveys is an innovative new data system that emphasizes 

timeliness and relevance over other dimensions of data quality.

 Rapid Surveys builds on the NCHS RANDS program, is informed by the 

NCHS Board of Scientific Counselors, and ongoing statistical and 

methodological research.

 Rapid Surveys content will include constructs that meet NCHS, CDC, 

and HHS priorities; estimates will be released data quarterly.

 Transparency and continuous development are key priorities.
Timeline

 RANDS data collection in late 2022 and early 2023 will continue to inform 

development of the “Rapid Surveys” program 

– Constructs selected can help with evaluating the types of variables that could be on 

Rapid Surveys

– Variables will be included for ongoing evaluation of calibration methods and 

benchmarking

 Rapid Surveys soft-launch in mid-2023, with full implementation in 2024
For More Information

sblumberg@cdc.gov or pscanlon@cdc.gov