National Center for Health Statistics

Case Study in the Use of a Probability Panel: NCHS’ 

Research and Development Survey (RANDS) Program

Paul Scanlon, PhD

National Center for Health Statistics

Division of Research and Methodology

Federal Committee on Statistical Methodology 

2022 Research and Policy 

October 25, 2022
Presentation Overview

• Introduction to RANDS

• Methodological Research Using RANDS

• Special Case: Data Dissemination during COVID-19
RANDS

 NCHS’ Division of Research and Methodology started the Research and 

Development Survey (RANDS), an ongoing set of surveys from commercial 

panel surveys, in 2015

 Established in response to the growing interest in alternative survey 

modes and the maturation of relatively inexpensive, probability-sampled 

commercial survey panels

 Three major goals have driven RANDS throughout the life of the program:

1. Research into Panel Properties

2. Supplement and Develop Methods for NCHS’ Question/Survey Evaluation 

Work

3. (Limited) Production of (Experimental) Estimates
Completed RANDS

Survey RANDS 1 RANDS 2 RANDS 3 RANDS 4 RANDS 5 RANDS 6

Year 2015 2016 2019 2020 2022 2022

Panel Gallup Gallup AmeriSpeak AmeriSpeak AmeriSpeak AmeriSpeak

Mode Web Web Web Web and phone Web and phone Web and phone

Survey Focus

Health 

conditions and 

behaviors

Health 

conditions and 

behaviors

Disabilities and 

opioids

Disabilities and 

opioids

Intimate 

partner 

violence

NSFG items, 

gender identity, 

discrimination

Completed 

Responses 2,304 2,480 2,646 3,442 6,896 2,312

Completion 

Rate 23.5% 30.1% 62.2% 70.0% 72.8% 78.3%

Cumulative 

Response 

Rate

NA NA 18.1% 14.0% 11.1% 12.9%
Completed RANDS during COVID-19

Survey Round 1 Round 2 Round 3

Collection Dates June 9, 2020 – July 6, 

2020

August 3, 2020 –

August 20, 2020

May 17, 2021 – June 

30, 2021

Completed 

Surveys 6,800 5,981 5,458

Completion 

Rate 78.5% 69.1% 69.5%

Cumulative 

Response Rate 23.0% 20.3% 11.8%
Research into Panel Properties

 Mainly concerned with the statistical properties of commercial survey panels

– Focus is on the “Errors of representation” side of TSE

 Coverage error

– Impact of including/excluding phone respondents

– Impact on including hard-to-reach pop—using non-response follow-up flag as proxy

 Non-response error

– Differential item non-response by mode/panel type/intersectional group

– Differential response quality (incl item non-response) in open-ended responses

 Adjustment Error

– Big question: Can we make panel data “look” like data from NCHS’ traditional household 

surveys?

– Evaluated various approaches, including propensity score and calibrated weighting


Question Evaluation Research

 Focus on the “Errors of Measurement” side of TSE

 Both research and “production”

– Methodological research—focused on developing/refining mixed method 

approaches to uncovering measurement error

• “Web probing”—structured, embedded, cognitive probes

• Experimental design

• Natural language processing for open-text probe data

– “Production”—actively using these mixed methods 

• When done after cognitive interviews: expanding/quantifying findings from 

interviews

• When done before/alongside cognitive interviews: producing primary qualitative 

information
n=10,855 non-blank 

responses...
Analysis of Open-Ended Responses

 Hand coding!

– Two coding schemes:

1. Construct captured: telemedicine access or telemedicine use

2. Potential cases of response error

 Experimenting/Implementing some Machine Learning approaches

– Data cleaning (i.e., removing PII)

– Error detection

– Machine coding


Answer to Telemedicine 

Availability Question

Eligible 

Sample

Percent Responding with Only the "Do 

not know" Close-Ended Probe Answer 

Category

Standard 

Error

Yes 2119 6.4 0.5

No 2784 54.1 1.0

Don't Know 219 55.4 3.3

NOTE: Eligible respondents were those respondents assigned to the close-ended probe condition and indicating that they had a 

usual place of health care, total n = 5,129. There were n=7 instances of item non-response within this eligible group of 

respondents to the telemedicine availability question; none of those respondents answered the follow-up probe question.

SOURCE: National Center for Health Statistics, RANDS during COVID-19 Round 2. 2020

Unweighted Percentages (With Standard Errors) Of Eligible RANDS During COVID-19 Round 

2 Respondents Using The "Do Not Know" Close-ended Probe Answer Category Only, By 

Response To The Telemedicine Availability Question

These are potential false 

negative responses!
Split Question Experiment in Round 3

 In the last two months, has this 

provider offered you an 

appointment with a doctor, 

nurse, or other health 

professional by video or by 

phone? [Yes/No]

 Does this provider offer 

telephone or video 

appointments, so that you don't 

need to physically visit their 

office or facility? [Yes/No]

Production of Public Estimates

 Shifted program away from a strictly methodological one in response to 

the pandemic

– A number of NCHS’ traditional surveys were impacted by stay-at-home orders 

and need to limit in-person interviews

 Worked with stakeholders to develop a short list of constructs that:

1. Were not being collected by other federal surveys at the time

2. Would provide useful information on the public health response and 

healthcare access

 Conducted and released findings from three special “RANDS during 

COVID-19” rounds
Published Estimates

Loss of work due to illness

in the previous week because either they or someone in their 

family was sick with COVID-19

Telemedicine access and use

Access and use in the past 2 months, access prior to the 

coronavirus pandemic

Reduced access to care

Inability to receive medical care for any reason or due to the 

coronavirus pandemic

https://www.cdc.gov/nchs/covid19/rands.htm


RANDS Data

 Tables/charts for select constructs available on NCHS’ COVID-19 website:

– https://www.cdc.gov/nchs/covid19/rands.htm

 Public use data files and documentation are available on the RANDS 

website 

– https://www.cdc.gov/nchs/rands/data.htm

 Research Data Center (RDC)

– https://www.cdc.gov/rdc/b1datatype/rdcrands.htm
For more information, contact CDC

1-800-CDC-INFO (232-4636)

TTY: 1-888-232-6348 www.cdc.gov

The findings and conclusions in this presentation are those of the authors and do not necessarily 

represent the official position of the National Center for Health Statistics, Centers for Disease Control 

and Prevention.

https://www.cdc.gov/nchs/rands

https://www.cdc.gov/nchs/covid19/rands.htm

Paul Scanlon

pscanlon@cdc.gov
APPENDIX: Weighting and Calibration
Overview

 RANDS 5 is a probability-based survey that uses a complex sampling 

design

 NORC provided panel weights and survey design variables (strata and PSU) 

that account for the sample design of the AmeriSpeak Panel and selection 

into the RANDS study

 Combining RANDS with high quality reference data can improve estimates 

of health outcomes and reduce potential selection bias

 To produce the estimates for the public release, NCHS implemented an 

additional weighting step to calibrate the weights to the NHIS

 Weighted national and subgroup estimates in this presentation were 

produced using the calibrated weights
NORC-Provided Weights

AmeriSpeak panel weights account for ...

 the sample design of the AmeriSpeak

Panel 

 differential non-response and under 

coverage of groups on the sampling 

frame by incorporating nonresponse 

follow up 

 population characteristics by raking to 

external population totals using 

information obtained from the U.S. 

Census Bureau Current Population 

Survey (CPS) and the NHIS

RANDS panel weights account for ...

 probability of selection of sampled 

panel member into RANDS

 nonresponse-adjustment by raking 

the overall survey sampling weights to 

adult population totals from the CPS 

associated with the following sociodemographic characteristics: age, sex, 

education, race/Hispanic ethnicity and 

Census Division

 extreme weights by trimming weights 

and re-raking to the same population 

totals
NHIS-Calibrated Weights

 The NHIS is a population health survey that has been conducted annually 

by NCHS since 1957 (https://www.cdc.gov/nchs/nhis)

 NHIS collects information on a range of health topics, primarily using 

personal household interviews 

 NHIS interviews are conducted for one adult and one child (if applicable) 

in each sampled household

 NHIS-calibrated weights were generated by calibrating NORC-provided 

RANDS panel weights to the NHIS sample adult weights
Raking Procedure

 Raking is an iterative method of adjusting the sample weights to reflect 

external population counts

 Variables with missing values in RANDS and in NHIS were included in the 

raking procedure

 Variables with missing values in one of the surveys were removed for the 

raking step with an adjustment to the weights for the non-missing values

 Calibrated weights were proportionally adjusted to sum to the total 

number of RANDS respondents for each round
Calibration Variables

Variables

Age

Sex

Race and Hispanic origin

Education

Income

Census region

Marital status

Diagnosed high cholesterol

Diagnosed asthma

Diagnosed hypertension

Diagnosed diabetes

Metropolitan area

Telephone status