Demographic Surveys Redesign Program: 

Evidence-Based, Participatory Decision-Making

Rich Levy

Demographic Statistical Methods Division

US Census Bureau

Washington, DC

Federal Committee on Statistical Methodology

October 2022

Disclaimer: Any opinions and conclusions expressed herein are mine 

and do not reflect the views of the U.S. Census Bureau

Cleared for Public Release 1
Topics

• What is Redesign?

• Governance structure

• Evidence-based decision-making (background)

• Use of evidence and buy-in: Case study

• Takeaways

• Future directions

Cleared for Public Release 2
Demographic 

Surveys 

Redesign 

Program

• A research and development program

• Directly funded by Congress to fund the 

coordinated drawing of new samples for major 

demographic surveys

• More recently focuses on both sampling AND 

nonsampling innovations which have 

applications across surveys
Surveys in scope

• American Housing Survey (AHS)

• Consumer Expenditures Survey (Diary (CED) and Quarterly (CEQ))

• Current Population Survey / State Children’s Health Insurance Program (CPS 

/ SCHIP)

• National Crime Victimization Survey (NCVS)

• National Health Interview Survey (NHIS)

• Survey of Income and Program Participation (SIPP)

Cleared for Public Release 4
Sample Redesign: Multistage sample design

• Country is divided into primary sampling units (PSUs)

• Counties or groups of counties

• Most populous PSUs selected with certainty

• Remaining PSUs are stratified by variables of interest

• Sample of remaining PSUs selected

• Secondary sampling units (usually housing units) are selected within the sampled 

PSUs

• Design helps minimize field cost while compiling estimates with desired level of 

precision

Cleared for Public Release 5
Governance structure

• Project leads for standalone research projects

• Workgroups carry out various phases or aspects of the sample design

• Redesign Advisory Group provides oversight to the program, makes 

recommendations based on MERIT

• Expert membership: Sample Design and Estimation – Survey Quality – Survey 

Methodology – Survey Management – Information Technology

• Division leadership provides final decisions based on merit and budgetary 

considerations

Cleared for Public Release 6
Business 

Environment: 

Foundations 

for EvidenceBased 

Policymaking 

Act of 2018

Cleared for Public Release 7

Requires agencies to use 

statistical data to determine 

policy

Agency plans should include 

policy questions, data to be 

used, and proposed methods

• Source: Public Law No: 115-435
Participatory Decision-Making and 

Evidence-Based Policy in Action

A case study

Cleared for Public Release 8
Background: Building a Vision for the 2020 

Redesign

• Meeting with diverse program stakeholders and bureau experts

• Ideas coalesced around five areas of interest:

• Sample design and methods

• Supplemental data (Big Data, administrative records)

• Adaptive design / data monitoring and intervention

• Nonsampling innovations (e.g., measuring and reducing nonsampling 

error)

• Business process improvement

Cleared for Public Release 9
Brainstorming meetings to identify specific 

topics

• In sessions on sample design, staff raised issue whether we needed to 

select a new sample of PSUs

• Staffing challenges

• Data anomalies with new sample, new staff or measuring actual 

change?

• Retaining prior PSUs (as feasible) would reduce these risks

• However, possible threat to representativity

Cleared for Public Release 10
Steps for analysis

11

1.

Assessing 

impact

2.

Identify pros 

and cons of 

various 

options

3.

Measure 

costs of 

options

Cleared for Public Release
First step: Assessing impact

• Staff researched various related items:

• Would we run out of sample in various primary sampling units if we 

sampled without replacement?

• How long do PSUs remain representative over time?

• How efficient (roughly speaking, homogenous) do PSU strata remain 

over time? 

Cleared for Public Release 12
Research findings

• Running out of sample, not a major concern

• PSUs, over time, became less representative on key estimates (a few 

surveys and specific states)

• Between-PSU variance increasing, strata were becoming less homogenous 

over time. More pronounced in specific states and specific surveys

• Authors of report recommended selecting new PSUs (maximizing overlap 

between 2010 and 2020 samples)

Source: Chesnut, T.J., Murphy, P., Park, Y., and Baker, S. (2018) 

Cleared for Public Release 13
Second step: Identify pros and cons of various 

sampling options (template)

14

Issue category Issue subcategory Keep PSUS, don’t 

reselect (minor 

adjustments)

Select new PSUs 

using prior 

geography (minor 

adjustments

Select new PSUs 

using 2020 area 

definitions

Statistical Design Effect

Statistical Variance

Budget Phase-in / phaseout

Operational

Geography

Cleared for Public Release
Third step: Measure costs of options

15

If don’t select new PSUs, save one-time costs in selecting new PSUs and training new 

workers

However, costs associated with increased sample sizes needed over time to maintain low 

variances outweigh one-time savings

THUS: Recommend selecting new PSUs (as applicable) with maximum overlap between 

designs

Source: Chesnut, Levy & Murphy (2019) (National Crime Victimization Survey used as test case.)

Cleared for Public Release
The Figures: Three Sampling Options (Final)

• In course of doing 

analysis, options changed

• Option 3 had the 

combination of smallest 

sample and lowest startup costs

• Source: Chesnut, Levy, & 

Murphy (2019)

Cleared for Public Release 16

Costs Option 1 (No 

new PSUs)

Option 2 (New 

PSUs, no 

attempt at 

maximum 

overlap)

Option 3 (New 

PSUs, 

maximum 

overlap with 

prior design)

Start-up costs $135,477 $1,020,232 $618,584

Long-term 

sample 

needed

193,000 175,000 175,000
Third-step: Communicating results through 

governance structure

• Memo written with recommendation

• Gave deadline for objections

Cleared for Public Release 17
Takeaways

• Decision-making structure in place BEFORE decisions need to be made

• Determine strategy for obtaining buy-in from diverse stakeholders

• Ensure institutional support / internal consensus

• If differing opinion, make sure they have a voice

• Have data repository available for decision-making purposes

• Demographic and administrative data

• Have staff capable of high-level analysis of various data

Cleared for Public Release 18
Sample Redesign — current status

• Nine workgroups each focused on different phase or aspect of Sample 

Redesign

• Innovations:

• Modifications to the program which stratifies the primary sampling 

units

• Updates to our sample database

• Integration of National Health Interview Survey into Redesign apparatus

Cleared for Public Release 19
Overall timing of Sample Redesign

Cleared for Public Release 20

Receive draft corebased statistical area 

(CBSA) data

Mar. 2023

Receive final CBSA data

June 2023

Create new Primary 

Sampling Units (PSUs)

July 2023

Stratify new PSUs

Oct. 2023

Select new PSUs 

(including Consumer 

Expenditures, due end 

of October 2023)

Nov. 2023

Select new unit samples 

(Within-PSU samples)

Mar. 2024

Disseminate new 

samples to Field (FLD)

Nov. 2024

Start fielding new 

samples based on 2020 

design

Jan. 2025
References

Chesnut, T.J., Murphy, P., Park, Y., and Baker, S. (2018). Evaluation of 

2010 Sample Redesign Primary Sampling Units. Washington, DC: US 

Census Bureau.

Chesnut, T.J., Levy, R., Murphy, P. (2019). Recommendation to Select 

New Sample of Primary Sampling Units for Household Surveys after the 

2020 Census. Washington, DC: US Census Bureau.

Public Law No: 115-435

Cleared for Public Release 21
Questions / Contact

Rich Levy

richard.a.levy@census.gov

1-301-763-4553

Cleared for Public Release 22