Adaptive Survey Design with Multiple Criteria:

the American Community Survey

Stephanie Coffey, Principal Statistician

Jonathan Eggleston, Senior Economist | Kendall Houghton, Economist | Carl Lieberman, Economist

Federal Committee on Statistical Methodology Conference

October 24th, 2023

1 Any opinions and conclusions expressed herein are those of the author and do not reflect the views of the U.S. Census Bureau.
Motivation for Adaptive Survey Design in the ACS

• Information collected in the ACS is critical

• Largest continuous household survey in the US

• Collects variety of information on household- and population-based topics

• Large overall sample sizes (3.5 million housing units per year)

• Sequential multimode survey design to control costs

• About 20% of ACS sample still ends up CAPI mode

• In the past (e.g., in 2022) data collection budgets were exhausted

• Data collection was stopped for all CAPI cases – no targeting

• No chance to change the respondent set at time of work stoppage

• Could exacerbate lead to nonresponse bias

• Goal: Create a Quality/Data-Driven Tool for Reallocating Effort

2
Adaptive Survey Design

• Framework for Data Collection

• Leverage tradeoffs

• Min(budget) for a fixed level of data quality

• Max(data quality) for a fixed budget

• Balance resources and quality

If we can save resources, willing to give up

“some” data quality

3
Adaptive Survey Design

• Framework for Data Collection

• Leverage tradeoffs

• Min(budget) for a fixed level of data quality

• Max(data quality) for a fixed budget

• Balance resources and quality

If we can save resources, willing to give up

“some” data quality

4

Workload Cases

Data Collection Costs

min Impact on Variance

max

max

Normal Data Collection
Adaptive Survey Design

• Framework for Data Collection

• Leverage tradeoffs

• Min(budget) for a fixed level of data quality

• Max(data quality) for a fixed budget

• Balance resources and quality

If we can save resources, willing to give up

“some” data quality

5

Workload Cases

Data Collection Costs

min Impact on Variance

max

max

Stop “Some” of the Cases
Adaptive Survey Design

• Framework for Data Collection

• Leverage tradeoffs

• Min(budget) for a fixed level of data quality

• Max(data quality) for a fixed budget

• Balance resources and quality

If we can save resources, willing to give up

“some” data quality

6

Workload Cases

Data Collection Costs

min Impact on Variance

max

max

Stop “Half” of the Cases
Adaptive Survey Design

• Framework for Data Collection

• Leverage tradeoffs

• Min(budget) for a fixed level of data quality

• Max(data quality) for a fixed budget

• Balance resources and quality

If we can save resources, willing to give up

“some” data quality

7

Data Collection Costs

min Impact on Variance

max

max

Stop “Most” of the Cases

Workload Cases
Adaptive Survey Design

• Need predictive models to determine the impact of stopping cases

• Typically care about:

• Quality – summary statistic (variance inflation, MSE(item), CVs, etc.)

• Cost – cost-per-outcome (response, nonresponse, etc.) – 2 viewpoints

• “Cut Costs” - other interviewer behavior stays the same, costs are reduced

• “Reallocate Costs” – shifts resources from stopped cases to retained cases

• Response Behavior – will a case actually respond?

• Likely nonrespondents have impact on cost, but not on quality (vs baseline)

• Use model output to identify which cases to stop and their impact on 

quality

8
Adaptive Survey Design

• Need predictive models to determine the impact of stopping cases

• Typically care about:

• Quality – summary statistic (variance inflation, MSE(item), CVs, etc.)

• Cost – cost-per-outcome (response, nonresponse, etc.) – 2 viewpoints

• “Cut Costs” - other interviewer behavior stays the same, costs are reduced

• “Reallocate Costs” – shifts resources from stopped cases to retained cases

• Response Behavior – will a case actually respond?

• Likely nonrespondents have impact on cost, but not on quality (vs baseline)

• Use model output to identify which cases to stop and their impact on 

quality

9
Which cases to stop?

• Juxtapose “importance” of case with “likelihood to respond”

10

�

�bbii ZZ

1 − ρ�ii XX

Overrepresented, given 

Z

Less likely to respond, given X
Which cases to stop?

• Juxtapose “importance” of case with “likelihood to respond”

11

Variance Inflation Factor Ratio of Alternate Strategy to Baseline Strategy

  Increasing Variance Inflation Factor  

Cost Ratio of Alternate Strategy to Baseline Strategy



 Decreasing Costs 





(1.00, 1.00) �

�bbii ZZ

1 − ρ�ii XX

Overrepresented, given 

Z

Less likely to respond, given X
Which cases to stop?

• Juxtapose “importance” of case with “likelihood to respond”

12

�

�bbii ZZ

1 − ρ�ii XX

Overrepresented, given 

Z

Less likely to respond, given X

stop work 

set of cases cc�ii

cc�ii

cc�ii

cc�ii cc�ii

cc�ii

cc�ii

�sscc�ii

Cost savings:

Variance Inflation Factor Ratio of Alternate Strategy to Baseline Strategy

  Increasing Variance Inflation Factor  

Cost Ratio of Alternate Strategy to Baseline Strategy



 Decreasing Costs 





(1.00, 1.00)

(1.01, 0.95)
Which cases to stop?

• Juxtapose “importance” of case with “likelihood to respond”

13

�

�bbii ZZ

1 − ρ�ii XX

Overrepresented, given 

Z

Less likely to respond, given X

stop work 

set of cases

�sscc�ii

Cost savings:

Variance Inflation Factor Ratio of Alternate Strategy to Baseline Strategy

  Increasing Variance Inflation Factor  

Cost Ratio of Alternate Strategy to Baseline Strategy



 Decreasing Costs 





(1.00, 1.00)

(1.01, 0.95)

(1.15, 0.70)
Which cases to stop?

• Juxtapose “importance” of case with “likelihood to respond”

14

�

�bbii ZZ

1 − ρ�ii XX

Overrepresented, given 

Z

Less likely to respond, given X

stop work 

set of cases

�sscc�ii

Cost savings:

Variance Inflation Factor Ratio of Alternate Strategy to Baseline Strategy

  Increasing Variance Inflation Factor  

Cost Ratio of Alternate Strategy to Baseline Strategy



 Decreasing Costs 





(1.00, 1.00)

(1.01, 0.95)

(1.15, 0.70)

(2.50, 0.60)
When to stop stopping cases?

• Optimization tool is flexible

• Can be adjusted given goals

15

Variance Inflation Factor Ratio of Alternate Strategy to Baseline Strategy

  Increasing Variance Inflation Factor  

Cost Ratio of Alternate Strategy to Baseline Strategy



 Decreasing Costs 





(1.00, 1.00)

(1.01, 0.95)

(1.15, 0.70)

(2.50, 0.60)
When to stop stopping cases?

• Optimization tool is flexible

• Can be adjusted given goals

• Goal 1: Need to reallocate 5% of budget

Need to accept a 1% variance inflation

16

Variance Inflation Factor Ratio of Alternate Strategy to Baseline Strategy

  Increasing Variance Inflation Factor  

Cost Ratio of Alternate Strategy to Baseline Strategy



 Decreasing Costs 





(1.00, 1.00)

(1.01, 0.95)

(1.15, 0.70)

(2.50, 0.60)
When to stop stopping cases?

• Optimization tool is flexible

• Can be adjusted given goals

• Goal 1: Need to reallocate 5% of budget

Need to accept a 1% variance inflation

• Goal 2: Maximize resource shift for a 15% 

increase (or less) in variance inflation

You can obtain a 30% resource shift

17

Variance Inflation Factor Ratio of Alternate Strategy to Baseline Strategy

  Increasing Variance Inflation Factor  

Cost Ratio of Alternate Strategy to Baseline Strategy



 Decreasing Costs 





(1.00, 1.00)

(1.01, 0.95)

(1.15, 0.70)

(2.50, 0.60)
When to stop stopping cases?

• Optimization tool is flexible

• Can be adjusted given goals

• Goal 1: Need to reallocate 5% of budget

Need to accept a 1% variance inflation

• Goal 2: Maximize resource shift for a 15% 

increase (or less) in variance inflation

You can obtain a 30% resource shift

• Goal 3: Balance cost and variance inflation

Find best tradeoff – minimum of product

18

Variance Inflation Factor Ratio of Alternate Strategy to Baseline Strategy

  Increasing Variance Inflation Factor  

Cost Ratio of Alternate Strategy to Baseline Strategy



 Decreasing Costs 





(1.00, 1.00)

(1.01, 0.95)

(1.15, 0.70)

(2.50, 0.60)

(1.20, 0.65)
Mathematical Optimization

• Formalize Idea

• φ A = mmmmmm

AA(iiεεss)

CĈAA|ssAA,RRAA

CĈ0|ss0,RR0

VV�AA|ssAA,RRAA

VV�0|ss0,RR0

• Response propensity model rit = pp(RR = 1|XX)

• Balancing propensity model (for ranking) bit = pp(RR = 1|ZZ)

• Cost model cĉ

ii

oo + cĉ

ff

oo

• Variance inflation formula VV� AA/ VV� 0

• Examples in the literature

• National Survey of College Graduates – minimize RMSE of key statistic (salary)

• National Survey of Family Growth – minimize the MSE of several key statistics

• Dutch Labor Force Survey – minimize mode effects in multimode survey

19

where 0 is the baseline strategy (normal data collection), and 

A is the alternate strategy (some set, s, of cases stopped)
What is Unique about the ACS?

• ACS is cross-sectional, with no past response data for modeling

• Need covariates for the balancing model

• Characteristics related to outcomes of interest

• Broad set of administrative data

[MAF, Commercial Housing Data, IRS, SSA, Demographic Data from 

2010/2020 Census, etc.]

• Assign characteristics to sample units for the balancing model

[sex, age, race/ethnicity, marital status, income, program participation flags, 

housing structure vars, etc.]

20
What is Unique about the ACS?

21

• ACS is Pseudo-longitudinal

• Data collected for one panel is not released independently

• 12 months of data combined into annual estimates (or 60 months for 5-yr)

• Interventions we make could impact estimates for a long time

• Use two balancing propensity models to account for time: b1it, b3it
What is Unique about the ACS?

22

�

�bbii ZZ

1 − ρ�ii XX

Overrepresented, given 

Z

Less likely to respond, given X

current month

• ACS is Pseudo-longitudinal

• Data collected for one panel is not released independently

• 12 months of data combined into annual estimates (or 60 months for 5-yr)

• Interventions we make could impact estimates for a long time

• Use two balancing propensity models to account for time: b1it, b3it
What is Unique about the ACS?

• ACS is Pseudo-longitudinal

• Data collected for one panel is not released independently

• 12 months of data combined into annual estimates (or 60 months for 5-yr)

• Interventions we make could impact estimates for a long time

• Use two balancing propensity models to account for time: b1it, b3it

23

�

�bbii ZZ

1 − ρ�ii XX

Overrepresented, given 

Z

Less likely to respond, given X

�

�bbii ZZ

1 − ρ�ii XX

Overrepresented, given 

Z

Less likely to respond, given X

current month last two months
What is Unique about the ACS?

• ACS estimation geographies are Census tracts (average pop ~4,000)

• Sample is spread across 12 months – very small sample sizes

• Can’t run balancing propensity models at the tract level

• Unstable, too noisy, sometimes only 1 or 2 cases!

• Run balancing propensities at the state level

• Could lead to stopping work on all cases in a tract

• Tract eligibility rules. At time of stop work, tracts must have:

• At least 3 sample cases

• At least one occupied interview

• Retain enough sample to have two complete interviews

• This has led to a conservative stop work algorithm

• Allowed us to develop models, code, evaluation metrics

• Release version 1 of process – for future improvements

24
What is Unique about the ACS?

• ACS estimation geographies are Census tracts (average pop ~4,000)

• Sample is spread across 12 months – very small sample sizes

• Can’t run balancing propensity models at the tract level

• Unstable, too noisy, sometimes only 1 or 2 cases!

• Run balancing propensities at the state level

• Could lead to stopping work on all cases in a tract

• Tract eligibility rules. At time of stop work, tracts must have:

• At least 3 sample cases

• At least one occupied interview

• Retain enough sample to have two complete interviews

• This has led to a conservative stop work algorithm

• Allowed us to develop models, code, evaluation metrics

• Release version 1 of process – for future improvements

25
What is Unique about the ACS?

• ACS estimation geographies are Census tracts (average pop ~4,000)

• Sample is spread across 12 months – very small sample sizes

• Can’t run balancing propensity models at the tract level

• Unstable, too noisy, sometimes only 1 or 2 cases!

• Run balancing propensities at the state level

• Could lead to stopping work on all cases in a tract

• Tract eligibility rules. At time of stop work, tracts must have:

• At least 3 sample cases

• At least one occupied interview

• Retain enough sample to have two complete interviews

• This has led to a conservative stop work algorithm

• Allowed us to develop models, code, evaluation metrics

• Release version 1 of process – for future improvements

26
What have we done so far?

• Development lasted from December 2022 – June 2023

• Data access and linkage

• Model development

• June 2023 – end-to-end test – stopped one case per state

• Implementation began July 2023

• Stop Work Interventions Delivered for July, August, September, October

• Midway through CAPI, carried out optimization procedures

• Stopped 50% of the optimal number of cases for stop work (after restrictions)

27
What are we monitoring?

• Initial Monitoring

• Unweighted CAPI completion rate

• Mean balancing propensities

• CV(mean) balancing propensities

• R-Indicators (overall- and state-level)

• Future Monitoring

• Attempts / hours on cases retained after stop work

• Longer-Term Monitoring

• ACS 1-year estimates

28
Next Steps

• Consider different geographies for the balancing (quality) model

• Urban-Rural? MSA? Something else?

• How does changing the geography change stop work patterns?

• Improving cost/resource model 

• Add in additional covariates, geography, interviewer/workload characteristics

• Incorporating mileage into the resource model

• Investigating impacts of resource reallocation

• More hours on cases retained? More attempts? 

• Impact on response propensities for cases that are retained?

• Continuing to monitor outcomes 

• Completion rates, hours spent on cases, R-indicators, etc.

29
Contact & Acknowledgements

Stephanie Coffey

stephanie.coffey@census.gov

Thanks to the entire ACS program... 

survey operations, survey design, field operations, household nonresponse teams

...for their interest and support of our work!!!

30