Katherine Jenny Thompson (U.S. Census Bureau)

Laura Bechtel (U.S. Census Bureau)*

Nicole Czaplicki (U.S. Census Bureau)

2018 Federal Committee on Statistical Methodology

March 7, 2018

Evaluating Hot Deck with 

Propensity Score Matching For 

the Advance Monthly Retail 

Trade Survey (MARTS)

* Any views expressed are those of the authors and not necessarily those of 

the U.S. Census Bureau. 1
Outline

▪ Hot Deck 

▪ Propensity Matching

▪ Relationship between Hot Deck Imputation 

and Propensity Matching

▪ Our Application

▪ Evaluation

▪ Concluding Remarks

▪ Related Research

2

Imputation Cell

Donated

Donated

Random Hot Deck

Donated

Assumes MCAR or MAR 

when imputation cells are 

used.

Assumes the expected 

value of outcome of 

interest is the same for all 

observations within the 

imputation cell.

Donors

Recipients

4
Imputation Cell

Nearest Neighbor Hot Deck

Calculate distances based 

on auxiliary information

D2

D1

D3

D4

D5

Imputing for this 

observation

5
Imputation Cell

Nearest Neighbor Hot Deck

Min Distance Donated

Assumes the outcome 

variable can be predicted 

by the auxiliary variables

6
Hot Deck With Business Surveys

▪ Skewed population

▪ Direct donation not a good idea for quantitative variables

▪ Nearest Neighbor often used (size predictive of 

response/outcome)

▪ Derived value – donor ratio

▪ More recipients than donors 

▪ Seasonal effects/trading day effects

7
Propensity 

Score Matching

▪ Background 

▪ Causal inference/causal assumptions

▪ Predicting outcome variable (response to 

treatment due to factors that are common to both 

treatment and control)

▪ Propensity Score

▪ One single score or combinations of variables

8
What About Propensity Scoring?

Everything 

into the 

score (all 

variables)

Develop a 

score 

within a 

block

No score 

(block)

How do you 

develop one 

appropriate score

function?

What about 

important 

continuous 

variables?

Compromise 

between the two 

methods

9
How propensity scoring works

▪ Matching

▪ Need to specify a distance function.

▪ Cannot re-use donors (one to one or many to one).

▪ Greedy matching1

▪ Pairs donors to recipients sequentially. 

▪ Sort matters (confounding with distance). 

▪ Need to have more donors than recipients to use. 

▪ Optimal matching1

▪ Pairs donors to recipients based on closest distance subject to 

minimizing total aggregated distance over all recipients. 

▪ Distance function matters.

1Used publicly available SAS code developed by Bergstralh and Kosanke at the Mayo Clinic 

(http://www.mayo.edu/research/departments-divisions/department-health-sciences-research/divisionbiomedical-statistics-informatics/software/locally-written-sas-macros)

10
Greedy matching 

Distance between donor W 

and Recipient A is 7

Donors

Recipients W X Y Z

A 7 8 5 13

B 10 9 4 6

C 11 17 8 10

D 25 14 7 8

Recipients sorted in 

ascending sequence

Donor Y is 

matched 

with 

Recipient A

11
Greedy matching 

Donors

Recipients W X Y Z

A 7 8 5 13

B 10 9 4 6

C 11 17 8 10

D 25 14 7 8

12
Greedy matching 

Donors

Recipients W X Y Z

A 7 8 5 13

B 10 9 4 6

C 11 17 8 10

D 25 14 7 8

13
Greedy matching 

Donors

Recipients W X Y Z

A 7 8 5 13

B 10 9 4 6

C 11 17 8 10

D 25 14 7 8

14
Greedy matching –sort matters 

Donors

Recipients W X Y Z

D 25 14 7 8

C 11 17 8 10

B 10 9 4 6

A 7 8 5 13

Recipients sorted in 

descending sequence

Donor W is 

matched 

with 

Recipient A

15
Optimal matching

Donors

Recipients W X Y Z

A 7 8 5 13

B 10 9 4 6

C 11 17 8 10

D 25 14 7 8

Donor X is 

matched 

with 

Recipient A

16

Our application - Advance Monthly 

Retail Trade Survey (MARTS)

▪ Monthly Economic Indicator

▪ Sales and month-to-month percent change

▪ Inputs into the quarterly Gross Domestic Product 

(GDP) produced by the Bureau of Economic 

Analysis

▪ MARTS is a subsample of Monthly Retail Trade 

Survey (MRTS)

▪ Certainties – selected with probability = 1

18
MARTS MRTS

Sample size 5,000 companies 12,000 companies

Sample frame MRTS sample Annual Retail Trade Survey 

sample

Sample design Stratified PPS -WOR (subsample 

of MRTS)

Stratified SRS-WOR

Sample redesign

cycle

Approximately every 2.5 years Approximately every 5 years

Time to respond Approximately 7 business days Approximately 5 weeks

Imputation Analyst impute for selected 

companies

Analyst imputes retained, ratio 

impute for remaining 

nonrespondents and edit-failing 

items

Estimation Link relative estimator Horvitz-Thompson estimator

Tabulation industries 30 83

19
Our application - Advance Monthly Retail 

Trade Survey (MARTS)

▪ The largest MRTS Certainties are selected with 

certainty for MARTS

▪ Low Response Rates & 

Size is Predictive of Response

▪ Data are seasonally adjusted

▪ Seasonal effects

▪ Trading day effects – many series

MRTS 

Certainties

MARTS 

Certainties

20
Simulation Study Design

Source Data:

▪ In Statistical Period

▪ March 2016 – Feb. 2017

▪ MRTS Certainty Units ONLY

▪ Responded to MRTS

▪ Current Period and Prior 

Period

▪ Both values of sales > 0

MRTS Certainty Units 

(Not In MARTS)

MARTS Certainty Units

MARTS Noncertainty Units

21
Simulation Study Design

MRTS Certainty Units 

(Not In MARTS) - Donors

MARTS Certainty Units – Donors

MARTS Certainty Units – Recipients

MARTS Noncertainty Units - Recipients

Randomly split within 

Statistical Period

22

Finding Matching Variables

▪ What variables are predictive of month-tomonth change?

▪ Industry – 6-digit NAICS (North American Industry 

Classification System) vs 3-digit NAICS

24
Distributions of Month-to-Month 

Change in NAICS 448

25
Finding Matching Variables

▪ What variables are predictive of month-tomonth change?

▪ A lot is built into the imputation cells

▪ Industry – 6-digit NAICS (North American Industry 

Classification System) vs 3-digit NAICS

▪ Unit size

26
Distributions of Month-to-Month 

Change in NAICS 448

27
Finding Matching Variables

▪ Predictive of m-t-m change

▪ A lot is built into the imputation cells

▪ Industry – 6-digit NAICS (North American Industry 

Classification System) vs 3-digit NAICS

▪ Unit size is important – but we are restricted to MRTS 

certainty only (historic data limitations)

▪ Other factors investigated

▪ Prior month sales (size)

▪ Sampling weight (size)

▪ Variables predictive of response

28
Actual Matches

▪ Blocks/Imputation Cells – 6-digit industry

▪ Matching variables

▪ Prior month sales

▪ Number of industries that the company operates 

in (proxy for complexity of the company)

29
Evaluation

Hot deck method Match Variables Sort Variables

Greedy 1 Random hot deck Random number Random 

number

2 Nearest neighbor Prior Month Sales Random 

number

3 Propensity Prior Month Sales Prior months 

sales 

(descending)

4 Propensity Prior Month Sales and Number 

of Identified Industries for 

Reporting Unit

Random 

number

5 Propensity Prior Month Sales and Number 

of Identified Industries for 

Reporting Unit

Prior month 

sales 

(descending)

Optimal 1 Propensity Prior Month Sales N/A

2 Propensity Prior Month Sales and Number 

of Identified Industries for 

Reporting Unit

N/A

30



One Match Variable Versus Two

▪ Chi-square tests for independence

▪ Treatment = two match variables

▪ Control = one match variable

▪ Optimal and Greedy match – no improvement 

with two

34
Phase 1 Summary

Looking at MAE and CRB

▪ Random Hot Deck worst performance

▪ Nearest Neighbor slight underperformance 

compared to Optimal and Greedy

▪ Greedy and Optimal similar performance 

▪ Greedy - ミeeded to さtriIkざ the Iode

▪ Phase 2 will focus on Optimal Matching

35
Phase 2: Selection of Hot Deck 

Donor Pool

Ratio Min. Q1 Med. Q3 Ma

x

Donors (1 Year Ago) to Recipients 0.89 1.69 2.14 3.19 5.58

Donors (5 Years Ago) to Recipients 0.55 0.97 1.38 1.69 2.70

36
Phase 2: Chi-Square Test for 

Independence to Assess Treatment 

Effect (Donor Choice)

▪ Example where p-value is misleading

▪ There is a aミ effeIt overall...Hut it igミores 

differences within industries

1 year ago 

outperformed

5 years ago 

outperformed

Tie between 

the 2 

treatments

MAE 17 11 2

URB 16 9 5

37
Concluding Remarks

▪ Optimal matching effective

▪ Parsimonious model works

▪ No need for a single score in our application

▪ Challenge in determining how to use donors

▪ No one-size-fits-all model with for choosing ratios

▪ Considering alternative calendar adjustments

38
Related Research

▪ Comparison to other missing data treatments 

as part of a larger study

▪ 10:30 tomorrow morning in 145AB Nikki Czaplicki

is preseミtiミg さFiミdiミg aミ Estiマator that Miミiマizes 

Revisioミs iミ a Moミthly IミdiIator “urveyざ

39
Thank you

Laura Bechtel

Laura.Bechtel@census.gov

40