1 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Imputation and Allocation of 

CE Data

Clayton Knappenberger

Economist

Division of Consumer Expenditure Surveys

2018 CE Microdata Users’ Workshop

July 20
2 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Outline

1. Process Overview

2. Imputation

3. Allocation

4. Edit Rates and Conclusion
3 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Process Overview

 CE’s goal is to map expenditures

As monthly amounts

To specific Universal Classification Codes (UCCs)

In a specific month and year

 However, collected data are often insufficient

Collected information has mistakes

Respondent does not know or refuses to provide
4 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Process Overview

1. Data Screening – check data for errors

2. Impute missing values

3. Allocate combined expenditures to 

components for mapping. 

4. Mapping expenditures to months and UCCs

(as well as higher level aggregations)
5 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Misclassified Records

Outliers

Data Screening

High Cost

Right Class

Wrong Class

Low Cost
6 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Misclassifications

 Specific keyword lookups for “hard to classify” 

items

iPad/iPhone/iPod

“Glasses”/”Cable”/”Nails”

 General text analysis of item descriptions

 Updates are made based on the reported item 

description and any interviewer notes
7 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Outlier Review

 Three methods:

1. Largest Gap – biggest gap between records 

above the mean

2. P-Index – value divided by gap minimum

3. Z-Score – value divided by IQR

 Updates are made by:

1. Correcting values with available information

2. Flag the expenditure for imputation
8 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Imputation

1. Weighted Mean Imputation

2. Hot Deck Imputation

Use valid records with similar characteristics to 

replace missing values 

3. Percent Distribution Imputation

 Randomly select a valid value based on the 

percent distribution of reported values
9 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Weighted Mean Imputation

 Use valid records with similar characteristics 

to define cells

 Calculate the weighted mean of that cell

 Assign the weighted mean of reported 

expenditures within a given cell to missing or 

invalid expenditures in the same cell
10 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Hot Deck Imputation Example

 A respondent reports buying a men’s jacket, 

but does not know the cost

 Imputation steps:

Select a valid random men’s jacket expenditure 

from all such purchases with the same:

– Region

– Area Type

– Income Class

The selected record’s expenditure amount is 

copied to the record being imputed
11 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Percent Distribution

 A respondent does not say how many people 

are covered by an insurance plan

Steps

1. Create CDF

2. Get Random Number

3. Assign value

0.98

0.60
12 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Allocation

 Example: Respondent reported spending $500 

on clothing

 We need two pieces of information:

1. Targets – shirts, pants, and shoes

2. Allocation Proportions

– 45% on shirts

– 35% on pants

– 20% on shoes 
13 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Picking the Target Items

1. Calculate 

Medians for items 

2. Calculate the 

CDF for items not 

already picked

3. Generate a 

random number 

to pick an item

4. Subtract item 

median from total 

expenditure

5. Is remainder 

greater than $0?

Repeat # 2 - 5

Items

1. Shirts

2. Pants

3. Shoes

Yes No
14 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Allocating the amounts

Item Mean ($) Share (%) Allocation ($)

Shirts $35.00 21.88% $109.40

Pants $67.00 41.87% $209.35

Shoes $58.00 36.25% $181.25

Total $160.00 100.00% $500.00

1. Get the weighted means for each item selected

2. Calculate the share of the sum of the means

3. Derive allocation amounts
15 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Imputation and Allocation Rates
16 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Why Impute and Allocate?

Benefits

 Meet internal needs 

for mapping

 Provide complete 

datasets to users

Concerns

 Our methods rely on 

MAR assumption

 Potential for 

underestimated 

variance
Contact Information

17 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Clayton Knappenberger

Economist

Division of Consumer Expenditure Surveys

www.bls.gov/cex

202-691-6236

knappenberger.clayton@bls.gov