Imputation and Allocation 

Consumer Expenditure Surv

Data

Meaghan Smith

Division of Consumer Expenditure Surveys

CE Microdata Users Workshop

July 14 – 17, 2015

Washington, DC
Outline

 Outlier Review Process Overview Imputation Allocation
Outlier Review

 Check for and validate 

extreme values

 Confirm that the record

is not misclassified
Outlier Review

 Outliers are determined using thr

methods.

Largest gapP-indexZ-score
Outlier Review

 Correction of an outlier is based on a r

of:

Consumer Unit characteristics like Income

Demographics, Area, Family size

Description of the expenseAudit trail investigationHistorical range of the category expense

 Values updated by either:

Correcting value based on examination of 

information in the Consumer Unit

Set to invalid and impute later in processi
Process Overview

 The goal of our expenditure processin

system is to map an expenditure repo

As a monthly amountTo a specific Universal Classification Code 

and

To a specific month and year

 However, we do not always get the de

needed to map the expenditures.

Consumer Unit cannot or will not provide 

detail

Field representative does not collect/ente

detail
So what do we do in these case
Process Overview

 The Consumer Expenditure Survey use

techniques to fill in missing data

Imputation Allocation

 Within these techniques we use:

Cell defining variablesSufficiency criteria

– Both controls vary by questionnaire section an
Process Overview

 Imputation procedures:

 Hot Deck Imputation Hot Deck Imputation at Item Level Weighting Class Imputation Percentage Distribution Imputation Month Imputation

 Allocation Procedures

 Allocation using Distribution Ratios Allocation using Fixed Ratios Allocation using Probability Distribution Ratios Allocation using Distribution Ratios of Reported Ta
IMPUTATION
Imputation

Hot Deck Imputation

This method uses valid entries from the c

quarter to assign values in fields containinvalues

The source records for this imputation are 

with similar characteristics
Hot Deck Imputation 

Example

 A respondent reports that a consumer unit m

purchased a men’s jacket, but does not know amount

 Imputation steps: 

 A random expenditure report for this specific item 

and specific age-sex classification (MEN) is selectematching values of:

– REGION– AREA TYPE– INCOME CLASS

 The expenditure amount from the randomly select

is assigned to the invalid cost variable on the origi
Method 1 in Concert Exa

 A respondent reports vehicle purchase, but d

know the principle amount borrowed

 Imputation steps: 

 A random expenditure report is selected from the 

defined by matching values of:

– Vehicle Type (car, truck, motorcycle, etc.)– New or Used Vehicle– Business Use Percentage– Income Class

 The principle amount borrowed (along with the ot

financing data) from the randomly selected record assigned to the record needing imputation (overlareported values in some cases)
Method 1 Same Schedu

Example

 A respondent reports three monthly ca

television bills, but only knows the bill amount for two of the months

 Imputation steps:

A random expenditure report is selected f

valid reports for the same consumer unit same expenditure code (cable television)

The expenditure amount from the random

selected record is assigned to the original 
Weighting Class Imputa

 Applies the mean of reported 

expenditures within a given cell to impute the missing or invalid expenditures in the same cell

 Source records for this imputation 

records with similar characteristic
Weighting Class Imputa

Example

 A respondent is unable to provide thei

monthly amount for dining out

 Imputation steps:

The weighted mean expenditure is calcula

the cell defined by matching values of:

– Income Class– Family Size

The weighted mean expenditure amount 

assigned to the original record
Percentage Distributio

Imputation

 This method randomly selects a valid v

from the weighted distribution of sourrecord values

 The source records for this imputation 

records with similar characteristics

 Primarily used for attribute variables o

variables used to classify expenditures more detailed levels
Percentage Distributio

Imputation Example

 A respondent is unable to say whether 

of the average monthly gasoline expenis for business

 Imputation steps:

The percentage and cumulative percentag

distributions of calibration weights are dethe attribute code in the cell defined by mvalues of:

– Income Class

A random number is selected between 0 a

and the attribute with the range containinrandom number is assigned to the origina
ALLOCATION
Distribution Ratio Bas

Allocation

 This method is used for allocation of b

reported and imputed combined expento five or less component items

 Randomly selects a valid value from th

weighted distribution based on weightmeans of source record values

 The source records for this imputation 

records with similar characteristics whconsumer unit reported separate valueeach item
Distribution Ratio Bas

Allocation Example

 A respondent reports a combined expe

infant’s clothing of $150, but does not the specific items included in the purch

 Allocation steps:

The percentage distributions of weighted 

are derived for all target items in the cell by matching values of:

– Income Class– Region– Area Type

The combined expense is allocated to all 

based on the percentage distribution
Fixed Ratio Based Alloca

 This method is used for gas and electr

proportions for non-vacation propertiedistribution ratio based allocation fails

 The allocation proportions are derived 

the previous calendar year’s Interview data
Probability Distribution R

Based Allocation

 This method uses percentiles for each 

component item in determining which component items are eligible for alloca

 This method is used for allocation of b

reported and imputed combined expento more than five component items
Probability Distribution R

Based Allocation

 If the combined expenditure is less tha

sum of the weighted median, allocate subset of component codes. Otherwisallocate to all component codes

 For records with combined expenditure

than the sum of the weighted median, a subset of target components from thpercentile that has at least two item cowith expenditures less than or equal to expenditure of the record requiring all
Probability Distribution R

Based Allocation

 Target components are randomly select

without replacement from the weighted cumulative distribution ratio

 After each selection, we subtract the me

expenditure of the selected component the reported expenditure requiring allocand continue with the next selection

 Once the selection process is complete, 

allocation is carried out based on percendistributions derived from weighted expenditures for each component item wmet the selection criteria
Probability Distribution R

Based Allocation

 Once the selection process is com

allocation is carried out based on percent distributions derived from weighted expenditures for each component item which met the selection criteria
Probability Distribution R

Based Allocation Exam

 A respondent reports combined appliance pu

but did not specify the items included in the 

 Allocation steps:

 Calculate the percentiles for all possible target com

for the combined item code in the cell defined by values of:

– Income Class

 The selection process randomly selects 6 of 12 tar

including stove, refrigerator, microwave, dishwashdisposal, and range hood

 The combined expense is allocated to selected tar

on the percentage distribution
Distribution Ratios to 

Reported Targets Alloca

 Beginning with 2005Q2, CE began coll

the specific item components of a comexpense

 This data allows CE to allocate expend

to the specific items purchased by the Consumer Unit in lieu of applying one previous allocation methods that use acomponents or a random sample

 This method is used for allocation of b

reported and imputed combined expento two or more specified targets
Distribution Ratios to Repo

Targets Allocation Examp

 A respondent reports a $500 clothing e

that includes pants, shirts, and shoes

 Allocation steps:

The percentage distributions of weighted 

are derived for the specified target items cell defined by matching values of:

– Age-Sex Classification– Income Class– Region– Area Type

The combined expense is allocated to the 

specified targets based on the percentage distribution
Imputation of Renta

Equivalence

 Beginning with processing of 200

regression model was implementeimpute for missing rental equivale

 Two different models

A model for owned homesAnother model for vacation propert
Imputation of Renta

Equivalence

 Both models use property value a

number of demographic and houscharacteristic variables

 Property value and demographic 

variables may have to be imputed

 Top coding is performed after 

imputation to account for any out
Meaghan Smith

Edit and Estimation Section Chief

Division of Consumer Expenditure Surv

www.bls.gov/cex

smith.meaghan@bls.gov