An Approach To Measurement Error Assessment 

For A Household Expenditure Survey

A Review And Application

Brandon Kopp, Lucilla Tan, & Scott Fricker

Bureau of Labor Statistics

Roger Tourangeau

Westat

Federal Committee on Statistical Methodology

5 November 2013
Introduction

 The Consumer Expenditure Survey (CE) is involved in a multiyear redesign effort

 One goal of the redesign is to “improve data quality through a 

verifiable reduction in measurement error”

 Goal of this project was to research specific measures that could 

be used on an ongoing basis to track measurement error in the 

CE over time

2

?

Time

Redesign Implemented
Introduction

 We reviewed 37 past studies (internal reports, 

conference proceedings, and journal articles) that 

examined error in the CE

 A variety of methods have been used to assess error 

in the CE, each with their strengths and weaknesses

 We recommend a multi-method-indicators (MMI) 

approach that consists of three main categories:

 Internal indicators (CE survey data or paradata)

 External indicators (Comparisons with external data sources)

 Record check studies (Comparisons with financial records)

3
Internal Indicators

 Those based solely on CE data or information about 

the data collection process

 Examples: 

 Comparing level of expenditures from earlier waves with 

later waves

 Number of contact attempts required for a completed 

interview

 Weaknesses -

 Can’t really estimate the magnitude of the errors (e.g., 

proportion of respondents consulting records);

 These are indirect indicators.

4
Internal Indicators

Criteria for Selection

 Sources of error: Should give some indication of 

the relative magnitude of different types of reporting 

error (recall, conditioning, satisficing)

 Availability/stability: Should tap data that will be 

available over time

 Utility for improving items or survey 

procedures: Should help identify problems that can 

be fixed

5
An Illustrative Set of 

Internal Indicators

 Entries in the CE Diary versus data collected by recall 

at pickup (recall error);

 The ratio of the number of entries in diary week one 

and diary week two (conditioning);

 The percentage of respondents who use records 

during the Quarterly Interviews (recall error);

 Interview duration (satisficing).

6
External Indicators

 Comparison to external data sources

 Examples:

 Personal Consumption Expenditures (PCE) from National 

Income and Product Accounts 

 Other household surveys (e.g., MEPS, PSID, RECS)

 Weaknesses —

 Not clear external benchmarks are really more accurate than 

the CE;

 Errors in PCE are not well established. 

7
External Indicators

Criteria for Selection

 Cover a range of categories, including some that 

differ in the likely availability of records;

 Include both regular (e.g., rent/mortgage, utilities) 

and irregular (e.g., clothing) expenditures;

 Include both large and small expenditures;

 Focus on categories in which the external source 

uses a definition that is reasonably consistent with 

the CE definition.

8
External Indicators

Other Household Surveys

9

Consumer 

Expenditure 

Survey

American 

Community 

Survey

Residential 

Energy 

Consumption 

Survey

Medical 

Expenditure 

Panel Survey

Panel Survey 

on Income 

Dynamics

Survey of 

Consumer 

Finances

Rent and Mortgage

Utilities and Fuel

Medical and Health

Utilities and Fuel

Medical and Health

Utilities and Fuel

Mortgage
Record Check/Validation Studies

 Compare CE reports to actual bills or other 

records

 In principle, this method most accurately 

quantifies error magnitude, but it has many 

practical difficulties

 Relatively expensive;

 Respondents don’t save all of their records;

 Respondents may not want to share their records;

 Some types of purchase unlikely to generate 

records;

 Likely to produce high rates of unit and item 

nonresponse.

10
Record Check/Validation Studies

Criteria for Selection

 Use many of the same categories as in 

external indicators to assess convergence

Utilities and fuel;

Rent and mortgages; and

Hospitalization and health insurance.

 And others that are likely to yield receipts

Men’s and boy’s clothing;

Women’s and girls clothing;

Major appliances; and

Food away from home.

11
Additional Considerations

 Cost: What inputs are needed to develop 

each MMI component?

 Duration for development: How long will 

the development efforts take?

 Applicability: Is the component applicable 

only to the current CE design or will it remain 

applicable to other designs?

 Periodicity: How often can the indicators be 

tracked?

12
Summary

 No one approach is perfect

 Develop a time series with multiple indicators

Internal indicators

External indicators

Supplemented with periodic (but regular) record 

check studies

 Have overlapping expenditure categories to 

assess convergence across methods

13
Contact Information

Brandon Kopp

Research Psychologist

Office of Survey Methods Research

www.bls.gov/osmr

202-691-7514

kopp.brandon@bls.gov